Scrapy爬取天天美剧封面照及剧集下载地址

其实我只是想试试爬取图片而已j_0007.gif,先看看网页，需要爬的地方有两个，一是封面图，二是下载地址，挺简单的

wKioL1ZKmlPhdfkBAAI4-swYALQ626.png

Item定义：

import scrapy

class TiantianmeijuItem(scrapy.Item):

    name = scrapy.Field()

    image_urls = scrapy.Field()

    images = scrapy.Field()

    image_paths = scrapy.Field()

    episode = scrapy.Field()

    episode_url = scrapy.Field()

name是保存名字

image_urls和images 是爬取图片的pipeline用的，一个是保存图片URL，一个是保存图片存放信息

image_paths其实没什么实际作用，只是记录下载成功的图片地址

epiosde和episode_url是保存集数和对应下载地址

Spider：

import scrapy

from tiantianmeiju.items import TiantianmeijuItem 

import sys

reload(sys) # Python2.5 初始化后会删除 sys.setdefaultencoding 这个方法，我们需要重新载入

sys.setdefaultencoding('utf-8')

class CacthUrlSpider(scrapy.Spider):

    name = 'meiju'

    allowed_domains = ['cn163.net']

    start_urls = ["http://cn163.net/archives/{id}/".format(id=id) for id in ['16355', '13470', '18766', '18805']]

    def parse(self, response):

        item = TiantianmeijuItem()

        item['name'] = response.xpath('//*[@id="content"]/div[2]/div[2]/h2/text()').extract()

        item['image_urls'] = response.xpath('//*[@id="entry"]/div[2]/img/@src').extract()

        item['episode'] = response.xpath('//*[@id="entry"]/p[last()]/a/text()').extract()

        item['episode_url'] = response.xpath('//*[@id="entry"]/p[last()]/a/@href').extract()

        yield item

页面比较简单

Pipelines：这里写了两个管道，一个是把下载链接保存到文件，一个是下载图片

import json

import os

from scrapy.pipelines.images import ImagesPipeline

from scrapy.exceptions import DropItem

from scrapy.http import Request

from settings import IMAGES_STORE

class TiantianmeijuPipeline(object):

    def process_item(self, item, spider):

        return item

class WriteToFilePipeline(object):  

    def process_item(self, item, spider):

        item = dict(item)

        FolderName = item['name'][0].replace('/', '')  

        downloadFile = 'download_urls.txt'

        with open(os.path.join(IMAGES_STORE, FolderName, downloadFile), 'w') as file:

            for name,url in zip(item['episode'], item['episode_url']):

                file.write('{name}: {url}\n'.format(name=name, url=url))

        return item

class MyImagesPipeline(ImagesPipeline):  

    def get_media_requests(self, item, info):

        for image_url in item['image_urls']:

            yield Request(image_url, meta={'item': item})

    def item_completed(self, results, item, info):

        image_paths = [x['path'] for ok,x in results if ok]

        if not image_paths:

            raise DropItem("Item contains no images")

        item['image_paths'] = image_paths

        return item

    def file_path(self, request, response=None, info=None):

        item = request.meta['item']

        FolderName = item['name'][0].replace('/', '')

        image_guid = request.url.split('/')[-1]

        filename = u'{}/{}'.format(FolderName, image_guid)

        return filename

get_media_requests和item_completed，因为默认的图片储存路径是

<IMAGES_STORE>/full/3afec3b4765f8f0a07b78f98c07b83f013567a0a.jpg，

我需要把full改成以美剧名字目录来保存，所以重写了file_path

settings打开pipelines相关配置：

ITEM_PIPELINES = {

    'tiantianmeiju.pipelines.WriteToFilePipeline': 2,

    'tiantianmeiju.pipelines.MyImagesPipeline': 1,

}

IMAGES_STORE = os.path.join(os.getcwd(), 'image')   # 图片存储路径

IMAGES_EXPIRES = 90

IMAGES_MIN_HEIGHT = 110

IMAGES_MIN_WIDTH = 110

爬下来之后就是这个效果了：

wKiom1ZK5eHwX9p2AACkME9RTbw155.png

wKioL1ZK5jPAQ7jKAAB1PPPDOCI622.png

©著作权归作者所有：来自51CTO博客作者lihuipeng的原创作品，如需转载，请注明出处，否则将追究法律责任