使用scrapy+redis从一定量的淘宝详情页url获取商品详情已设置user-agent,已传入cookie,已设置proxy-ip获取url,response.status有时是200,有时是302,随机改变1000个url,成功获取商品信息大概有400多是否为cookie未传入成功,还是proxy-ip不稳定?或者其他原因。请帮忙分析,谢谢!报错Traceback:2017-07-1415:51:12[scrapy.core.engine]DEBUG:Crawled(200)(referer:None)2017-07-1415:51:12[requests.packages.urllib3.connectionpool]INFO:StartingnewHTTPSconnection(1):rate.taobao.com2017-07-1415:51:12[requests.packages.urllib3.connectionpool]DEBUG:"GET/detailCommon.htm?auctionNumId=10245430841HTTP/1.1"200None2017-07-1415:51:12[scrapy.core.scraper]DEBUG:ScrapedfromNone2017-07-1415:51:12[taobao]DEBUG:Read1requestsfrom'taobao:start_urls'2017-07-1415:51:12[scrapy.downloadermiddlewares.cookies]DEBUG:Sendingcookiesto:2017-07-1415:51:12[scrapy.downloadermiddlewares.redirect]DEBUG:Redirecting(302)tofrom2017-07-1415:51:12[scrapy.downloadermiddlewares.cookies]DEBUG:Sendingcookiesto:2017-07-1415:51:12[scrapy.core.engine]DEBUG:Crawled(200)(referer:None)['partial']2017-07-1415:51:12[scrapy.core.scraper]ERROR:Spidererrorprocessing(referer:None)
添加回答
举报
0/150
提交
取消