为了账号安全,请及时绑定邮箱和手机立即绑定

scrapy ERROR:蜘蛛错误处理问题

scrapy ERROR:蜘蛛错误处理问题

慕森卡 2021-08-17 16:42:37
我对scrapy很陌生,在运行我的代码时,遇到了这个错误。我的代码import urlparsefrom scrapy.http import Requestfrom scrapy.spiders import BaseSpiderclass legco(BaseSpider):name = "sec_gov"allowed_domains = ["www.sec.gov", "search.usa.gov", "secsearch.sec.gov"]start_urls = ["https://www.sec.gov/cgi-bin/browse-edgar?company=&match=&CIK=&filenum=&State=&Country=&SIC=2834&owner=exclude&Find=Find+Companies&action=getcompany"]#extract home page search resultsdef parse(self, response):for link in response.xpath('//div[@id="seriesDiv"]//table[@class="tableFile2"]/a/@href').extract():    req = Request(url = link, callback = self.parse_page)    print link    yield req#extract second link search resultsdef parse_second(self, response):for link in response.xpath('//div[@id="seriesDiv"]//table[@class="tableFile2"]//*[@id="documentsbutton"]/a/@href').extract():    req = Request(url = link, callback = self.parse_page)    print link    yield req一旦我尝试运行此代码:scrapy crawl sec_gov 出现此错误。2018-11-14 15:37:26 [scrapy.core.engine] INFO: Spider opened2018-11-14 15:37:26 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)2018-11-14 15:37:26 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:60232018-11-14 15:37:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.sec.gov/cgi-bin/browse-edgar?company=&match=&CIK=&filenum=&State=&Country=&SIC=2834&owner=exclude&Find=Find+Companies&action=getcompany> (referer: None)2018-11-14 15:37:27 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.sec.gov/cgi-bin/browse-edgar?company=&match=&CIK=&filenum=&State=&Country=&SIC=2834&owner=exclude&Find=Find+Companies&action=getcompany> (referer: None)Traceback (most recent call last):File "/home/surukam/.local/lib/python2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbackscurrent.result = callback(current.result, *args, **kw)谁能帮我这个 ?提前致谢
查看完整描述

1 回答

  • 1 回答
  • 0 关注
  • 582 浏览
慕课专栏
更多

添加回答

举报

0/150
提交
取消
意见反馈 帮助中心 APP下载
官方微信