求助：scrapy爬取数据失败，反复调试都不成功

Question

目标：爬取某一学习网站上课程信息，前期调试仅获取课程名称爬虫文件： import scrapyfrom xtzx.items import XtzxItem from scrapy.http import Request class LessonSpider(scrapy.Spider): name = 'lesson' allowed_domains = ['xuetangx.com'] start_urls = ['http://www.xuetangx.com/courses/course-v1:TsinghuaX+80512073X+2018_T1/about'] ''' def start_requests(self): ua={"User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko"} yield Request("www.xuetangx.com/courses/course-v1:TsinghuaX+80512073X+2018_T1/about",headers=ua) ''' def parse(self, response): item=XtzxItem() item["title"]=response.xpath("//div[@class='title_detail'/h3[@class='courseabout_title']/text()").extract() print(item["title"]) 执行日志： 2018-04-28 11:08:33 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: xtzx)2018-04-28 11:08:33 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 2.9.7, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted 17.9.0, Python 3.5.4 (v3.5.4:3f56838, Aug 8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)], pyOpenSSL 17.5.0 (OpenSSL 1.1.0h 27 Mar 2018), cryptography 2.2.2, Platform Windows-10-10.0.16299-SP02018-04-28 11:08:33 [scrapy.crawler] INFO: Overridden settings: {'SPIDER_MODULES': ['xtzx.spiders'], 'BOT_NAME': 'xtzx', 'NEWSPIDER_MODULE': 'xtzx.spiders', 'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko'}2018-04-28 11:08:33 [scrapy.middleware] INFO: Enabled extensions:['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.logstats.LogStats']2018-04-28 11:08:34 [scrapy.middleware] INFO: Enabled downloader middlewares:['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats']2018-04-28 11:08:34 [scrapy.middleware] INFO: Enabled spider middlewares:['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware']2018-04-28 11:08:34 [scrapy.middleware] INFO: Enabled item pipelines:[]2018-04-28 11:08:34 [scrapy.core.engine] INFO: Spider opened ----------好像从这开始出问题 2018-04-28 11:08:34 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)2018-04-28 11:08:34 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:60232018-04-28 11:08:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xuetangx.com/cours...:TsinghuaX+80512073X+2018_T1/about> (referer: None)2018-04-28 11:08:34 [scrapy.core.scraper] ERROR: Spider error processing <GET http://www.xuetangx.com/cours...:TsinghuaX+80512073X+2018_T1/about> (referer: None)Traceback (most recent call last): File "d:python3.5libsite-packagesparselselector.py", line 228, in xpath **kwargs) File "srclxmletree.pyx", line 1577, in lxml.etree._Element.xpath File "srclxmlxpath.pxi", line 307, in lxml.etree.XPathElementEvaluator.__call__ File "srclxmlxpath.pxi", line 227, in lxml.etree._XPathEvaluatorBase._handle_resultlxml.etree.XPathEvalError: Invalid predicate During handling of the above exception, another exception occurred: Traceback (most recent call last): File "d:python3.5libsite-packagestwistedinternetdefer.py", line 653, in _runCallbacks current.result = callback(current.result, *args, **kw) File "E:pythonxtzxxtzxspiderslesson.py", line 16, in parse item["title"]=response.xpath("//div[@class='title_detail'/h3[@class='courseabout_title']/text()").extract() File "d:python3.5libsite-packagesscrapyhttpresponsetext.py", line 119, in xpath return self.selector.xpath(query, **kwargs) File "d:python3.5libsite-packagesparselselector.py", line 232, in xpath six.reraise(ValueError, ValueError(msg), sys.exc_info()[2]) File "d:python3.5libsite-packagessix.py", line 692, in reraise raise value.with_traceback(tb) File "d:python3.5libsite-packagesparselselector.py", line 228, in xpath **kwargs) File "srclxmletree.pyx", line 1577, in lxml.etree._Element.xpath File "srclxmlxpath.pxi", line 307, in lxml.etree.XPathElementEvaluator.__call__ File "srclxmlxpath.pxi", line 227, in lxml.etree._XPathEvaluatorBase._handle_resultValueError: XPath error: Invalid predicate in //div[@class='title_detail'/h3[@class='courseabout_title']/text()2018-04-28 11:08:35 [scrapy.core.engine] INFO: Closing spider (finished)2018-04-28 11:08:35 [scrapy.statscollectors] INFO: Dumping Scrapy stats:{'downloader/request_bytes': 301, 'downloader/request_count': 1, 'downloader/request_method_count/GET': 1, 'downloader/response_bytes': 24409, 'downloader/response_count': 1, 'downloader/response_status_count/200': 1, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2018, 4, 28, 3, 8, 35, 118088), 'log_count/DEBUG': 2, 'log_count/ERROR': 1, 'log_count/INFO': 7, 'response_received_count': 1, 'scheduler/dequeued': 1, 'scheduler/dequeued/memory': 1, 'scheduler/enqueued': 1, 'scheduler/enqueued/memory': 1, 'spider_exceptions/ValueError': 1, 'start_time': datetime.datetime(2018, 4, 28, 3, 8, 34, 418003)}2018-04-28 11:08:35 [scrapy.core.engine] INFO: Spider closed (finished) 感觉程序很简单，但是就是不行，其他items都是常规的设置，pipelines里面没有添加新的内容，然后settings里面就修改了一下ROBOTSTXT_OBEY的值网上查了很久这样的错误，都没找到相应的方法，也试过伪装浏览器爬取也没用，自学，没有老师，完全没辙了，求助各位.

Answer 1

xpath. div[@class='title_detail'这里是否少个]?

item["title"]=response.xpath("//div[@class='title_detail'/h3[@class='courseabout_title']/text()").extract()

Answer 2

File "srclxmlxpath.pxi", line 227, in lxml.etree._XPathEvaluatorBase._handle_result
ValueError: XPath error: Invalid predicate in //div[@class='title_detail'/h3[@class='courseabout_title']/text()

xpath 写错了，少了个]