soup 对象为none的问题
craw 350 : https://baike.baidu.com/item/%E6%A2%85%E7%88%B1%E8%8A%B3/983014
craw 351 : https://baike.baidu.com/item/%E5%93%88%E5%A8%81%C2%B7%E5%87%AF%E7%89%B9%E5%B0%94/4528376
Traceback (most recent call last):
File "c:/Users/VULCAN/Desktop/文件/爬虫学习.py", line 127, in <module>
obj_spider.craw(root_url)
File "c:/Users/VULCAN/Desktop/文件/爬虫学习.py", line 114, in craw
new_urls, new_data = self.parser.parse(new_url, html_cont)
File "c:/Users/VULCAN/Desktop/文件/爬虫学习.py", line 69, in parse
new_data = self._get_new_data(page_url, soup)
File "c:/Users/VULCAN/Desktop/文件/爬虫学习.py", line 57, in _get_new_data
title_node = soup.find('dd', class_="lemmaWgt-lemmaTitle-title").find("h1")
AttributeError: 'NoneType' object has no attribute 'find'
爬取到一半后报错,相关代码如下
def parse(self, page_url, html_cont): if page_url is None or html_cont is None: return soup = BeautifulSoup(html_cont, 'html.parser', from_encoding='utf-8') new_urls = self._get_new_urls(page_url, soup) new_data = self._get_new_data(page_url, soup) return new_urls, new_data
求大神解答!!