Web 抓取 - 挑战在我的代码中阐明层次结构

Question

目标：我正在尝试抓取 100 多个网页，特别是每个网页的配方成分。如果我们举一个例子——其中包含鸡蛋三明治 ( url )的食谱，我为此使用了许多 Python 依赖项，包括BeautifulSoup, splinter.Browser, ChromeDrivermanager.预期输出：一旦我收集了成分，我想将它们保存在字典中。下面的例子 -recipes = {"quick_and_easy_egg_salad_sandwich_recipe":['1-2 tablespoons mayonnaise (to taste)', '2 tablespoons chopped celery', '2 slices white, wheat, multigrain, or rye bread, toasted or plain']我取得的成就：1. 我已经能够“粗略地”确定（通过 Web Inspector）我需要关注什么—— 看起来每种成分都有它自己的，但看起来我要么误解了层次结构，要么误解了我的代码是不正确的。<li class='ingredient'>2.我的代码如下-executable_path = {'executable_path': ChromeDriverManager().install()}browser = Browser('chrome', **executable_path)webpage_url = 'https://www.simplyrecipes.com/recipes/egg_salad_sandwich/'browser.visit(webpage_url)time.sleep(1)website_html = browser.htmlwebsite_soup = BeautifulSoup(website_html, 'html.parser')ingredients = website_soup.find('h3', class_="Ingredients")ingredientsList = ingredients.find('li', class_ = "ingredient")print({ingredients})当我尝试打印时，{ingredients}我得到一个AttributeError: 'NoneType' object has no attribute 'find'我知道我的代码有缺陷的消息，但是我只是不知道如何解决这个问题，想知道是否有人有任何建议？

Answer 1

尝试这个，

import requests

from bs4 import BeautifulSoup

resp = requests.get("https://www.simplyrecipes.com/recipes/egg_salad_sandwich/")

soup = BeautifulSoup(resp.text, "html.parser")

div_ = soup.find("div", attrs={"class": "recipe-callout"})

recipes = {"_".join(div_.find("h2").text.split()):

               [x.text for x in div_.findAll("li", attrs={"class": "ingredient"})]}

Answer 2

听起来你的代码应该在下面，在我删除了不必要的h3检索之后

executable_path = {'executable_path': ChromeDriverManager().install()}

browser = Browser('chrome', **executable_path)

webpage_url = 'https://www.simplyrecipes.com/recipes/egg_salad_sandwich/'

browser.visit(webpage_url)

time.sleep(1)

website_html = browser.html

website_soup = BeautifulSoup(website_html, 'html.parser')

ingredientsList = website_soup.find('li', class_ = "ingredient")

print({ingredients})

您正在尝试查找具有不存在的类名的h3元素Ingredients