使用 BeautifulSoup 和 pandas 将列表项内标题下方的文本抓取到列中

Question

我正在尝试使用 BeautifulSoup 和 pandas 来抓取和存储一些项目。下面的代码仅部分有效。正如您所看到的，它刮掉了“Engine426/425 HP”，而我只希望将字符串“426/425 HP”存储在“engine”列中。我想抓取下面 HTML 中的所有 4 个 h5 字符串（请参阅下面所需的输出）。我希望有人能帮助我，谢谢！import numpy as npimport pandas as pdfrom bs4 import BeautifulSoupimport requestsimport remain_url = "https://www.example.com/"def getAndParseURL(url):    result = requests.get(url)    soup = BeautifulSoup(result.text, 'html.parser')    return(soup)soup = getAndParseURL(main_url)engine = []engine.append(soup.find("ul", class_ = re.compile('list-inline lot-breakdown-list')).li.text)scraped_data = pd.DataFrame({'engine': engine})scraped_data.head()              engine0   Engine426/425 HP超文本标记语言<div class="lot-breakdown">    <ul class="list-inline lot-breakdown-list">        <li>            <h5>Engine</h5>426/425 HP</li>        <li>            <h5>Trans</h5>Automatic</li>        <li>            <h5>Color</h5>Alpine White</li>        <li>            <h5>Interior</h5>Black</li>    </ul></div>所需输出scraped_data[['engine', 'trans', 'color', 'interior']] = pd.DataFrame([['426/425 HP', 'Automatic', 'Alpine White', 'Black']], index=scraped_data.index)scraped_data              engine        trans          color  interior0         426/425 HP    Automatic   Alpine White     Black

Answer 1

您可以通过多种方式实现这一目标：

    from bs4 import BeautifulSoup , NavigableString

    import requests

    main_url = "https://www.example.com/"

    def getAndParseURL(url):

        result = requests.get(url)

        soup = BeautifulSoup(result.text, 'html.parser')

        return(soup)

    soup = getAndParseURL(main_url)

    #ul   = soup.select('ul[class="list-inline lot-breakdown-list"] li')

    #for li in ul :

         #x = li.find(text=True, recursive=False) # Will give you the text of the li skipping the text of child tag

         #y = ' '.join([t for t in li.contents if type(t)== NavigableString]) # contents [<h5>Engine</h5>, '426/425 HP'] the text you want has a type of NavigableString and That's what we are returning .

    ul = soup.select('ul[class="list-inline lot-breakdown-list"] li', recursive=True)

    lis_e = []

    for li in ul:

        lis = []

        lis.append(li.contents[1])

        lis_e.extend(lis)

    engine.append(lis_e[0])

    trans.append(lis_e[1])

    color.append(lis_e[2])

    interior.append(lis_e[3])

    scraped_data = pd.DataFrame({'engine': engine, 'transmission': trans, 'color': color, 'interior': interior})

    scraped_data