网页抓取：如何从 html 中提取与与其他标签匹配的关键字相对应的链接，而 url 中没有关键字？

Question

我试图从网页中提取与某些关键字匹配的职位描述并且这有效，但是我也想提取与 HTML 中找到的描述相对应的链接。问题是链接出现在描述的关键字之前，并且 URL 不包含要搜索的关键字。如何提取与通过关键字找到的职位描述相匹配的链接？这是我的代码：import re, requests, time, os, csv, subprocessfrom bs4 import BeautifulSoupdef get_jobs(url):keywords = ["KI", "AI", "Big Data", "Data", "data", "big data", "Analytics", "analytics", "digitalisierung", "ML",            "Machine Learning", "Daten", "Datenexperte", "Datensicherheitsexperte"]headers = {'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36'}html = requests.get(url, headers=headers, timeout=5)time.sleep(2)soup = BeautifulSoup(html.text, 'html.parser')jobs = soup.find_all('p',text=re.compile(r'\b(?:%s)\b' % '|'.join(keywords)))# links = jobs.find_all('a')jobs_found = []for word in jobs:    jobs_found.append(word)with open("jobs.csv", 'a', encoding='utf-8') as toWrite:    writer = csv.writer(toWrite)    writer.writerows(jobs_found)    # subprocess.call('./Autopilot3.py')    print("Matched Jobs have been collected.")get_jobs('https://www.auftrag.at//tenders.aspx')

Answer 1

通过网络我看到链接总是比描述高两个级别。然后你冷使用find_parent()函数来获取a找到的作业的标签。

你的代码中有：

jobs = soup.find_all('p',text=re.compile(r'\b(?:%s)\b' % '|'.join(keywords)))

然后在那之后添加：

for i in jobs:

   print(i.find_parent('a').get('href'))

这将打印链接。请注意，这些链接是相对链接而不是绝对链接。您应该添加根以查找特定页面。例如，如果您发现一个链接是：ETender.aspx?id=ed60009c-8d64-4759-a722-872e21cf9ea7&action=show。您必须在开头添加：https://www.auftrag.at/。作为最后一个链接：https : //www.auftrag.at/ETender.aspx?id=ed60009c-8d64-4759-a722-872e21cf9ea7&action=show

如果需要，您可以像处理职位描述一样将它们添加到列表中。完整代码（不保存在 csv 中）将是：

import re, requests, time, os, csv, subprocess

from bs4 import BeautifulSoup

def get_jobs(url):

    keywords = ["KI", "AI", "Big Data", "Data", "data", "big data", "Analytics", "analytics", "digitalisierung", "ML",

                "Machine Learning", "Daten", "Datenexperte", "Datensicherheitsexperte"]

    headers = {'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36'}

    html = requests.get(url, headers=headers, timeout=5)

    time.sleep(2)

    soup = BeautifulSoup(html.text, 'html.parser')

    jobs = soup.find_all('p',text=re.compile(r'\b(?:%s)\b' % '|'.join(keywords)))

    # links = jobs.find_all('a')

    jobs_found = []

    links = []

    for word in jobs:

        jobs_found.append(word)

        links.append(word.find_parent('a').get('href'))

    with open("jobs.csv", 'a', encoding='utf-8') as toWrite:

        writer = csv.writer(toWrite)

        writer.writerows(jobs_found)

        # subprocess.call('./Autopilot3.py')

        print("Matched Jobs have been collected.")

    return soup, jobs

soup, jobs = get_jobs('https://www.auftrag.at//tenders.aspx')

如果要添加完整的 url，只需更改行：

links.append(word.find_parent('a').get('href'))

到：

links.append("//".join(["//".join(url.split("//")[:2]),word.find_parent('a').get('href')]))