如何从具有不变 url 的网页中抓取 pdf 链接？

Question

我正在做一个关于网络抓取的项目，我被要求从一个网站抓取所有 pdf 链接：https: //www.sebi.gov.in/sebiweb/home/HomeAction.do ?doListing=yes&sid=3& s。该网站有 397 个页面，但每个页面都有相同的 URL。我尝试了 inspect element 工具，发现 javascript 代码有助于导航到不同的页面。但我仍然无法弄清楚如何为所有页面运行我的脚本。下面是我的代码。from bs4 import BeautifulSoupimport lxmlurl = 'https://www.sebi.gov.in/sebiweb/home/HomeAction.do?doListing=yes&sid=3&s'conn = urllib2.urlopen(url)html = conn.read()soup = BeautifulSoup(html)links = soup.find_all('a')urls=[]for tag in links:    link = tag.get('href',None)    if link is not None and link.endswith('html'):        #urls.append(link)        purl=link        new=urllib2.urlopen(purl)        htm=new.read()        sp=BeautifulSoup(htm)        nl=sp.find_all('a')        nm=sp.find_all('iframe')        for i in nl:            q=i.get('href',None)            title=i.get('title',None)            if q is not None and q.endswith('pdf'):                print(q)                urls.append(q)        for j in nm:            z=j.get('src',None)            title=j.get('title',None)            if z is not None and z.endswith('pdf')and title is not None:                print(z)                print(title)                urls.append(z)        print(len(urls))

Answer 1

您可以使用他们位于 上的 APIhttps://www.sebi.gov.in/sebiweb/ajax/home/getnewslistinfo.jsp来加载数据。

例如：

from bs4 import BeautifulSoup

from requests import get

api_url = 'https://www.sebi.gov.in/sebiweb/ajax/home/getnewslistinfo.jsp'

payload = {

    'nextValue': "1",

    'next': "n",

    'search': "",

    'fromDate': "",

    'toDate': "",

    'fromYear': "",

    'toYear': "",

    'deptId': "",

    'sid': "3",

    'ssid': "-1",

    'smid': "0",

    'intmid': "-1",

    'sText': "Filings",

    'ssText': "-- All Sub Section --",

    'smText': "",

    'doDirect':  "1",

}

page = 0

while True:

    print('Page {}...'.format(page))

    payload['doDirect'] = page

    soup = BeautifulSoup(requests.post(api_url, data=payload).content, 'html.parser')

    rows = soup.select('tr:has(td)')

    if not rows:

        break

    for tr in rows:

        row = [td.get_text(strip=True) for td in tr.select('td')] + [tr.a['href']]

        print(*row, sep='\t')

    page += 1

印刷：

...

Page 1...

Jun 25, 2020    Mindspace Business Parks REIT – Addendum to Draft Prospectus    https://www.sebi.gov.in/filings/reit-issues/jun-2020/mindspace-business-parks-reit-addendum-to-draft-prospectus_46928.html

Jun 25, 2020    Amrit Corp. Ltd. - Public Announcement  https://www.sebi.gov.in/filings/buybacks/jun-2020/amrit-corp-ltd-public-announcement_46927.html

Jun 24, 2020    NIIT Technologies Buyback - Post Buyback - Public Advertisement https://www.sebi.gov.in/filings/buybacks/jun-2020/niit-technologies-buyback-post-buyback-public-advertisement_46923.html

Jun 23, 2020    Addendum to Letter of Offer of Arvind Fashions Limited  https://www.sebi.gov.in/filings/rights-issues/jun-2020/addendum-to-letter-of-offer-of-arvind-fashions-limited_46941.html

Jun 23, 2020    Genesis Exports Limited - Draft letter of Offer https://www.sebi.gov.in/filings/buybacks/jun-2020/genesis-exports-limited-draft-letter-of-offer_46911.html

Jun 23, 2020    Genesis Exports Limited - Public Announcement   https://www.sebi.gov.in/filings/buybacks/jun-2020/genesis-exports-limited-public-announcement_46909.html

Jun 19, 2020    Coral India Finance and Housing Limited – Post Buy-back Public Announcement https://www.sebi.gov.in/filings/buybacks/jun-2020/coral-india-finance-and-housing-limited-post-buy-back-public-announcement_46900.html

Jun 19, 2020    Network Limited https://www.sebi.gov.in/filings/takeovers/jun-2020/network-limited_46890.html

Jun 17, 2020    KSOLVES INDIA LIMITED   https://www.sebi.gov.in/filings/public-issues/jun-2020/ksolves-india-limited_46996.html

Jun 10, 2020    Happiest Minds Technologies Limited https://www.sebi.gov.in/filings/public-issues/jun-2020/happiest-minds-technologies-limited_46843.html

Jun 08, 2020    IM+ Capitals Limited    https://www.sebi.gov.in/filings/takeovers/jun-2020/im-capitals-limited_46786.html

Jun 05, 2020    HealthCare Global Enterprises Limited   https://www.sebi.gov.in/filings/takeovers/jun-2020/healthcare-global-enterprises-limited_46773.html

Jun 02, 2020    Jaikumar Constructions Ltd. - DRHP  https://www.sebi.gov.in/filings/public-issues/jun-2020/jaikumar-constructions-ltd-drhp_46774.html

Jun 02, 2020    Mahindra Focused Equity Yojana  https://www.sebi.gov.in/filings/mutual-funds/jun-2020/mahindra-focused-equity-yojana_46767.html

Jun 02, 2020    GRANULES INDIA LIMITED - Dispatch advertisement https://www.sebi.gov.in/filings/buybacks/jun-2020/granules-india-limited-dispatch-advertisement_46765.html

Jun 02, 2020    GRANULES INDIA LIMITED  - Letter of Offer   https://www.sebi.gov.in/filings/buybacks/jun-2020/granules-india-limited-letter-of-offer_46764.html

Jun 02, 2020    Motilal Oswal Multi Asset Fund  https://www.sebi.gov.in/filings/mutual-funds/jun-2020/motilal-oswal-multi-asset-fund_46762.html

Jun 02, 2020    Principal Large Cap Fund    https://www.sebi.gov.in/filings/mutual-funds/jun-2020/principal-large-cap-fund_46761.html

Jun 02, 2020    Mahindra Arbitrage Yojana   https://www.sebi.gov.in/filings/mutual-funds/jun-2020/mahindra-arbitrage-yojana_46760.html

Jun 02, 2020    HSBC Mid Cap Equity Fund    https://www.sebi.gov.in/filings/mutual-funds/jun-2020/hsbc-mid-cap-equity-fund_46759.html

Jun 01, 2020    Tanla Solutions Limited - DLOF  https://www.sebi.gov.in/filings/buybacks/jun-2020/tanla-solutions-limited-dlof_46750.html

Jun 01, 2020    Axis Banking ETF    https://www.sebi.gov.in/filings/mutual-funds/jun-2020/axis-banking-etf_46748.html

Jun 01, 2020    Kalpataru Power Transmission Limited - Public Announcement  https://www.sebi.gov.in/filings/buybacks/jun-2020/kalpataru-power-transmission-limited-public-announcement_46746.html

Jun 01, 2020    Reliance Industries Limited - Addendum dated May 22, 2020   https://www.sebi.gov.in/filings/rights-issues/jun-2020/reliance-industries-limited-addendum-dated-may-22-2020_46745.html

Jun 01, 2020    Reliance Industries Limited - Addendum dated May 19, 2020   https://www.sebi.gov.in/filings/rights-issues/jun-2020/reliance-industries-limited-addendum-dated-may-19-2020_46744.html

Page 2...

Jun 01, 2020    Reliance Industries Limited - Addendum dated May 18, 2020   https://www.sebi.gov.in/filings/rights-issues/jun-2020/reliance-industries-limited-addendum-dated-may-18-2020_46743.html

May 29, 2020    Muthoottu Mini Financiers Limited- Prospectus   https://www.sebi.gov.in/filings/debt-offer-document/may-2020/muthoottu-mini-financiers-limited-prospectus_46769.html

May 29, 2020    Coral India Housing and Finance Limited - Advertisement https://www.sebi.gov.in/filings/buybacks/may-2020/coral-india-housing-and-finance-limited-advertisement_46732.html

May 29, 2020    TANLA SOLUTIONS LIMITED - Public Announcement   https://www.sebi.gov.in/filings/buybacks/may-2020/tanla-solutions-limited-public-announcement_46731.html

May 28, 2020    Tips Industries Limited - Dispatch Advertisement    https://www.sebi.gov.in/filings/buybacks/may-2020/tips-industries-limited-dispatch-advertisement_46723.html

May 27, 2020    KLM Axiva Finvest Limited - Prospectus  https://www.sebi.gov.in/filings/debt-offer-document/may-2020/klm-axiva-finvest-limited-prospectus_46755.html

May 26, 2020    Tips Industries Limited - Letter of Offer   https://www.sebi.gov.in/filings/buybacks/may-2020/tips-industries-limited-letter-of-offer_46708.html

May 26, 2020    Axis Capital Protection Oriented Fund - Series 7-10 https://www.sebi.gov.in/filings/mutual-funds/may-2020/axis-capital-protection-oriented-fund-series-7-10_46707.html

May 26, 2020    ICICI Prudential Alpha Low Vol 30 ETF   https://www.sebi.gov.in/filings/mutual-funds/may-2020/icici-prudential-alpha-low-vol-30-etf_46706.html

May 22, 2020    NIIT Technologies Ltd. - Letter of Offer    https://www.sebi.gov.in/filings/buybacks/may-2020/niit-technologies-ltd-letter-of-offer_46700.html

May 22, 2020    NIIT Technologies Ltd. - Dispatch Advertisement https://www.sebi.gov.in/filings/buybacks/may-2020/niit-technologies-ltd-dispatch-advertisement_46699.html

May 22, 2020    Coral India Finance and Housing Limited - Letter of Offer   https://www.sebi.gov.in/filings/buybacks/may-2020/coral-india-finance-and-housing-limited-letter-of-offer_46698.html

May 22, 2020    Jay Ushin Limited   https://www.sebi.gov.in/filings/takeovers/may-2020/jay-ushin-limited_46697.html

May 22, 2020    Pennar Industries - Post Buyback Public Announcement    https://www.sebi.gov.in/filings/buybacks/may-2020/pennar-industries-post-buyback-public-announcement_46696.html

May 22, 2020    Axis Global Equity Alpha Fund of Fund.  https://www.sebi.gov.in/filings/mutual-funds/may-2020/axis-global-equity-alpha-fund-of-fund-_46695.html

May 21, 2020    Axis Global Disruption Fund of Fund https://www.sebi.gov.in/filings/mutual-funds/may-2020/axis-global-disruption-fund-of-fund_46694.html

May 18, 2020    Reliance Industries Limited https://www.sebi.gov.in/filings/rights-issues/may-2020/reliance-industries-limited_46675.html

May 14, 2020    Public Advertisement of Spencer's Retail Limited    https://www.sebi.gov.in/filings/rights-issues/may-2020/public-advertisement-of-spencer-s-retail-limited_46693.html

May 12, 2020    Spencer's Retail Limited    https://www.sebi.gov.in/filings/rights-issues/may-2020/spencer-s-retail-limited_46692.html

May 12, 2020    Sequent Scientific Limited  https://www.sebi.gov.in/filings/takeovers/may-2020/sequent-scientific-limited_46662.html

May 11, 2020    Arvind Fashions Limited https://www.sebi.gov.in/filings/rights-issues/may-2020/arvind-fashions-limited_46659.html

May 05, 2020    JK Paper Limited - Public Announcement  https://www.sebi.gov.in/filings/buybacks/may-2020/jk-paper-limited-public-announcement_46647.html

May 05, 2020    Aurionpro Solutions Limited - Post BuyBack Advertisement    https://www.sebi.gov.in/filings/buybacks/may-2020/aurionpro-solutions-limited-post-buyback-advertisement_46646.html

May 04, 2020    KSOLVES INDIA LIMITED   https://www.sebi.gov.in/filings/public-issues/may-2020/ksolves-india-limited_46644.html

May 04, 2020    SBI ETF Consumption https://www.sebi.gov.in/filings/mutual-funds/may-2020/sbi-etf-consumption_46639.html

Page 3...

... and so on.

Answer 2

该网站似乎正在向 POST 请求getnewslistinfo.jsp并将新表格内容作为 html 取回。您可以打开您的网络（在 Firefox 上为 Ctrl+Shift+E），然后导航到下一页并查看发出的请求及其参数。

您可以模仿该 POST 请求并使用urllib2（或者最好是requests ）更改下一页的适当参数（从我看到的应该是nextValueand ） 。获取内容后，您可以简单地使用 BeautifulSoup 解析它并按照您已经执行的方式提取 a 标签。doDirect

另外给你一个提示：你应该把你的代码分成不同的函数来做不同的事情，比如getPage(pageNum)给定一个页码返回 html 内容，getLinks(html)给定一个 html 页面它从表中获取所有链接并将它们作为列表返回。这样您的代码将更具可读性并且更易于调试和使用。