使用 Python 的 requests.post 函数抓取动态 HTML 时如何指定“页码”？

Question

我正在尝试学习如何使用requestsPython 中的库使用动态 HTML 抓取网页。我已经能够获取 HTML，但我不知道如何实际获取不同页面的数据。如果您注意到图像底部有页码：如何指定页码requests.post我如何知道有多少页可用？它会像页面范围然后和一样简单try吗except？我如何知道哪些参数可用于我的data字典requests.post？以下是我访问请求的方式：import requestsr = requests.post('https://search.aap.org/nicu/', data = {'SearchCriteria.Level':'1', 'X-Requested-With':'XMLHttpRequest'}) #key:valueprint(r.text)这是输出：https://pastebin.com/raw/HMBZWR36这是显示页码的部分：# <div id="myPager">#     <div class="pagination-container"><ul class="pagination blue-pager"><li class="blue-pager active"><span>1</span></li></ul></div># </div>

Answer 1

该页面上的数据是使用 JavaScript 动态加载的，因此您不能仅通过该页面的请求来获取它。使用浏览器中的 Developer 标签，您可以拦截生成数据的 xhr 并将其用于请求。这就是本例中的情况。

import requests

headers = {

    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0',

    'Accept': '*/*',

    'Accept-Language': 'en-US,en;q=0.5',

    'X-Requested-With': 'XMLHttpRequest',

    'Connection': 'keep-alive',

    'Referer': 'https://search.aap.org/nicu/',

}

params = (

    ('Model', 'X.PagedList.StaticPagedList`1[AAP.Website.Search.NICU.Web.ViewModels.NicuVm]'),

    ('jsonSearchVmString', '{"SearchCriteria":{"Name":null,"City":null,"State":null,"Zip":null,"Level":2,"LevelAssigner":null,"BedNumberRange":0},"States":null,"Levels":null,"LevelAssigners":null,"BedNumberRangeList":null,"NumberOfPages":0}'),

    ('page', '2'),

    ('class', 'blue-pager'),

    ('X-Requested-With', 'XMLHttpRequest'),

    ('_', '1603318486561'),

)

response = requests.get('https://search.aap.org/nicu/', headers=headers, params=params)

参数之一是('page', '2')；如果您将请求包装在for适当范围内的循环中，您应该能够requests.get()在所有页面上执行此操作。

Answer 2

在这种情况下，您不必使用 POST 请求。如果单击下一页时查看 URL，您将看到可以更改的所有参数。此外，要获取所有页面，您可以解析 HTML 并查找<a rel="next">标签。如果标签不存在，则您位于最后一页（尝试将参数更改"LevelAssigner"为3其他值）：

import requests

from bs4 import BeautifulSoup

url = 'https://search.aap.org/nicu/?Model=X.PagedList.StaticPagedList`1[AAP.Website.Search.NICU.Web.ViewModels.NicuVm]&jsonSearchVmString={"SearchCriteria":{"Name":null,"City":null,"State":null,"Zip":null,"Level":0,"LevelAssigner":"3","BedNumberRange":0},"States":null,"Levels":null,"LevelAssigners":null,"BedNumberRangeList":null,"NumberOfPages":0}&class=blue-pager&X-Requested-With=XMLHttpRequest&page='

page = 1

while True:

    print('Page {}..'.format(page))

    soup = BeautifulSoup(requests.get(url + str(page)).content, 'html.parser')

    for row in soup.select('.row.font-700'):

        print(row.get_text(strip=True, separator=' '))

    next_page = soup.select_one('a[rel="next"]')

    if not next_page:

        break

    page += 1

印刷：

Page 1..

ANDERSON REGIONAL MEDICAL CENTER Meridian, MS 39301

Ann & Robert H. Lurie Children's Hospital of Chicago Chicago, IL 60611

BAYSTATE MEDICAL CENTER Springfield, MA 01199

Beacon Children's Hospital South Bend, IN 46601-1033

BEN TAUB GENERAL HOSPITAL Houston, TX 77030

Page 2..

BETH ISRAEL DEACONESS MEDICAL CENTER Boston, MA 02215

BRIGHAM AND WOMEN'S HOSPITAL Boston, MA 02115

CHILDREN'S HOSPITAL OF NEW ORLEANS New Orleans, LA 70118

CLEVELAND CLINIC formally MERIDIA HILLCREST HOSPITAL Mayfield Heights, OH 44124

EAST JEFFERSON GENERAL HOSPITAL Metairie, LA 70006

Page 3..

GEORGETOWN UNIVERSITY HOSPITAL Washington, DC 20007

GREENWICH HOSPITAL Greenwich, CT 06830

HELEN DEVOS CHILDREN'S HOSPITAL Grand Rapids, MI 49503

INOVA LOUDOUN HOSPITAL Leesburg, VA 20176

JACOBI MEDICAL CENTER Bronx, NY 10461

Page 4..

LAKEVIEW REGIONAL MEDICAL CENTER HOSPITAL Covington, LA 70433

LOYOLA UNIVERSITY MEDICAL CENTER Maywood, IL 60153

MELROSE WAKEFIELD HOSPITAL Melrose, MA 02176

MERCYHEALTH HOSPITAL ROCKTON AVENUE CAMPUS formally ROCKFORD HEALTH SYSTEM Rockford, IL 61103

RUSH FOUNDATION HOSPITAL Meridian, MS 39301

Page 5..

Shetal Shah Valhalla, NY 10595

SLIDELL MEMORIAL HOSPITAL Slidell, LA 70458

SOUTHERN NEW HAMPSHIRE MEDICAL CENTER Nashua, NH 03061

St Mary Medical Center Apple Valley, CA 92307

ST. AGNES HOSPITAL Baltimore, MD 21229

Page 6..

THE CHILDREN'S HOSPITAL AT NAVICENT HEALTH Macon, GA 31208-6000

TOURO INFIRMARY New Orleans, LA 70115

TULANE LAKESIDE HOSPITAL Metairie, LA 70001

UNIVERSITY MEDICAL CENTER -  SUNY STONY BROOK Stony Brook, NY 11794-8111

UNIVERSITY OF IOWA CHILDREN'S HOSPITAL Iowa City, IA 52242

Page 7..

UNIVERSITY OF LOUISVILLE SCHOOL OF MEDICINE Louisville, KY 40232-5070

UNIVERSITY OF MISSISSIPPI MEDICAL CENTER Jackson, MS 39216-4505

UNIVERSITY OF ROCHESTER MEDICAL CENTER Rochester, NY 14642

UNIVERSITY OF WASHINGTON MEDICAL CENTER Seattle, WA 98195

WakeMed Cary Hospital Cary, NC 27518

Page 8..

WakeMed North Family Health & Women’s Hospital Raleigh, NC 27614

WakeMed Raleigh Hospital Raleigh, NC 27610

Wellstar Kennestone Hospital NICU Marietta, GA 30066

WILLIS KNIGHTON CENTER FOR WOMEN'S HEALTH Shreveport, LA 71118

WOMAN'S HOSPITAL Baton Rouge, LA 70817