Soup.find_all 返回一个空列表

Question

requests我正在尝试使用和来抓取 NBA 统计数据的球员统计表BeautifulSoup，但我得到的响应与我使用“检查元素”看到的不同包含div该表的类属性为：class="nba-stat-table__overflow。但是，每当我运行以下代码时，我都会得到一个空列表：table = soup.find_all('div',attrs={'class="nba-stat-table__overflow'})这是我的完整代码：import osimport pandas as pdimport numpy as npfrom bs4 import BeautifulSoupimport requestsurl = 'https://stats.nba.com/players/boxscores/?Season=2018-19&SeasonType=Regular%20Season'response = requests.get(url)soup = BeautifulSoup(response.content,'html.parser')table = soup.find_all('div',attrs={'class="nba-stat-table__overflow'})

Answer 1

基本上页面是通过加载的JavaScript，所以bs4或requests模块将无法JavaScript动态渲染。

你应该使用selenium或requests_html模块来渲染JS，但我注意到网站正在使用API，它可以用来获取数据，所以我调用了它并提取了数据。

import requests

import pandas as pd

params = {

    "Counter": "1000",

    "DateFrom": "",

    "DateTo": "",

    "Direction": "DESC",

    "LeagueID": "00",

    "PlayerOrTeam": "P",

    "Season": "2018-19",

    "SeasonType": "Regular Season",

    "Sorter": "DATE"

}

headers = {

    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0',

    "x-nba-stats-origin": "stats",

    "x-nba-stats-token": "true",

    "Referer": "https://stats.nba.com/players/boxscores/?Season=2018-19&SeasonType=Regular%20Season"

}

def main(url):

    r = requests.get(url, params=params, headers=headers).json()

    goal = []

    for item in r['resultSets']:

        df = pd.DataFrame(item['rowSet'], columns=item['headers'])

        goal.append(df)

    new = pd.concat(goal)

    print(new)

    new.to_csv("data.csv", index=False)

main("https://stats.nba.com/stats/leaguegamelog")

输出：在线查看