Scrapping json网页

Question

我对网络抓取非常陌生，并且在从nba.com抓取一些NBA球员数据时遇到了一些麻烦。我首先试图用bs4来刮页面，但碰到一个问题，经过一些研究后，我认为这是由于我阅读的文章中的“XHR”。我能够找到json格式数据的网址，但我的python程序似乎陷入了困境，并且从未加载数据。再次，我在网络抓取方面很新颖，但是我想我会看看我是否在这里偏离轨道......有什么建议吗？谢谢！ （下面的代码）

import requests 
import json 

url = "http://stats.nba.com/stats/leaguedashplayerstats?College=&Conference=&Country=&DateFrom=&DateTo=&Division=&DraftPick=&DraftYear=&GameScope=&GameSegment=&Height=&LastNGames=0&LeagueID=00&Location=&MeasureType=Base&Month=0&OpponentTeamID=0&Outcome=&PORound=0&PaceAdjust=N&PerMode=PerGame&Period=0&PlayerExperience=&PlayerPosition=&PlusMinus=N&Rank=N&Season=2017-18&SeasonSegment=&SeasonType=Regular+Season&ShotClockRange=&StarterBench=&TeamID=0&VsConference=&VsDivision=&Weight=" 

resp = requests.get(url=url) 
data = json.loads(resp.text) 
print(data) 

Answer 1

给这个一杆。它将根据我定义的标题生成该页面中的所有类别。顺便说一句，你没有得到第一个与你的初始尝试的反应，因为网页预计在您的请求User-Agent，以确保请求不是来自机器人，而是来自任何真正的浏览器。但是，我伪造了它并找到了答案。

import requests 

url = "http://stats.nba.com/stats/leaguedashplayerstats?College=&Conference=&Country=&DateFrom=&DateTo=&Division=&DraftPick=&DraftYear=&GameScope=&GameSegment=&Height=&LastNGames=0&LeagueID=00&Location=&MeasureType=Base&Month=0&OpponentTeamID=0&Outcome=&PORound=0&PaceAdjust=N&PerMode=PerGame&Period=0&PlayerExperience=&PlayerPosition=&PlusMinus=N&Rank=N&Season=2017-18&SeasonSegment=&SeasonType=Regular+Season&ShotClockRange=&StarterBench=&TeamID=0&VsConference=&VsDivision=&Weight=" 
resp = requests.get(url,headers={'User-Agent':'Mozilla/5.0'}) 
data = resp.json() 

storage = data['resultSets'] 
for elem in storage: 
    all_list = elem['rowSet'] 

    for item in all_list: 
     Player_Id = item[0] 
     Player_name = item[1] 
     Team_Id = item[2] 
     Team_abbr = item[3] 
     print("Player_Id: {} Player_name: {} Team_Id: {} Team_abbr: {}".format(
      Player_Id,Player_name,Team_Id,Team_abbr)) 

Answer 2

就意识到，这是因为用户代理头是不同的......一旦这些被添加它的工作原理