如何从网页的 JSON/Javascript 中抓取数据？

Question

我是 Python 新手，今天就开始吧。
我的系统环境是Python 3.5，在Windows10 上有一些库。

我想从以下站点提取足球运动员数据作为 CSV 文件。

问题：我无法将soup.find_all('script')[17] 中的数据提取为预期的 CSV 格式。如何根据需要提取这些数据？

我的代码如下所示。

from bs4 import BeautifulSoup
import re
from urllib.request import Request, urlopen

req = Request('http://www.futhead.com/squad-building-challenges/squads/343', headers={'User-Agent': 'Mozilla/5.0'})
webpage = urlopen(req).read()
soup = BeautifulSoup(webpage,'html.parser') #not sure if i need to use lxml
soup.find_all('script')[17] #My target data is in 17th

我的预期输出将与此类似

position,slot_position,slug
ST,ST,paulo-henrique
LM,LM,mugdat-celik

Answer 1

正如@josiah Swain 所说，它不会很漂亮。对于这种事情，更推荐使用JS，因为它可以理解你所拥有的。

话说，python 很棒，这就是你的解决方案！

#Same imports as before
from bs4 import BeautifulSoup
import re
from urllib.request import Request, urlopen

#And one more
import json

# The code you had 
req = Request('http://www.futhead.com/squad-building-challenges/squads/343',
               headers={'User-Agent': 'Mozilla/5.0'})
webpage = urlopen(req).read()
soup = BeautifulSoup(webpage,'html.parser')

# Store the script 
script = soup.find_all('script')[17]

# Extract the oneline that stores all that JSON
uncleanJson = [line for line in script.text.split('\n') 
         if line.lstrip().startswith('squad.register_players($.parseJSON') ][0]

# The easiest way to strip away all that yucky JS to get to the JSON
cleanJSON = uncleanJson.lstrip() \
                       .replace('squad.register_players($.parseJSON(\'', '') \
                       .replace('\'));','')

# Extract out that useful info
data = [ [p['position'],p['data']['slot_position'],p['data']['slug']] 
         for p in json.loads(cleanJSON)
         if p['player'] is not None]


print('position,slot_position,slug')
for line in data:
    print(','.join(line))

我将其复制并粘贴到 python 中的结果是：

position,slot_position,slug
ST,ST,paulo-henrique
LM,LM,mugdat-celik
CAM,CAM,soner-aydogdu
RM,RM,petar-grbic
GK,GK,fatih-ozturk
CDM,CDM,eray-ataseven
LB,LB,kadir-keles
CB,CB,caner-osmanpasa
CB,CB,mustafa-yumlu
RM,RM,ioan-adrian-hora
GK,GK,bora-kork

---

编辑：经过反思，这对于初学者来说并不是最容易阅读的代码。这是一个更容易阅读的版本

# ... All that previous code 
script = soup.find_all('script')[17]

allScriptLines = script.text.split('\n')

uncleanJson = None
for line in allScriptLines:
     # Remove left whitespace (makes it easier to parse)
     cleaner_line = line.lstrip()
     if cleaner_line.startswith('squad.register_players($.parseJSON'):
          uncleanJson = cleaner_line

cleanJSON = uncleanJson.replace('squad.register_players($.parseJSON(\'', '').replace('\'));','')

print('position,slot_position,slug')
for player in json.loads(cleanJSON):
     if player['player'] is not None:
         print(player['position'],player['data']['slot_position'],player['data']['slug']) 

Answer 2

所以我的理解是，beautifulsoup 更适合 HTML 解析，但您正在尝试解析嵌套在 HTML 中的 javascript。

所以你有两个选择

1. 只需创建一个函数，获取 soup.find_all('script')[17] 的结果，循环并手动搜索字符串以查找数据并提取它。您甚至可以使用 ast.literal_eval(string_thats_really_a_dictionary) 让它变得更容易。这可能不是最好的方法，但如果您是 python 新手，您可能希望这样做只是为了练习。
2. Use the json library like in this example. 或 alternatively like this way. 这可能是更好的方法。