用 Beautiful Soup 解析 html 文档答案

【问题标题】：Parsing html document with Beautiful Soup用 Beautiful Soup 解析 html 文档
【发布时间】：2019-07-03 06:46:38
【问题描述】：

我正在尝试使用漂亮的汤来解析一个 html 页面。具体来说，我正在查看这个名为“g_rgTopCurators”的非常大的数组，可以总结如下：

g_rgTopCurators = 

[{\"curator_description\":\"Awesome and sometimes overlooked indie games 
curated by the orlygift.com team\",
\"last_curation_date\":1538400354,
\"discussion_url\":null,
\"rgTagLineLocalizations\":[],
\"broadcasters\":[],
\"broadcasters_info_available\":1,
\"bFollowed\":null,
\"m_rgAppRecommendations\":
    [{  \"appid\":495600,
        \"clanid\":9254464,
        \"link_url\":\"https:\\\/\\\/www.orlygift.com\\\/games\\\/asteroid-fight\",
        \"link_text\":\"\",
        \"blurb\":\"Overall, we found Asteroid Fight to be a cool space game. If you want to manage a base and also handle asteroids, this is the right game for you. It\\u2019s definitely fun, unique and it has its own twist.\",
        \"time_recommended\":1538400354,
        \"comment_count\":0,
        \"upvote_count\":0,
        \"accountid_creator\":10142231,
        \"recommendation_state\":0,
        \"received_compensation\":0,
        \"received_for_free\":1},

        {other app with same params as above}, 
        {other app},
        {other app}
    ],

\"m_rgCreatedApps\":[],
\"m_strCreatorVanityURL\":\"\",
\"m_nCreatorPartnerID\":0,
\"clanID\":\"9254464\",
\"name\":\"Orlygift\",
\"communityLink\":\"https:\\\/\\\/steamcommunity.com\\\/groups\\\/orlygift\",
\"strAvatarHash\":\"839146c7ccac8ee3646059e3af616cb7691e1440\",
\"link\":\"https:\\\/\\\/store.steampowered.com\\\/curator\\\/9254464-Orlygift\\\/\",
\"youtube\":null,
\"facebook_page\":null,
\"twitch\":null,
\"twitter\":null,
\"total_reviews\":50,
\"total_followers\":38665,
\"total_recommended\":50,
\"total_not_recommended\":0,
\"total_informative\":0
},
{another curator},
{another curator}
];

我正在尝试弄清楚如何正确使用soup.select() 来获取这个大型数组中每个策展人的每个“名称”。

soup = bs4.BeautifulSoup(data["results_html"], "html.parser")
curators = soup.select(" ??? ")

【问题讨论】：

该数据结构真的在html 中吗？它看起来更像是一个字典列表，或JSON。为什么要使用 BeautifulSoup？可能更容易使用json.loads()。然后你可以做 import json data = json.loads(g_rgTopCurators) print([d['name'] for d in data]) 将返回所有 name
@davedwards 整体结构是一个 json 对象，但包含包含这个大数组的 html。所以，我认为你的方法行不通。
@jimbob542，实际上 davedwards 确实有正确的方法。但我也明白你的意思。基本上，beautifulsoup 将用于解析出这个 json 对象，然后您将解析 json 对象。您能否提供完整的 html，以便我可以看到如何找到 g_rgTopCurators。或者提供拉动 html 请求的代码将是有益的
@chitown88 store.steampowered.com/curators/ajaxgetcurators/…

标签： python beautifulsoup

【解决方案1】：

由于响应是包含 HTML 的 JSON，其中包含包含更多 JSON 的脚本元素，我的第一个方法是：

import requests
import json
from bs4 import BeautifulSoup

url="https://store.steampowered.com/curators/ajaxgetcurators/render?start=0&count=50"
response = requests.get(url, headers = {"Accept": "application/json"})
loaded_response = response.json() # Get the JSON response containing the HTML containing the required JSON.
results_html = loaded_response['results_html'] # Get the HTML from the JSON
soup = BeautifulSoup(results_html, 'html.parser')
text = soup.find_all('script')[1].text # Get the script element from the HTML.
# Get the JSON in the HTML script element
jn = json.loads(text[text.index("var g_rgTopCurators = ")+ len("var g_rgTopCurators = "):text.index("var fnCreateCapsule")].strip().rstrip(';'))
for i in jn:  # Iterate through JSON
    print (i['name'])

输出：

Cynical Brit Gaming
PC Gamer
Just Good PC Games
...

WGN Chat
Bloody Disgusting Official
Orlygift

有一种更快的方法，只需将响应作为字节解码并转义，然后通过字符串操作直接转到所需的 JSON：

import requests
import json

url="https://store.steampowered.com/curators/ajaxgetcurators/render?start=0&count=50"
response = requests.get(url, headers = {"Accept": "application/json"})
text = response.content.decode("unicode_escape") # response body as bytes decode and escape
# find the JSON
jn = json.loads(text[text.index("var g_rgTopCurators = ")+ len("var g_rgTopCurators = "):text.index("var fnCreateCapsule")].strip().rstrip(';'))
for i in jn:  # Iterate through JSON
    print (i['name'])

【讨论】：