JSONdata使用beautifulsoup解析并输出错误

Question

当我运行以下代码时会产生以下错误：

import requests
import json
from bs4 import BeautifulSoup

JSONDATA = requests.request("GET", "https://thisiscriminal.com/wp-json/criminal/v1/episodes?posts=1000000&page=1")
JSONDATA = JSONDATA.json()

for line in JSONDATA['posts']:
    soup = BeautifulSoup(line['episodeNumber'])
    soup = BeautifulSoup(line['title'])
    soup = BeautifulSoup(line['audioSource'])
    soup = BeautifulSoup(line['large'])
    soup = BeautifulSoup(line['long'])
    print soup.prettify()

产生了以下错误（我已经尝试了各种关于它建议的关于 LXML 的变体）：

• LXML 问题
• 关于不喜欢 .mp3 链接的问题，但这应该不是问题，因为此链接是正确的？
• 在查找“大”缩略图时遇到问题，但使用标题、audioSource 等的等效字段不会产生相同的错误，但查看网站数据是正确的框？

输出错误

python ./test2.py
./test2.py:14: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

The code that caused this warning is on line 14 of the file ./test2.py. To get rid of this warning, pass the additional argument 'features="lxml"' to the BeautifulSoup constructor.

  soup = BeautifulSoup("features=lxml")(line['episodeNumber'])
./test2.py:16: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

The code that caused this warning is on line 16 of the file     ./test2.py. To get rid of this warning, pass the additional argument 'features="lxml"' to the BeautifulSoup constructor.

  soup = BeautifulSoup(line['title'])
./test2.py:18: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

The code that caused this warning is on line 18 of the file ./test2.py. To get rid of this warning, pass the additional argument 'features="lxml"' to the BeautifulSoup constructor.

  soup = BeautifulSoup(line['audioSource'])

/home/leo/.local/lib/python2.7/site-packages/bs4/init.py:335:
用户警告：
“https://dts.podtrac.com/redirect.mp3/dovetail.prxu.org/criminal/85cd4e4d-fa8b-4df2-8a8c-78ad0e800574/Episode_116_190504_audition_mix_neg18_part_1.mp3”看起来像一个 URL。 Beautiful Soup 不是 HTTP 客户端。您可能应该使用类似请求的 HTTP 客户端来获取 URL 后面的文档，并将该文档提供给 Beautiful Soup。 '那份给美丽汤的文件。 % 解码标记 回溯（最近一次通话最后）： 文件“./test2.py”，第 20 行，在 汤 = BeautifulSoup(line['large']) KeyError：“大”

Answer 1

如果您只是尝试获取 json 中的数据，这将起作用。

import pandas as pd

import requests
import json
from bs4 import BeautifulSoup

JSONDATA = requests.request("GET", "https://thisiscriminal.com/wp-json/criminal/v1/episodes?posts=1000000&page=1")
JSONDATA = JSONDATA.json()

#loads the Json in a dataframe
df = pd.io.json.json_normalize(JSONDATA['posts'])
df.to_csv('posts.csv')

lxml 问题将通过以下方式解决： BeautifulSoup(line['episodeNumber'],'lxml') 这是因为 BeautifulSoup 需要一个 html 解析器来制作一个汤对象。 如果你没有 lxml 做。

pip install lxml

第二个警告是关于您传递一个 url 来创建不起作用的汤对象，因为正如警告所说，它不知道如何下载链接。

最后你的最后一个错误是由于链接 json 没有名为“大”的键

你需要一个异常块。