为什么我无法使用 Beautifulsoup 抓取此 HTML 的末尾？答案

【问题标题】：Why am I not able to scrape till end of this HTML using Beautifulsoup?为什么我无法使用 Beautifulsoup 抓取此 HTML 的末尾？
【发布时间】：2021-07-17 14:40:53
【问题描述】：

我想刮掉这个 reddit 页面中的所有 cmets，然后将数据写入 csv 文件。我已经写了这段代码。

我注意到所有 cmets 都是具有“RichTextJSON-root”类的“div”元素。

代码编写：

from bs4 import BeautifulSoup
import requests
import csv

# Reddit

source = requests.get(
    'https://www.reddit.com/r/sysadmin/comments/gjkqvj/who_has_made_the_switch_from_dell_to_lenovo/').text

soup = BeautifulSoup(source, 'html.parser')

countreddit = 0

csv_file = open('reddit_scrape.csv', 'w')
writer = csv.writer(csv_file)
writer.writerow(['Comment'])

comments = soup.find_all('div', class_='RichTextJSON-root')

for comment in comments:
    for para in comment.find_all('p'):
        paratext = para.text
        writer.writerow([paratext])
        countreddit += 1

print(f'Reddit Count: {countreddit}')
csv_file.close()

reddit 帖子链接：https://www.reddit.com/r/sysadmin/comments/gjkqvj/who_has_made_the_switch_from_dell_to_lenovo/

但是，我只设法刮到网页的中间，停在评论“那些使用 WD15 扩展坞的人想和你谈谈”。请问如何才能刮到本页末尾？

我在其他帖子上读到可能是因为beautifulsoup抓取时，它只抓取网页上呈现的内容，此时网页尚未完全呈现。

谢谢！

【问题讨论】：

请发布您的代码而不是图片。
你应该使用 Reddit API！

标签： python web-scraping beautifulsoup

【解决方案1】：

您可以使用 reddit JSON 功能 - 以 json 格式加载页面（只需在 URL 末尾添加 .json）：

import json
import requests

url = "https://old.reddit.com/r/sysadmin/comments/gjkqvj/who_has_made_the_switch_from_dell_to_lenovo/.json"
headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0"
}
data = requests.get(url, headers=headers).json()

#uncomment this to print all data:
#print(json.dumps(data, indent=4))

def get_comments(d):
    if isinstance(d, dict):
        for k, v in d.items():
            if k == "body":
                yield v
            else:
                yield from get_comments(v)
    elif isinstance(d, list):
        for v in d:
            yield from get_comments(v)


for c in get_comments(data):
    print(c)
    print("-" * 80)

打印：

And all that sucked was the clunkpad. Replace with a 50 series trackpad and you’re good to go.
--------------------------------------------------------------------------------
except the 4th-generation core series. Those models don't exist.
--------------------------------------------------------------------------------
They've been good...forever
--------------------------------------------------------------------------------
Well, a little bit. But of course, there is always a room at the bottom (looking at a half eaten Apple).
--------------------------------------------------------------------------------
Yes, they sure did, enough people complained and they learned their lesson. At least they listen, they *could* be Apple. ;)
--------------------------------------------------------------------------------
Well, they did with the 440/540 series when they slashed all Thinkpad features (bento box, leds, Thinklight, replacing the mousepad with this stupid "glass touchpad") away to make those laptops look more "mainstream". A few series after they had to bring back a few features because they went too far.
--------------------------------------------------------------------------------

...

【讨论】：

喜欢您递归搜索密钥的方式 - JSON 中的 body。

【解决方案2】：

此 URL 将以 JSON 格式为您提供所有详细信息（也包括 cmets）。

https://www.reddit.com/r/sysadmin/comments/gjkqvj.json?

我认为您可以解析该 JSON 文件并提取 cmets。

import requests
url = 'https://www.reddit.com/r/sysadmin/comments/gjkqvj.json?'

resp = requests.get(url)
json_string = resp.json()

【讨论】：