【发布时间】：2021-12-06 22:09:11
【问题描述】：
我对网络抓取还很陌生，如果我的问题的答案很明显，我深表歉意。我制作了一个 Web Scraper，它可以查看 Steam 游戏（文明 6）的评论，并获取诸如在游戏上花费的时间、他们是否推荐、他们拥有的产品等信息。
import pandas as pd
import requests
from bs4 import BeautifulSoup as bs
url = "https://steamcommunity.com/app/289070/reviews/?browsefilter=toprated&snr=1_5_100010_"
review_dict = {
"found_helpful": [],
"title": [], #recommended or not
"hours": [],
"prods_in_account": [],
"words_in_review": []
}
def data_scrapper():
"""
get's the reviews from the steam page.
"""
response = requests.get(url)
soup = bs(response.content, "html.parser")
card_div = soup.findAll("div",attrs={"class","apphub_Card modalContentLink interactable"})
for cards in card_div:
found_helpful = cards.find("div", attrs={"class": "found_helpful"})
vote_header = cards.find("div", attrs={"class": "vote_header"})
hours = cards.find("div", attrs={"class": "hours"})
products = cards.find("div", attrs={"class": "apphub_CardContentMoreLink ellipsis"})
words_in_review = cards.find("div", attrs={"class": "apphub_CardTextContent"})
review_dict["found_helpful"].append(found_helpful)
review_dict["title"].append(vote_header)
review_dict["hours"].append(hours)
review_dict["prods_in_account"].append(products)
review_dict["words_in_review"].append(len(words_in_review))
data_scrapper()
review_df = pd.DataFrame.from_dict(review_dict)
review_df.to_csv("review.csv", sep=",")
我的问题是，当我运行我的代码时，我期待一个有组织的 CSV 文件，但是我得到了这个：
,found_helpful,title,hours,prods_in_account,words_in_review
0,"<div class=""found_helpful"">
3,398 people found this review helpful<br/>159 people found this review funny <div class=""review_award_aggregated tooltip"" data-tooltip-class=""review_reward_tooltip"" data-tooltip-html='<div class=""review_award_ctn_hover""> <div class=""review_award"" data-reaction=""6"" data-reactioncount=""5"">
<img class=""review_award_icon tooltip"" src=""https://store.akamai.steamstatic.com/public/images/loyalty/reactions/still/6.png?v=5""/>
<span class=""review_award_count "">5</span>
</div>
<div class=""review_award"" data-reaction=""3"" data-reactioncount=""3"">
<img class=""review_award_icon tooltip"" src=""https://store.akamai.steamstatic.com/public/images/loyalty/reactions/still/3.png?v=5""/>
<span class=""review_award_count "">3</span>
</div>
<div class=""review_award"" data-reaction=""5"" data-reactioncount=""2"">
<img class=""review_award_icon tooltip"" src=""https://store.akamai.steamstatic.com/public/images/loyalty/reactions/still/5.png?v=5""/>
<span class=""review_award_count "">2</span>
</div>
<div class=""review_award"" data-reaction=""1"" data-reactioncount=""1"">
<img class=""review_award_icon tooltip"" src=""https://store.akamai.steamstatic.com/public/images/loyalty/reactions/still/1.png?v=5""/>
<span class=""review_award_count hidden"">1</span>
</div>
<div class=""review_award"" data-reaction=""9"" data-reactioncount=""1"">
<img class=""review_award_icon tooltip"" src=""https://store.akamai.steamstatic.com/public/images/loyalty/reactions/still/9.png?v=5""/>
<span class=""review_award_count hidden"">1</span>
</div>
<div class=""review_award"" data-reaction=""18"" data-reactioncount=""1"">
<img class=""review_award_icon tooltip"" src=""https://store.akamai.steamstatic.com/public/images/loyalty/reactions/still/18.png?v=5""/>
<span class=""review_award_count hidden"">1</span>
</div>
<div class=""review_award"" data-reaction=""19"" data-reactioncount=""1"">
<img class=""review_award_icon tooltip"" src=""https://store.akamai.steamstatic.com/public/images/loyalty/reactions/still/19.png?v=5""/>
<span class=""review_award_count hidden"">1</span>
</div>
</div>'><img class=""reward_btn_icon"" src=""https://community.akamai.steamstatic.com/public/shared/images//award_icon_blue.svg""/>14</div>
</div>","<div class=""vote_header"">
<div class=""reviewInfo"">
<div class=""thumb"">
<img height=""44"" src=""https://community.akamai.steamstatic.com/public/shared/images/userreviews/icon_thumbsDown.png?v=1"" width=""44""/>
</div>
<div class=""title"">Not Recommended</div>
<div class=""hours"">8,028.3 hrs on record</div>
</div>
<div style=""clear: left""></div>
</div>","<div class=""hours"">8,028.3 hrs on record</div>","<div class=""apphub_CardContentMoreLink ellipsis"">167 products in account</div>",38
我修改了用于提取和附加数据的函数，但我仍然得到这个奇怪的文件，任何关于我做错了什么的线索？
【问题讨论】：
-
如您所见，
found_helpful包含整个<div>标签。您想从该标记中提取文本，该标记位于found_helpful.text。
标签： python pandas csv web-scraping beautifulsoup