【发布时间】：2021-05-30 23:23:46
【问题描述】：
我从“数据科学的实用网络抓取”开始练习网络抓取作为新人。当我回溯时，我遇到了“JSONDecodeError：期望值：第 1 行第 1 列（字符 0）”我从一开始就有问题。如果有人帮助我，那将对我很有帮助。
# Required packages
import requests
import json
import re
from bs4 import BeautifulSoup as bs
import dataset
# Creating Dataset into Mongodb / SQLite
db = dataset.connect('sqlite:/// reviews.db')
review_url = 'https://www.amazon.com/ss/customer-reviews/ajax/reviews/get/'
product_id = '1449355730'
session = requests.Session()
session.headers.update({
'User-Agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36' +
'(KHTML, like Gecko) Chrome/ 62.0.0.3202.62 Safari/537.36'
})
session.get('https://www.amazon.com/product-reviews/{}/'.format(product_id))
def parse_reviews(reply):
reviews = []
for fragment in reply.split('&&&'):
if not fragment.strip():
continue
json_fragment = json.loads(fragment)
if json_fragment[0] != 'append':
continue
html_soup = bs(json_fragment[2], 'html.parser')
div = html_soup.find('div', class_='review')
if not div:
continue
review_id = div.get('id')
# find & clean the rating :
review_classes = ' '.join(html_soup.find(class_ = 'review-rating').get('class'))
rating = re.search('a-star-(\d+)', review_classes).group(1)
title = html_soup.find(class_='review-title').get_text(strip = True)
review = html_soup.find(class_='review-text').get_text(strip = True)
review.append({'review_id' : review_id,
'rating' : rating,
'title' : title,
'review' : review})
return reviews
def get_reviews(product_id, page):
data = {
'sortBy' : '',
'reveiwerType' : 'all_reviews',
'formatType' : '',
'mediaType' : '',
'filterByStar' : 'all_stars',
'pageNumber' : page,
'filterByKeyword' : '',
'shouldAppend' : 'undefined',
'deviceType' : 'desktop',
'reftag' : 'cm_cr_getr_d_paging_btm_{}'.format(page),
'pageSize' : 15,
'asin' : product_id,
'scope' : 'reviewsAjax1'
}
r = session.post(review_url + 'ref=' + data['reftag'], data = data)
reviews = parse_reviews(r.text)
return reviews
page = 1
while True:
print("Scraping page", page)
reviews = get_reviews(product_id, page)
if not reviews:
break
for review in reviews:
print(' -', review['rating'], review['title'])
db['reviews'].upsert(review, ['review_id'])
page += 1
以下错误消息给我 -
**JSONDecodeError** Traceback (most recent call last)
<ipython-input-5-75cef79b98a4> in <module>
60 while True:
61 print("Scraping page", page)
---> 62 reviews = get_reviews(product_id, page)
63 if not reviews:
64 break
<ipython-input-5-75cef79b98a4> in get_reviews(product_id, page)
54 }
55 r = session.post(review_url + 'ref=' + data['reftag'], data = data)
---> 56 reviews = parse_reviews(r.text)
57 return reviews
58
<ipython-input-5-75cef79b98a4> in parse_reviews(reply)
17 if not fragment.strip():
18 continue
---> 19 json_fragment = json.loads(fragment)
20 if json_fragment[0] != 'append':
21 continue
**JSONDecodeError:** Expecting value: line 1 column 1 (char 0)
请帮帮我，我已经尝试了其中的所有内容，但仍然卡住了。 提前致谢
【问题讨论】：
-
你需要检查
fragment- 它是一个有效的json字符串吗？ -
我认为您的意思是“刮擦”。报废意味着扔掉。
-
这个错误提示，当你尝试
json.loads它时，fragment是一个空字符串。
标签： python json web-scraping beautifulsoup