抓取 Twitter 数据时出现 JSONDecodeError答案

【问题标题】：JSONDecodeError when scraping Twitter data抓取 Twitter 数据时出现 JSONDecodeError
【发布时间】：2019-05-06 23:07:23
【问题描述】：

所以我正在尝试从 Twitter 上抓取数据。我设法根据特定主题标签检索推文，然后使用以下代码将它们转储到 JSON 格式的文本文件中：

import json
import tweepy
import pandas as pd
import time

with open('consumer_key.txt', 'r') as f:
    consumer_key =  f.read()
f.closed

with open('consumer_secret.txt', 'r') as f:
    consumer_secret = f.read()
f.closed

with open('access_key.txt', 'r') as f:
    access_key = f.read()
f.closed

with open('access_secret.txt', 'r') as f:
    access_secret = f.read()
f.closed

#Authentication
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_key, access_secret)
api = tweepy.API(auth)

api = tweepy.API(auth, wait_on_rate_limit=True,
             wait_on_rate_limit_notify=True) # Connect to the api

if (not api):
    print ("Error")
    sys.exit(-1)

public_tweets = api.home_timeline() # Get the post in the timeline
for tweet in public_tweets:
    print(tweet.text) 

with open('tweets.txt', 'a') as f:
    for tweet in tweepy.Cursor(api.search, q='#apple', count=2, lang='en', since='2017-04-03').items():
        json.dump(tweet._json, f, indent=4)

但是，当我尝试使用读取文件时

with open('tweets.txt', 'r') as f:
    for line in f:
        tweet = json.loads(line)
        tweet_text = tweet['text']
        print(tweet_text)

它给了我一个错误提示“JSONDecodeError：期望用双引号括起来的属性名称：第 2 行第 1 列（字符 2）”

这是我的文本文件的片段：

{
    "created_at": "Wed Dec 05 10:37:07 +0000 2018",
    "id": 1070265807492530176,
    "id_str": "1070265807492530176",
    "text": "RT @evankirstel: The Apple Watch Series 3 Made Up Majority of 
Estimated 4.2 Million Q3 2018 Apple Watch Sales #applewatch #apple  
@mactrast\u2026",
    "truncated": false,
    "entities": {
        "hashtags": [
            {
                "text": "applewatch",
                "indices": [
                    110,
                    121
                ]
            },
            {
                "text": "apple",
                "indices": [
                    122,
                    128
                ]
            }
        ],
        "symbols": [],
        "user_mentions": [
            {
                "screen_name": "evankirstel",
                "name": "Evan Kirstel",
                "id": 35203319,
                "id_str": "35203319",
                "indices": [
                    3,
                    15
                ]
            },
            {
                "screen_name": "MacTrast",
                "name": "MacTrast",
                "id": 16711478,
                "id_str": "16711478",
                "indices": [
                    130,
                    139
                ]
            }
        ],

我该如何解决这个问题？最终，我想做的是根据键提取数据并将其保存在 pandas 数据框中。

【问题讨论】：

您正在逐行读取文件并将每一行传递给 json.loads。但该文件的单行并不代表正确的 JSON 对象。

标签： python python-3.x twitter web-scraping tweepy

【解决方案1】：

您需要阅读整个内容而不是逐行阅读，并且不要忘记添加strict=False以允许/将\u2026转换为…

with open('tweets.txt', 'r') as f:
    tweet = json.load(f, strict=False)
    tweet_text = tweet['text']
    print(tweet_text)

【讨论】：