Python：如何搜索推文并存储在数据库中？

Question

我有一个不错的 Python 脚本，它当前打印出来自给定用户名的过去 200 条推文。

但是，我想对其进行修改，以便它收集过去 200 条包含特定主题标签（来自任何用户名）的推文，然后我想将这些结果存储在数据库中。

谁能提供关于如何修改下面代码的建议？

import sys
import operator
import requests
import json
import twitter

twitter_consumer_key = 'XXXX'
twitter_consumer_secret = 'XXXX'
twitter_access_token = 'XXXX'
twitter_access_secret = 'XXXX'

twitter_api = twitter.Api(consumer_key=twitter_consumer_key, consumer_secret=twitter_consumer_secret, access_token_key=twitter_access_token, access_token_secret=twitter_access_secret)

statuses = twitter_api.GetUserTimeline(screen_name=handle, count=200, include_rts=False)

for status in statuses:
  if (status.lang == 'en'):
    print status

Answer 1

抱歉，我一直在寻找 Python 解决方案，我相信我终于找到并成功测试了它。代码如下。仍在寻找一种方法来修改脚本以将每一行输入到 SQL 数据库中，但我希望我能在其他地方找到它。

pip 安装 TwitterSearch

from TwitterSearch import *
try:
    tso = TwitterSearchOrder() # create a TwitterSearchOrder object
    tso.set_keywords(['Guttenberg', 'Doktorarbeit']) # let's define all words we would like to have a look for
    tso.set_language('de') # we want to see German tweets only
    tso.set_include_entities(False) # and don't give us all those entity information

    # it's about time to create a TwitterSearch object with our secret tokens
    ts = TwitterSearch(
        consumer_key = 'aaabbb',
        consumer_secret = 'cccddd',
        access_token = '111222',
        access_token_secret = '333444'
     )

     # this is where the fun actually starts :)
    for tweet in ts.search_tweets_iterable(tso):
        print( '@%s tweeted: %s' % ( tweet['user']['screen_name'], tweet['text'] ) )

except TwitterSearchException as e: # take care of all those ugly errors if there are some
    print(e)

Answer 2

我附上了一个 java 代码，它将打印出过去 100 条推文，包括“#engineeringproblems”主题标签（来自任何用户）。您需要在库中添加 twitter API 'twitter4J'。

API下载链接-http://twitter4j.org/en/index.html#download

Java 源代码：

public static void main(String[] args) {

    ConfigurationBuilder cb = new ConfigurationBuilder();
    cb.setDebugEnabled(true)
     .setOAuthConsumerKey("xxxx")
     .setOAuthConsumerSecret("xxxx")
     .setOAuthAccessToken("xxxx")
     .setOAuthAccessTokenSecret("xxxx");

    Twitter twitter = new TwitterFactory(cb.build()).getInstance();
    Query query = new Query("#engineeringproblems ");
    int numberOfTweets = 100;
    long lastID = Long.MAX_VALUE;
    ArrayList<Status> tweets = new ArrayList<Status>();

    while (tweets.size() < numberOfTweets) {
        if (numberOfTweets - tweets.size() > 100) {
            query.setCount(100);
        } else {
            query.setCount(numberOfTweets - tweets.size());
        }
        try {
            QueryResult result = twitter.search(query);
            tweets.addAll(result.getTweets());
            System.out.println("Gathered " + tweets.size() + " tweets" + "\n");
            for (Status t : tweets) {
                if (t.getId() < lastID) {
                    lastID = t.getId();
                }
            }

        } catch (TwitterException te) {
            System.out.println("Couldn't connect: " + te);
        };
        query.setMaxId(lastID - 1);
    }
    for (int i = 0; i < tweets.size(); i++) {
        Status t = (Status) tweets.get(i);


        String user = t.getUser().getScreenName();
        String msg = t.getText();

        System.out.println(i + " USER: " + user + " wrote: " + msg + "\n");
    }
}

Answer 3

不熟悉 twitter 包，但这可能是您可以处理的建议。取决于你想如何保存推文，你可以用你想要的方式替换“打印状态”。 但是，这只允许您过滤 200 条推文，而不是获取包含特定主题标签的 200 条推文。

import sys
import operator
import requests
import json
import twitter

twitter_consumer_key = 'XXXX'
twitter_consumer_secret = 'XXXX'
twitter_access_token = 'XXXX'
twitter_access_secret = 'XXXX'

twitter_api = twitter.Api(consumer_key=twitter_consumer_key, consumer_secret=twitter_consumer_secret, access_token_key=twitter_access_token, access_token_secret=twitter_access_secret)

statuses = twitter_api.GetUserTimeline(screen_name=handle, count=200, include_rts=False)

tag_list = ["Xmas", "Summer"]
for status in statuses:
  if (status.lang == 'en'):
    #assume there exists a hashtag in the tweet
    for hashtag in status.entities.hashtags:
      if hashtag.text in tag_list:
        print status