使用 Tweepy 避免 Twitter API 限制

Question

我在 Stack Exchange 上的一些问题中看到，限制可以是每 15 分钟请求数的函数，还取决于算法的复杂性，只不过这并不复杂。

所以我使用这个代码：

import tweepy
import sqlite3
import time

db = sqlite3.connect('data/MyDB.db')

# Get a cursor object
cursor = db.cursor()
cursor.execute('''CREATE TABLE IF NOT EXISTS MyTable(id INTEGER PRIMARY KEY, name TEXT, geo TEXT, image TEXT, source TEXT, timestamp TEXT, text TEXT, rt INTEGER)''')
db.commit()

consumer_key = ""
consumer_secret = ""
key = ""
secret = ""

auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(key, secret)

api = tweepy.API(auth)

search = "#MyHashtag"

for tweet in tweepy.Cursor(api.search,
                           q=search,
                           include_entities=True).items():
    while True:
        try:
            cursor.execute('''INSERT INTO MyTable(name, geo, image, source, timestamp, text, rt) VALUES(?,?,?,?,?,?,?)''',(tweet.user.screen_name, str(tweet.geo), tweet.user.profile_image_url, tweet.source, tweet.created_at, tweet.text, tweet.retweet_count))
        except tweepy.TweepError:
                time.sleep(60 * 15)
                continue
        break
db.commit()
db.close()

我总是收到 Twitter 限制错误：

Traceback (most recent call last):
  File "stream.py", line 25, in <module>
    include_entities=True).items():
  File "/usr/local/lib/python2.7/dist-packages/tweepy/cursor.py", line 153, in next
    self.current_page = self.page_iterator.next()
  File "/usr/local/lib/python2.7/dist-packages/tweepy/cursor.py", line 98, in next
    data = self.method(max_id = max_id, *self.args, **self.kargs)
  File "/usr/local/lib/python2.7/dist-packages/tweepy/binder.py", line 200, in _call
    return method.execute()
  File "/usr/local/lib/python2.7/dist-packages/tweepy/binder.py", line 176, in execute
    raise TweepError(error_msg, resp)
tweepy.error.TweepError: [{'message': 'Rate limit exceeded', 'code': 88}]

Answer 1

对于在 Google 上偶然发现这一点的任何人，tweepy 3.2+ 为 tweepy.api 类提供了额外的参数，特别是：

• wait_on_rate_limit – 是否自动等待速率限制补充
• wait_on_rate_limit_notify – 当 Tweepy 等待速率限制补充时是否打印通知

将这些标志设置为 True 会将等待委托给 API 实例，这对于大多数简单的用例来说已经足够了。

Answer 2

问题是您的try: except: 块在错误的位置。将数据插入数据库永远不会引发TweepError - 它会迭代Cursor.items()。我建议重构您的代码以在无限循环中调用Cursor.items() 的next 方法。该调用应放在try: except: 块中，因为它可能引发错误。

这是（大致）代码的样子：

# above omitted for brevity
c = tweepy.Cursor(api.search,
                       q=search,
                       include_entities=True).items()
while True:
    try:
        tweet = c.next()
        # Insert into db
    except tweepy.TweepError:
        time.sleep(60 * 15)
        continue
    except StopIteration:
        break

这是因为当 Tweepy 引发 TweepError 时，它没有更新任何游标数据。下次它发出请求时，它将使用与触发速率限制的请求相同的参数，有效地重复它直到它通过为止。

Answer 3

只需替换

api = tweepy.API(auth)

与

api = tweepy.API(auth, wait_on_rate_limit=True)

Answer 4

如果您想避免错误并遵守速率限制，您可以使用以下函数，它将您的 api 对象作为参数。它检索与上次请求类型相同的剩余请求数，并等待直到重置速率限制（如果需要）。

def test_rate_limit(api, wait=True, buffer=.1):
    """
    Tests whether the rate limit of the last request has been reached.
    :param api: The `tweepy` api instance.
    :param wait: A flag indicating whether to wait for the rate limit reset
                 if the rate limit has been reached.
    :param buffer: A buffer time in seconds that is added on to the waiting
                   time as an extra safety margin.
    :return: True if it is ok to proceed with the next request. False otherwise.
    """
    #Get the number of remaining requests
    remaining = int(api.last_response.getheader('x-rate-limit-remaining'))
    #Check if we have reached the limit
    if remaining == 0:
        limit = int(api.last_response.getheader('x-rate-limit-limit'))
        reset = int(api.last_response.getheader('x-rate-limit-reset'))
        #Parse the UTC time
        reset = datetime.fromtimestamp(reset)
        #Let the user know we have reached the rate limit
        print "0 of {} requests remaining until {}.".format(limit, reset)

        if wait:
            #Determine the delay and sleep
            delay = (reset - datetime.now()).total_seconds() + buffer
            print "Sleeping for {}s...".format(delay)
            sleep(delay)
            #We have waited for the rate limit reset. OK to proceed.
            return True
        else:
            #We have reached the rate limit. The user needs to handle the rate limit manually.
            return False 

    #We have not reached the rate limit
    return True

Answer 5

import tweepy
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
# will notify user on ratelimit and will wait by it self no need of sleep.
api = tweepy.API(auth, wait_on_rate_limit=True, wait_on_rate_limit_notify=True)

Answer 6

我建议您使用新的 api v2 并使用带有 wait_on_rate_limit=True 标志的 Client obj，v1 将尽快被弃用

client = tweepy.Client(consumer_key=auth.consumer_key, consumer_secret=auth.consumer_secret, access_token_secret=auth.access_token_secret, access_token=auth.access_token,
                       bearer_token=twitter_bearer_token, wait_on_rate_limit=True)

这一切都是自动的