Python Geopy Nominatim 请求过多

Question

以下脚本与包含 2 行的文件完美配合，但是当我尝试 2500 行文件时，我得到了 429 个异常。因此，我将查询时间增加到 5 秒。我还填写了用户代理。尝试失败后，我连接到 VPN 以更改“新鲜”，但我再次收到 429 错误。我在这里缺少什么吗？ Nominatim 政策规定每秒连接数不超过 1 个，我每 5 秒做一个...任何帮助都会有所帮助！

from geopy.geocoders import Nominatim
import pandas
from functools import partial

from geopy.extra.rate_limiter import RateLimiter

nom = Nominatim(user_agent="xxx@gmail.com")
geocode = RateLimiter(nom.geocode, min_delay_seconds=5)


df=pandas.read_csv('Book1.csv', engine='python')
df["ALL"] = df['Address'].apply(partial(nom.geocode, timeout=1000, language='en'))
df["Latitude"] = df["ALL"].apply(lambda x: x.latitude if x != None else None)
df["Longitude"] = df["ALL"].apply(lambda x: x.longitude if x != None else None)

writer = pandas.ExcelWriter('Book1.xlsx')
df.to_excel(writer, 'new_sheet')
writer.save()

错误信息：

Traceback (most recent call last):
  File "C:\Users\u6022697\AppData\Local\Programs\Python\Python37\lib\site-packages\geopy\geocoders\base.py", line 355, in _call_geocoder
    page = requester(req, timeout=timeout, **kwargs)
  File "C:\Users\u6022697\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 531, in open
    response = meth(req, response)
  File "C:\Users\u6022697\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Users\u6022697\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 569, in error
    return self._call_chain(*args)
  File "C:\Users\u6022697\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 503, in _call_chain
    result = func(*args)
  File "C:\Users\u6022697\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 429: Too Many Requests

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/u6022697/Documents/python work/Multiple GPS Nom Pandas.py", line 14, in <module>
    df["ALL"] = df['Address'].apply(partial(nom.geocode, timeout=1000, language='en'))
  File "C:\Users\u6022697\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\core\series.py", line 3849, in apply
    mapped = lib.map_infer(values, f, convert=convert_dtype)
  File "pandas\_libs\lib.pyx", line 2327, in pandas._libs.lib.map_infer
  File "C:\Users\u6022697\AppData\Local\Programs\Python\Python37\lib\site-packages\geopy\geocoders\osm.py", line 406, in geocode
    self._call_geocoder(url, timeout=timeout), exactly_one
  File "C:\Users\u6022697\AppData\Local\Programs\Python\Python37\lib\site-packages\geopy\geocoders\base.py", line 373, in _call_geocoder
    raise ERROR_CODE_MAP[code](message)
geopy.exc.GeocoderQuotaExceeded: HTTP Error 429: Too Many Requests

Answer 1

我在不到一天的时间里就对大约 10K 不同的经纬组合进行了反向地理编码。 Nominatim 不喜欢批量查询，所以这个想法是为了防止看起来像一个。这是我的建议：

1. 确保您只查询唯一项。我发现 Nominatim 阻止了对相同经纬度组合的重复查询。地址也是如此。您可以使用unq_address = df['address'].unique()，然后使用该系列进行查询。你甚至可以得到更少的地址。

2. 查询之间的时间应该随机。我还将 user_agent 设置为每次都有一个随机数。就我而言，我使用以下代码：

   from time import sleep
   from random import randint
   from geopy.geocoders import Nominatim
   from geopy.exc import GeocoderTimedOut, GeocoderServiceError
   
   user_agent = 'user_me_{}'.format(randint(10000,99999))
   geolocator = Nominatim(user_agent=user_agent)
   def reverse_geocode(geolocator, latlon, sleep_sec):
       try:
           return geolocator.reverse(latlon)
       except GeocoderTimedOut:
           logging.info('TIMED OUT: GeocoderTimedOut: Retrying...')
           sleep(randint(1*100,sleep_sec*100)/100)
           return reverse_geocode(geolocator, latlon, sleep_sec)
       except GeocoderServiceError as e:
           logging.info('CONNECTION REFUSED: GeocoderServiceError encountered.')
           logging.error(e)
           return None
       except Exception as e:
           logging.info('ERROR: Terminating due to exception {}'.format(e))
           return None
   

我发现 sleep(randint(1*100,sleep_sec*100)/100) 行对我有用。

Answer 2

经过一些研究，事实证明 Nominatim 每天有 1000 次查询限制，因此该脚本试图执行超过 1k 次。

https://getlon.lat/