【发布时间】：2020-07-31 19:24:01
【问题描述】：
问题和输出
使用我的 python 脚本时，我从返回 json 的 API 调用中得到了相同的响应，而且它似乎同时发生了几个小时。
我正在使用Coindesk BPI API，它每分钟更新一次。所以我们知道，比特币的价格不会在 5 个小时内保持稳定。请参阅下面的输出示例：
# results.txt
{"timestamp": 16-Apr-2020 22:50, "price": 7078, "gCount": 28, "rCount": 48}
{"timestamp": 16-Apr-2020 23:00, "price": 7085, "gCount": 29, "rCount": 50}
{"timestamp": 16-Apr-2020 23:10, "price": 7011, "gCount": 33, "rCount": 52}
{"timestamp": 16-Apr-2020 23:20, "price": 7002, "gCount": 31, "rCount": 55}
{"timestamp": 16-Apr-2020 23:30, "price": 7020, "gCount": 30, "rCount": 52}
{"timestamp": 16-Apr-2020 23:40, "price": 7027, "gCount": 33, "rCount": 54}
{"timestamp": 16-Apr-2020 23:50, "price": 7047, "gCount": 35, "rCount": 58}
{"timestamp": 17-Apr-2020 00:01, "price": 7060, "gCount": 36, "rCount": 57}
{"timestamp": 17-Apr-2020 00:10, "price": 7051, "gCount": 34, "rCount": 45}
{"timestamp": 17-Apr-2020 00:20, "price": 7052, "gCount": 41, "rCount": 48}
{"timestamp": 17-Apr-2020 00:31, "price": 7054, "gCount": 47, "rCount": 48}
# It worked! Now the price is stuck for 2 get requests.
{"timestamp": 17-Apr-2020 00:40, "price": 7054, "gCount": 48, "rCount": 47}
{"timestamp": 17-Apr-2020 00:50, "price": 7054, "gCount": 50, "rCount": 48}
{"timestamp": 17-Apr-2020 01:01, "price": 7051, "gCount": 48, "rCount": 43}
# Price stuck again for around 30 get requests.
{"timestamp": 17-Apr-2020 01:10, "price": 7051, "gCount": 46, "rCount": 47}
{"timestamp": 17-Apr-2020 01:20, "price": 7051, "gCount": 49, "rCount": 46}
{"timestamp": 17-Apr-2020 01:30, "price": 7051, "gCount": 48, "rCount": 47}
{"timestamp": 17-Apr-2020 01:40, "price": 7051, "gCount": 50, "rCount": 48}
{"timestamp": 17-Apr-2020 01:50, "price": 7051, "gCount": 50, "rCount": 52}
{"timestamp": 17-Apr-2020 02:00, "price": 7051, "gCount": 51, "rCount": 56}
{"timestamp": 17-Apr-2020 02:10, "price": 7051, "gCount": 50, "rCount": 55}
{"timestamp": 17-Apr-2020 02:20, "price": 7051, "gCount": 57, "rCount": 57}
{"timestamp": 17-Apr-2020 02:30, "price": 7051, "gCount": 48, "rCount": 54}
{"timestamp": 17-Apr-2020 02:40, "price": 7051, "gCount": 52, "rCount": 54}
{"timestamp": 17-Apr-2020 02:51, "price": 7051, "gCount": 54, "rCount": 57}
{"timestamp": 17-Apr-2020 03:00, "price": 7051, "gCount": 53, "rCount": 59}
{"timestamp": 17-Apr-2020 03:11, "price": 7051, "gCount": 53, "rCount": 59}
{"timestamp": 17-Apr-2020 03:21, "price": 7051, "gCount": 50, "rCount": 55}
{"timestamp": 17-Apr-2020 03:31, "price": 7051, "gCount": 51, "rCount": 55}
{"timestamp": 17-Apr-2020 03:41, "price": 7051, "gCount": 52, "rCount": 56}
{"timestamp": 17-Apr-2020 03:51, "price": 7051, "gCount": 50, "rCount": 55}
{"timestamp": 17-Apr-2020 04:01, "price": 7051, "gCount": 48, "rCount": 56}
{"timestamp": 17-Apr-2020 04:10, "price": 7051, "gCount": 39, "rCount": 50}
{"timestamp": 17-Apr-2020 04:20, "price": 7051, "gCount": 39, "rCount": 49}
{"timestamp": 17-Apr-2020 04:31, "price": 7051, "gCount": 41, "rCount": 53}
{"timestamp": 17-Apr-2020 04:40, "price": 7051, "gCount": 43, "rCount": 53}
{"timestamp": 17-Apr-2020 04:50, "price": 7051, "gCount": 39, "rCount": 51}
{"timestamp": 17-Apr-2020 05:00, "price": 7051, "gCount": 37, "rCount": 52}
{"timestamp": 17-Apr-2020 05:11, "price": 7051, "gCount": 38, "rCount": 54}
{"timestamp": 17-Apr-2020 05:20, "price": 7051, "gCount": 31, "rCount": 49}
{"timestamp": 17-Apr-2020 05:30, "price": 7051, "gCount": 0, "rCount": 0}
{"timestamp": 17-Apr-2020 05:41, "price": 7051, "gCount": 32, "rCount": 49}
{"timestamp": 17-Apr-2020 05:50, "price": 7051, "gCount": 37, "rCount": 49}
{"timestamp": 17-Apr-2020 06:01, "price": 7051, "gCount": 39, "rCount": 51}
{"timestamp": 17-Apr-2020 06:11, "price": 7051, "gCount": 41, "rCount": 47}
{"timestamp": 17-Apr-2020 06:21, "price": 7051, "gCount": 42, "rCount": 46}
# Now it works again as intended.
{"timestamp": 17-Apr-2020 06:31, "price": 7082, "gCount": 45, "rCount": 49}
{"timestamp": 17-Apr-2020 06:40, "price": 7084, "gCount": 48, "rCount": 50}
{"timestamp": 17-Apr-2020 06:51, "price": 7095, "gCount": 45, "rCount": 51}
{"timestamp": 17-Apr-2020 07:01, "price": 7097, "gCount": 44, "rCount": 45}
{"timestamp": 17-Apr-2020 07:11, "price": 7068, "gCount": 45, "rCount": 46}
{"timestamp": 17-Apr-2020 07:21, "price": 7070, "gCount": 43, "rCount": 45}
Python 脚本和我的尝试
我正在使用 python 2.7 和请求。默认情况下，请求不缓存。所以我认为连接只是随机保持，python 重用它，得到相同的 json。
我尝试通过设置keep alive to false、using the with block 和尝试requests.session().close() 来关闭请求会话。下面找到相关的python代码：
import requests, json, sys, time
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
def request_json():
print 'Begin request to get the json...'
# Try get request once
response = requests_retry_session().get('https://api.coindesk.com/v1/bpi/currentprice/USD.json')
if (response.status_code == 200):
# Close the connection
# requests.session().close() <-- tried, doesn't do the trick
print 'Fetched price succesfully.\n'
return response.json()
# If first request didn't succeed, retry 3 times using session
with requests.Session() as s:
s.get('https://api.coindesk.com/v1/bpi/currentprice/USD.json')
# Close the connection
# s.config['keep_alive'] = False <-- tried, doesn't do the trick
response = requests_retry_session(session=s).get(
'https://api.coindesk.com/v1/bpi/currentprice/USD.json'
)
# When requests succeed using session
if (response.status_code == 200):
# Close the connection
# requests.session().close() <-- tried, doesn't do the trick
print 'Fetched price succesfully.\n'
return response.json()
print 'Couldn\'t fetch price json.'
return 'error'
def requests_retry_session(
retries=3,
backoff_factor=0.3,
status_forcelist=(500, 502, 504),
session=None,
):
session = session or requests.Session()
retry = Retry(
total=retries,
read=retries,
connect=retries,
backoff_factor=backoff_factor,
status_forcelist=status_forcelist,
)
adapter = HTTPAdapter(max_retries=retry)
session.mount('https://', adapter)
return session
def get_price_data(json):
price = str(json['bpi']['USD']['rate'])
# Strip the ',' from price, convert to float and to int
price = int(float(price.replace(',', '')))
return price
def main():
# Send a request for the bitcoin price json
priceJson = request_json()
# Check if the request and retries failed
if (json == 'error'):
print 'Terminating bitcoinPrice.py script.'
sys.exit()
# Get the data from the response json
priceInt = get_price_data(priceJson)
# Get timestamp as milliseconds
milli_sec = int(round(time.time() * 1000))
# Read the colordata from colors.txt
# The format is: '63,61' where greenCount,redCount
fh = open('colors.txt', 'r')
colorData = fh.read()
gCount = colorData.split(',')[0]
rCount = colorData.split(',')[1]
# Create a string in json format with the price and color data
dataString = "{\"timestamp\": \"%d\", \"price\": \"%d\", \"gCount\": \"%s\", \"rCount\": \"%s\"}" % (milli_sec, priceInt, gCount, rCount)
print dataString
# Read and write to results.txt
fh = open('results/results.txt', 'a')
fh.write(dataString + '\n')
fh.close()
print '\nSuccesfully saved BTC price and color data to results.txt'
if __name__ == '__main__':
main()
我无法通过以普通用户身份每分钟运行 crontab 来重现该错误，并且只能使用这个 bitcoinPrice.py 脚本。
当我的根 crontab 每 10 分钟运行一次时，当其他几个脚本在此之前运行时，似乎会出现该错误。由 root 用户运行的实际 crontab 简化了其他脚本，如下所示：
*/10 * * * * node script1.js && python2 script2.py && python2 bitcoinPrice.py && /home/user/clearcache.sh
所有其他脚本都按预期工作。最后一个脚本 clearcache.sh 通过以下方式重置缓存和缓冲区，如 here 所述：
#!/bin/sh
sync; echo 3 > /proc/sys/vm/drop_caches
我想了解这个错误是怎么回事。如果找不到解决方案，我将开始使用 curl 并将 API json 响应转储到文件中并从那里读取它。任何想法表示赞赏！
【问题讨论】：
-
您正在写入文件的“priceEtc”变量中的内容。看不到它在您的程序中的任何地方使用。
-
@AmalTs 非常感谢您花时间检查我的代码。我更新了python代码。
-
据我观察，这与 crontab 无关。尝试在标头中传递
{'Cache-Control': 'no-cache'}以禁用缓存。由于您无法使用相同的脚本重现它，这可能是特定于环境的问题。我建议在每个步骤之后进行一些日志记录，以确定在特定场景下出了什么问题。 -
欣赏您的想法。我尝试使用“无缓存”，但问题仍然存在。我现在将进入选项 B 并将 json 卷曲到一个文件中并从那里读取它。
标签： python http caching python-requests centos