【发布时间】:2019-02-20 16:44:19
【问题描述】:
我尝试在 python 中编写两个函数,可用于从站点下载数据并将其保存为 csv 文件。
功能有:
def get_site_url( site_number='01589440', date_from="2008-01-01",date_to="2017-12-31" ):
# input: site_number is a string
# the parameters in the url can have different formats. cb_00065 is the same as parameterCd=00060
#url = 'https://nwis.waterdata.usgs.gov/nwis/uv/cb_00065=on&format=rdb&site_no=%speriod=&begin_date=%s\
#&end_date=%s&siteStatus=all'%(site_number, date_from, date_to)
url = "https://nwis.waterdata.usgs.gov/nwis/uv/?parameterCd=00060,00065&format=rdb&site_no=%s&period=&begin_date=%s\
&end_date=%s&siteStatus=all"%(site_number, date_from, date_to)
return url
def download_data_from_url( url, savename='test.csv' ):
'''
One can download data with a different method that supports resume. If data is two large then it takes
lot of time and the connection to the server might be interrupted.
'''
buffer = BytesIO()
c = pycurl.Curl()
c.setopt(c.URL, url)
c.setopt(c.WRITEDATA, buffer)
c.perform()
c.close()
body = buffer.getvalue() # Body is a byte string.
with open( savename, 'w' ) as output:
output.write( body.decode('utf-8')) # We have to know the encoding in order to print it to a text file
接下来使用这两个函数:
FirstExcUrl = get_site_url()
download_data_from_url(FirstExcUrl, '%s.csv'%'01589440')
产生了错误:
errorTraceback (most recent call last)
<ipython-input-5-f460d226bec2> in <module>()
1 FirstExcUrl = get_site_url()
----> 2 download_data_from_url(FirstExcUrl, '%s.csv'%'01589440')
<ipython-input-3-e6e6e1be72e0> in download_data_from_url(url, savename)
19 c.setopt(c.URL, url)
20 c.setopt(c.WRITEDATA, buffer)
---> 21 c.perform()
22 c.close()
23
error: (1, 'Protocol "https" not supported or disabled in libcurl')
它是一个 jupyter 笔记本的输出,是从一个经过深思熟虑的 Canopy 编辑器窗口启动的。我从 Canpy 包管理器安装了 curl 和 PycURL。安装的curl版本为:7.58.0-1,安装的PycURL版本为:7.19.5-5。
非常感谢任何帮助、建议或解决方案。
【问题讨论】:
标签: curl jupyter-notebook canopy pycurl macos-high-sierra