POST 查询以访问网站上的表数据 (Python)

Question

我想使用this website 来提取数据，但是这个网站没有方便提取数据的API。因此，网站上有一个包含数据的“表格”，但是该表格分布在多个页面上，并且下载选项不支持“批量”下载。所以我想想办法通过 Python 下载它。
我发现我可以通过发布请求来做到这一点。我所做的是从第 1 页开始，然后单击“下一页”按钮并观看了网络选项卡，这给了我：

Request URL:https://www.patricbrc.org/api/genome/
Request Method:POST
Status Code:200 OK
Remote Address:128.173.97.11:443
Referrer Policy:no-referrer-when-downgrade

Response Headers
Access-Control-Allow-Origin:https://www.patricbrc.org
Access-Control-Expose-Headers:facet_counts,x-facet-count,Content-Range,X-Content-Range,ETag
Content-Encoding:gzip
Content-Range:items 200-399/45999
Content-Type:application/json; charset=utf-8
Date:Sat, 26 Aug 2017 09:36:40 GMT
ETag:W/"8ac0b-cFdEBUwfdiyTQm/gpJHYzQ"
Server:nginx/1.9.1
Transfer-Encoding:chunked
Vary:Origin, Accept
Vary:Accept-Encoding
X-Powered-By:Express

Request Headers
Accept:application/javascript, application/json, application/json
Accept-Encoding:gzip, deflate, br
Accept-Language:nl-NL,nl;q=0.8,en-US;q=0.6,en;q=0.4
Connection:keep-alive
Content-Length:82
Content-Type:application/rqlquery+x-www-form-urlencoded
Cookie:sprod=hZ36nWTBiHHaDwdAhCChFGZo; _ga=GA1.2.582935779.1503602311; _gid=GA1.2.174755110.1503602311; _gat=1
Host:www.patricbrc.org
Origin:https://www.patricbrc.org
Range:items=200-399
Referer:https://www.patricbrc.org/view/Taxonomy/2
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36
X-Range:items=200-399

Request Payload
eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)
Name
genome/

要查看页面切换后发生的变化，我再次执行此操作（因此再次单击“下一页”按钮）。

Request URL:https://www.patricbrc.org/api/genome/
Request Method:POST
Status Code:200 OK
Remote Address:128.173.97.11:443
Referrer Policy:no-referrer-when-downgrade

Response Headers
Access-Control-Allow-Origin:https://www.patricbrc.org
Access-Control-Expose-Headers:facet_counts,x-facet-count,Content-Range,X-Content-Range,ETag
Content-Encoding:gzip
Content-Range:items 400-599/45999
Content-Type:application/json; charset=utf-8
Date:Sat, 26 Aug 2017 09:44:24 GMT
ETag:W/"5dc73-ODopjbbyl5M2vUJvUGN0Gw"
Server:nginx/1.9.1
Transfer-Encoding:chunked
Vary:Origin, Accept
Vary:Accept-Encoding
X-Powered-By:Express

Request Headers
Accept:application/javascript, application/json, application/json
Accept-Encoding:gzip, deflate, br
Accept-Language:nl-NL,nl;q=0.8,en-US;q=0.6,en;q=0.4
Connection:keep-alive
Content-Length:82
Content-Type:application/rqlquery+x-www-form-urlencoded
Cookie:sprod=hZ36nWTBiHHaDwdAhCChFGZo; _ga=GA1.2.582935779.1503602311; _gid=GA1.2.174755110.1503602311
Host:www.patricbrc.org
Origin:https://www.patricbrc.org
Range:items=400-599
Referer:https://www.patricbrc.org/view/Taxonomy/2
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36
X-Range:items=400-599

Request Payload
eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)

因此，在从第 1 页 --> 2 和从第 2 页 --> 3 开始发布请求期间（在我看来）发生变化的重要事情是：

• 响应标题 Content-Range：items 200-399/45999 --> Content-Range：items 400-599/45999
• 请求标头 范围：items=200-399 --> 范围：items=400-599
• 请求标头 X-Range：items=200-399 --> X-Range：items=400-599

网站是这样说的：

• 第 1 页：45999 个结果中的 1 - 200 个（但 0-199 个帖子，上面未显示）
• 第 2 页：45999 个结果中的 201 - 400 个（但帖子为 200-399，见上文）
• 第 3 页：45999 个结果中的 401 - 600 个（但 400-599 个帖子，见上文）

我对阅读这些 POST 请求的知识零。不过我想应该有办法说：

requests.post(some url) and extract the total page numbers (or number of genomes and devide these by the max number per page, which is 200 see post requests). 

然后是这样的：

for page_numb in range(page_numbers):
    r = requests.post(some_url...... + page_numb)
    #extract the information from the table

最后的问题:) 我不知道如何设置像

这样的 POST 请求

p1 = requests.post(https://www.patricbrc.org/api/genome/... 0-199)
p2 = requests.post(https://www.patricbrc.org/api/genome/... 200-399)
etc...

（并且我希望回复中包含由一些分隔符分隔的表格数据）

---

我试过的代码：

import requests
import json

url = 'https://www.patricbrc.org/api/genome/'
payload = {"eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)"}
headers = {
    "X-Range": "items=0-199",
    "Range": "items=0-199",
    "content-type": "application/json"}


r = requests.post(url, data=json.dumps(payload), headers=headers)
print(r)

这段代码不起作用，但我想应该是这样的。
我知道我应该将有效负载作为 JSON 发布，但我不知道如何将 eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score) 转换为 JSON，它是键：值对。

Answer 1

这是一个源代码（其丑陋的代码，但它只是一个例子，我建议你用request.Session来做）：

import sys
import json
import requests

URL = 'https://www.patricbrc.org/view/Taxonomy/2'
API = 'https://www.patricbrc.org/api/genome/'

# we need to get a Cookie
response = requests.get(URL)
if response.status_code != 200:
    sys.exit('Cookie request failed')
cookie = response.headers['Set-Cookie'].split(';')[0]
etag   = response.headers['ETag']

# now with cookie request a data
headers = {
    'Host'            : 'www.patricbrc.org',
    'Accept'          : 'application/javascript, application/json, application/json',
    'Accept-Language' : 'en-US,en;q=0.5',
    'Referer'         : 'https://www.patricbrc.org/view/Taxonomy/2',
    'X-Range'         : 'items=200-399',  # your pages
    'Range'           : 'items=200-399',  # your pages
    'Content-Type'    : 'application/rqlquery+x-www-form-urlencoded',
    'Cookie'          : '{};'.format(cookie),
    'ETag'            : etag
}

data = 'eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)'
response = requests.post(API, headers=headers, data=data)
if response.status_code != 200:
    sys.exit('API request failed')
data = json.loads(response.text)

# test data
row200 = data[0]
print 'FAMILY     :', row200['family']
print 'GC_CONTENT :', row200['gc_content'] 
print 'GENUS      :', row200['genus']
print 'ORDER      :', row200['order']

test data 块的输出：

FAMILY     : Eubacteriaceae
GC_CONTENT : 38.78
GENUS      : Eubacterium
ORDER      : Clostridiales

编辑 您不需要 cookie，这也可以：

import json
import requests

API = 'https://www.patricbrc.org/api/genome/'

# now with cookie request a data
headers = {
    'Host'            : 'www.patricbrc.org',
    'Accept'          : 'application/javascript, application/json, application/json',
    'Accept-Language' : 'en-US,en;q=0.5',
    'Referer'         : 'https://www.patricbrc.org/view/Taxonomy/2',
    'X-Range'         : 'items=200-399',  # your pages
    'Range'           : 'items=200-399',  # your pages
    'Content-Type'    : 'application/rqlquery+x-www-form-urlencoded',
}

query = 'eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)'
response = requests.post(API, headers=headers, data=query)
data = json.loads(response.text)