Python 请求 401 错误，但 url 在浏览器中打开

Question

我正在尝试从该位置提取 json - https://www.nseindia.com/api/option-chain-indices?symbol=BANKNIFTY

这在我的浏览器中可以正常打开，但是在 python 中使用请求会引发 401 权限错误。我尝试添加具有不同参数的标题，但无济于事。 有趣的是，此页面上的 json 在单独打开 https://www.nseindia.com 之前也不会在浏览器中打开。我相信它需要某种身份验证，但令人惊讶的是它可以在没有任何身份验证的情况下在浏览器中运行。

有没有办法从这个 url 中提取信息？非常感谢任何帮助。

这是我的实现 -

import requests

url = 'https://www.nseindia.com/api/option-chain-indices?symbol=BANKNIFTY'

# This throws a 401 response error
page = requests.get(url, headers={"User-Agent": "Mozilla/5.0"})

# This throws a 'Connection aborted' error
page = requests.get(url, headers={"User-Agent": "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 GTB7.1 (.NET CLR 3.5.30729)"})

Answer 1

要获取正确的数据，首先使用requests.get()从其他 URL 加载 cookie，然后执行 Ajax 请求以加载 JSON：

import json
import requests
from bs4 import BeautifulSoup


headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:80.0) Gecko/20100101 Firefox/80.0'}
url = 'https://www.nseindia.com/api/option-chain-indices?symbol=BANKNIFTY'

with requests.session() as s:

    # load cookies:
    s.get('https://www.nseindia.com/get-quotes/derivatives?symbol=BANKNIFTY', headers=headers)

    # get data:
    data = s.get(url, headers=headers).json()

    # print data to screen:
    print(json.dumps(data, indent=4))

打印：

{
    "records": {
        "expiryDates": [
            "03-Sep-2020",
            "10-Sep-2020",
            "17-Sep-2020",
            "24-Sep-2020",
            "01-Oct-2020",
            "08-Oct-2020",
            "15-Oct-2020",
            "22-Oct-2020",
            "29-Oct-2020",
            "26-Nov-2020"
        ],
        "data": [
            {
                "strikePrice": 18100,
                "expiryDate": "03-Sep-2020",
                "CE": {
                    "strikePrice": 18100,
                    "expiryDate": "03-Sep-2020",
                    "underlying": "BANKNIFTY",
                    "identifier": "OPTIDXBANKNIFTY03-09-2020CE18100.00",
                    "openInterest": 1,
                    "changeinOpenInterest": 1,
                    "pchangeinOpenInterest": 0,
                    "totalTradedVolume": 2,
                    "impliedVolatility": 95.51,
                    "lastPrice": 6523.6,
                    "change": 2850.1000000000004,
                    "pChange": 77.58540901048048,
                    "totalBuyQuantity": 2925,
                    "totalSellQuantity": 2800,
                    "bidQty": 25,
                    "bidprice": 6523.6,
                    "askQty": 25,
                    "askPrice": 6570.3,
                    "underlyingValue": 24523.8
                },
                "PE": {
                    "strikePrice": 18100,
                    "expiryDate": "03-Sep-2020",
                    "underlying": "BANKNIFTY",

...and so on.