进入 403 页面时的网页抓取答案

【问题标题】：Web scraping when goes to 403 page进入 403 页面时的网页抓取
【发布时间】：2021-08-26 07:22:19
【问题描述】：

我是网络抓取的初学者，需要使用 Beautifulsoup 抓取 https://mirror-h.org/archive/page/1。但它给出了一个错误并转到 403 页面。我该如何解决这个问题？非常感谢您的帮助。

这是我的代码：

import requests
from bs4 import BeautifulSoup
import pandas

url = "https://mirror-h.org/archive/page/1"
page = pandas.read_html(url)
headers = {
    'user-agent:' 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36'
    }
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.content, 'html.parser')
print(soup)

我得到的错误是：

 raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

【问题讨论】：

您好，Tharika，HTTP 错误代码 403 表示您无权访问此页面。可能有很多原因。例如，该页面可能有某种 WAF 前端，不允许“机器人”请求通过。

标签： python beautifulsoup web-scraping-language

【解决方案1】：

import requests
import pandas as pd
from bs4 import BeautifulSoup


# make sure you insert the headers as a dict as you missed the : within your original code
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0'
}


def main(url):
    # included headers in request
    r = requests.get(url, headers=headers)
    soup = BeautifulSoup(r.text, 'lxml')
    # response 200
    print(r)


    # this is how you can use pandas with the previous headers to get 200 response text
    df = pd.read_html(r.text)
    print(df)  # you will get error --> ValueError: No tables found because you are dealing with JS website behind CloudFlare protection! try selenium then!
    


main('https://mirror-h.org/archive/page/1 ')

【讨论】：