我需要获取新闻文章数据。我正在使用来自 python 的请求/获取，但出现此错误：403 禁止答案

【问题标题】：I need to get news article data. I'm using request/get from python but I got this error: 403 forbidden我需要获取新闻文章数据。我正在使用来自 python 的请求/获取，但出现此错误：403 禁止
【发布时间】：2023-04-03 10:59:02
【问题描述】：

代码如下：

from requests import get
from bs4 import BeautifulSoup

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'}

url = 'https://business.inquirer.net/category/latest-stories/page/10'
response = get(url)
print(response.text[:500])
html_soup = BeautifulSoup(response.text, 'html.parser')
type(html_soup)

这是我得到的结果：

<html>
<head><title>403 Forbidden</title></head>
<body bgcolor="white">
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>

我已经读过放置一个标题将解决错误，但我尝试放置我在检查站点时从 devtool 复制的标题，但它并没有解决我的问题请帮帮我

【问题讨论】：

标签： python html python-requests

【解决方案1】：

您不会在任何地方使用 headers 变量，因此您不会通过请求传递它。你可以用这样的代码做到这一点：

from bs4 import BeautifulSoup
from urllib.request import Request, urlopen

siteurl = "https://business.inquirer.net/category/latest-stories/page/10"
hdr = {'User-Agent': 'Mozilla/5.0'}
req = Request(siteurl,headers=hdr)
page = urlopen(req)
soup = BeautifulSoup(page)
print(soup)

【讨论】：

【解决方案2】：

当尝试使用BeautifulSoap 从该站点抓取数据时，站点不会显示其数据。

当你尝试时：

from bs4 import BeautifulSoup
from urllib import urlopen

url = "https://business.inquirer.net/category/latest-stories/page/10"

open_page = urlopen(url)
source = BeautifulSoup(open_page,"html.parser")

print source

你会看到这样一行：

<p>The owner of this website (business.inquirer.net) has banned your access based on your browser's signature (4af0dedd3eebcb40-ua48).</p>

所以不要注意使用 BeautifulSoap 来做到这一点。使用Selenium 更容易。

from selenium import webdriver


options = webdriver.ChromeOptions()
driver=webdriver.Chrome(chrome_options=options, executable_path=r'your driver path')
driver.get('https://business.inquirer.net/category/latest-stories/page/10')

x = driver.find_elements_by_css_selector("div[id='ch-ls-head']")


for a in x:
  print a.text
driver.close()

输出：

TAXATION
DOF clarifies: Rice tariffication law takes effect on March 5
FEBRUARY 19, 2019 BY:  BEN O. DE VERA
BANKS
HSBC reports net profit at $12.6B in 2018
FEBRUARY 19, 2019
CURRENCIES
Asian shares gain on hopes for progress on China-US trade
FEBRUARY 19, 2019
ECONOMY
Amro sees higher PH growth in 2019 on easing inflation, infra boost
FEBRUARY 19, 2019 BY:  BEN O. DE VERA
TELECOMMUNICATIONS
Poe to DICT: Stop ‘dilly-dallying’ over 3rd telco project
FEBRUARY 19, 2019 BY:  CHRISTIA MARIE RAMOS
SOCIAL SECURITY
SSS contribution collections grow by P22.19B in 2018
FEBRUARY 18, 2019 BY:  CHRISTIA MARIE RAMOS
STOCKS
World stocks mixed ahead of further China-US trade talks
FEBRUARY 18, 2019
TRADE
Rice tariffication starts on March 3
FEBRUARY 18, 2019 BY:  BEN O. DE VERA
AGRICULTURE/AGRIBUSINESS
NFA-Bohol workers wear black to mourn ‘death of the rice industry’
FEBRUARY 18, 2019 BY:  LEO UDTOHAN
BONDS
Treasury: RTBs to be sold to individual investors online in Q1
FEBRUARY 18, 2019 BY:  BEN O. DE VERA

【讨论】：

【解决方案3】：

简单地为我工作

from bs4 import BeautifulSoup
import urllib.request 
response = urllib.request.urlopen('https://business.inquirer.net/category/latest-stories/page/10') 
html = response.read()
soup = BeautifulSoup(html,"html5lib")
text = soup.get_text(strip=True)
print (text)

【讨论】：

【解决方案4】：

尝试包含标头，许多网站会阻止没有标头的请求：

r = requests.get(url, headers=...)

查看请求文档了解更多信息：http://docs.python-requests.org/en/master/user/quickstart/

【讨论】：