使用 Python 进行网页抓取：BeautifulSoup 的问题答案

【问题标题】：Web Scraping with Python: problem with BeautifulSoup使用 Python 进行网页抓取：BeautifulSoup 的问题
【发布时间】：2019-06-08 12:06:57
【问题描述】：

请帮助我使用 BeautifulSoup 使用 Python 3 从investment.com 网络抓取财务值。无论我做什么都不会获得任何价值，并且过滤类正在从网页永久更改，它是一个实时值。

import requests

from bs4 import BeautifulSoup

url = "https://es.investing.com/indices/spain-35-futures"
precio_objetivo = input("Introduce el PRECIO del disparador:")
precio_objetivo = float(precio_objetivo)
print (precio_objetivo)

while True:
html = requests.get(url).text
soup = BeautifulSoup(html, "html.parser")
precio_actual = soup.find('span', attrs={'class': 'arial_26 inlineblock pid-8828-last','id':'last_last','dir':'ltr'})
print (precio_actual)
break;

当我不在soup.find 应用任何过滤器时（至少尝试获取所有网页）我得到这个结果：

<bound method Tag.find_all of 
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"

"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<html>
<head>
<title>403 You are banned from this site.  Please contact via a different client configuration if you believe that this is a mistake.                                </title>
</head>
<body>
<h1>Error 403 You are banned from this site.  Please contact via a different client configuration if you believe that this is a mistake.</h1>
<p>You are banned from this site.  Please contact via a different client configuration if you believe that this is a mistake.</p>
<h3>Guru Meditation:</h3>
<p>XID: 850285196</p>
<hr/>
<p>Varnish cache server</p>
</body>
</html>

【问题讨论】：

标签： beautifulsoup

【解决方案1】：

看起来该网站检测到请求的来源，因此我们需要“愚弄”它以为我们在浏览器上。

from bs4 import BeautifulSoup
from urllib.request import Request, urlopen

r = Request("https://es.investing.com/indices/spain-35-futures", headers={"User-Agent": "Mozilla/5.0"})
c = urlopen(r).read()
soup = BeautifulSoup(c, "html.parser")
print(soup)

【讨论】：

【解决方案2】：

Web 服务器将 python 脚本检测为机器人并因此阻止它。通过使用标题，您可以防止它发生，下面的代码可以做到：

import requests
from bs4 import BeautifulSoup

url = "https://es.investing.com/indices/spain-35-futures"

header={'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36'}
page=requests.get(url,headers=header)

soup=BeautifulSoup(page.content,'html.parser')
#this soup returns <span class="arial_26 inlineblock pid-8828-last" dir="ltr" id="last_last">9.182,5</span>

result = soup.find('span',attrs={'id':'last_last'}).get_text()
#use the get_text() function to extract the text

print(result)

【讨论】：

【解决方案3】：

您可以尝试使用 selenium 网络驱动程序。否则，如果请求数量很高，您将面临更多的问题。有时，带有 JavaScript 的网站也会出现问题。

from selenium import webdriver
url = 'https://example.com/'
options = webdriver.ChromeOptions()
options.add_argument('headless')
driver = webdriver.Chrome(options=options,executable_path='/usr/local/bin/chromedriver')
driver.get(url)

【讨论】：