Python3 beautifulsoup4 和 selenium答案

【问题标题】：Python3 beautifulsoup4 and seleniumPython3 beautifulsoup4 和 selenium
【发布时间】：2020-07-13 02:35:05
【问题描述】：

我编写了这段代码，用于从 livecore.com 抓取分数详细信息。但我有一些问题。也许我写了不正确的代码。请帮帮我。

代码运行输出：

Traceback (most recent call last):
  File "web.py", line 15, in <module>
    box2 = box.find_all('a',{'class' : 'match-row scorelink'})
AttributeError: 'NoneType' object has no attribute 'find_all'


from bs4 import BeautifulSoup
import requests
from selenium import webdriver

driver = webdriver.Chrome()
driver.get('https://livescore.com')
res = driver.execute_script("return document.documentElement.outerHTML")
driver.quit()
#page = requests.get('https://livescore.com')
soup = BeautifulSoup(res, 'lxml')


box = soup.find('div',{'class':'container'})

box2 = box.find_all('a',{'class' : 'match-row scorelink'})

for data in box2:
    test = data.find('div',{'class': 'sco'}).text.replace('\n', '')
print (test)

【问题讨论】：

标签： python-3.x selenium web-scraping beautifulsoup

【解决方案1】：

感谢您的回答。解决问题


from bs4 import BeautifulSoup
import requests
from selenium import webdriver

driver = webdriver.Chrome()
driver.get('https://livescore.com')
res = driver.execute_script("return document.documentElement.outerHTML")
driver.quit()
#page = requests.get('https://livescore.com')
soup = BeautifulSoup(res, 'lxml')


box = soup.find('div',{'data-type':'container'})

box2 = box.find_all('a',{'class' : 'match-row'})

for data in box2:
    test1 = data.find('div',{'class': 'sco'}).text.replace('\n', '')
    test2 = data.find('div',{'class': 'ply tright name'}).text.replace('\n', '')
    test3 = data.find('div',{'class': 'ply name'}).text.replace('\n', '')
    print(test2,test1,test3)

【讨论】：

【解决方案2】：

试一试。我跳过了“box2”，因为它并不是真正需要获得分数。另外，从我获取的数据来看，.replace('\n', '') 也不需要，但如果你认为你会得到包含“\n”字符的分数，请随意使用它。

from bs4 import BeautifulSoup
from selenium import webdriver

driver = webdriver.Chrome()
driver.get('https://livescore.com')
res = driver.execute_script("return document.documentElement.outerHTML")
driver.quit()

soup = BeautifulSoup(res, 'lxml')
box = soup.find('div',{'data-type':'container'})
scores=box.find_all('div',{'class': 'sco'})

for score in scores:
    print(score.text)

【讨论】：

【解决方案3】：

使用下面的css选择器。但是container不是类属性值。它的data-type='container'属性值。

from bs4 import BeautifulSoup
from selenium import webdriver

driver = webdriver.Chrome()
driver.get('https://livescore.com')
res = driver.execute_script("return document.documentElement.outerHTML")
driver.quit()
soup = BeautifulSoup(res, 'lxml')
for item in soup.select("div[data-type='container'] .match-row.scorelink>.sco"):
    test=item.text.replace('\n', '')
    print(test)

【讨论】：

【解决方案4】：

试试这个：

from bs4 import BeautifulSoup
import requests
from selenium import webdriver

driver = webdriver.Chrome()
driver.get('https://livescore.com')
#page = requests.get('https://livescore.com')
soup = BeautifulSoup(driver.page_source, 'lxml')
driver.quit()

box = soup.find('div',{'class':'container'})

box2 = box.find_all('a',{'class' : 'match-row scorelink'})

for data in box2:
    test = data.find('div',{'class': 'sco'}).text.replace('\n', '')
print (test)

【讨论】：