使用 Beautiful Soup 从站点抓取数据时无法获取完整的 HTML答案

【问题标题】：Cannot Get Full HTML When Scraping Data from Site with Beautiful Soup使用 Beautiful Soup 从站点抓取数据时无法获取完整的 HTML
【发布时间】：2021-04-24 21:08:38
【问题描述】：

我正在尝试 scrape this site 但我得到了 HTML 的一部分。我想得到token的价格

它应该包含在里面

<span class="text-success">$0.0000000218121</span>

这是抓取结果

<html lang="en"><head><meta charset="utf-8"/><link href="https://bsc-dataseed1.defibit.io" rel="preconnect"/><link href="https://bsc-dataseed1.defibit.io" rel="preconnect"/><link href="https://bsc-dataseed1.defibit.io" rel="preconnect"/><link href="https://poocoin.app/favicon.ico" rel="shortcut icon"/><link href="https://poocoin.app/images/logo/poocoin512.png" rel="apple-touch-icon"/><link href="https://poocoin.app/manifest.json" rel="manifest"/><title>PooCoin BSC Charts</title><meta content="PooCoin live streaming charts DAPP for Binance Smart Chain (BSC) tokens." name="description"/><meta content="width=device-width,initial-scale=1,maximum-scale=1,minimum-scale=1" name="viewport"/><meta content="#000000" name="theme-color"/><meta content="https://poocoin.app/images/logo/poocoin512.png" name="twitter:image"/><meta content="PooCoin live streaming charts DAPP for Binance Smart Chain (BSC) tokens." name="twitter:description"/><meta content="summary" name="twitter:card"/><meta content="PooCoin Charts" name="twitter:title"/><link href="https://unpkg.com/aos@2.3.1/dist/aos.css" rel="stylesheet"/><link crossorigin="anonymous" href="https://pro.fontawesome.com/releases/v5.10.0/css/all.css" integrity="sha384-AYmEC3Yw5cVb3ZcuHtOA93w35dYTsvhLPVnYs9eStHfGJvOvKxVfELGroGkvsg+p" rel="stylesheet"/><script src="/charting_library/charting_library.js"></script><script src="/datafeeds/udf/dist/polyfills.js"></script><script src="/datafeeds/udf/dist/bundle.js"></script><script async="" src="https://www.googletagmanager.com/gtag/js?id=G-JKD153X2H2"></script><script>function gtag(){dataLayer.push(arguments)}window.dataLayer=window.dataLayer||[],gtag("js",new Date),gtag("config","G-JKD153X2H2")</script><link href="https://fonts.gstatic.com" rel="preconnect"/><link href="https://fonts.googleapis.com/css2?family=Lato:ital,wght@0,100;0,300;0,400;0,700;0,900;1,100;1,300;1,400;1,700;1,900&amp;display=swap" rel="stylesheet"/><link href="https://poocoin.app/static/css/3.ec734770.chunk.css" rel="stylesheet"/><link href="https://poocoin.app/static/css/main.850a7f96.chunk.css" rel="stylesheet"/></head><body data-aos-delay="0" data-aos-duration="400" data-aos-easing="ease-out-cubic"><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div><script async="" src="https://appsha1.cointraffic.io/js/?wkey=ANDNNe0Jx9"></script><script>!function(e){function t(t){for(var n,u,i=t[0],c=t[1],f=t[2],p=0,s=[];p<i.length;p++)u=i[p],Object.prototype.hasOwnProperty.call(o,u)&&o[u]&&s.push(o[u][0]),o[u]=0;for(n in c)Object.prototype.hasOwnProperty.call(c,n)&&(e[n]=c[n]);for(l&&l(t);s.length;)s.shift()();return a.push.apply(a,f||[]),r()}function r(){for(var e,t=0;t<a.length;t++){for(var r=a[t],n=!0,i=1;i<r.length;i++){var c=r[i];0!==o[c]&&(n=!1)}n&&(a.splice(t--,1),e=u(u.s=r[0]))}return e}var n={},o={2:0},a=[];function u(t){if(n[t])return n[t].exports;var r=n[t]={i:t,l:!1,exports:{}};return e[t].call(r.exports,r,r.exports,u),r.l=!0,r.exports}u.e=function(e){var t=[],r=o[e];if(0!==r)if(r)t.push(r[2]);else{var n=new Promise((function(t,n){r=o[e]=[t,n]}));t.push(r[2]=n);var a,i=document.createElement("script");i.charset="utf-8",i.timeout=120,u.nc&&i.setAttribute("nonce",u.nc),i.src=function(e){return u.p+"static/js/"+({}[e]||e)+"."+{0:"6635f59a",4:"ac5da336",5:"07770253",6:"dba614ec",7:"a4b96383",8:"5a8a1858",9:"9f6657f5",10:"4e40d844",11:"15e0af30"}[e]+".chunk.js"}(e);var c=new Error;a=function(t){i.onerror=i.onload=null,clearTimeout(f);var r=o[e];if(0!==r){if(r){var n=t&&("load"===t.type?"missing":t.type),a=t&&t.target&&t.target.src;c.message="Loading chunk "+e+" failed.\n("+n+": "+a+")",c.name="ChunkLoadError",c.type=n,c.request=a,r[1](c)}o[e]=void 0}};var f=setTimeout((function(){a({type:"timeout",target:i})}),12e4);i.onerror=i.onload=a,document.head.appendChild(i)}return Promise.all(t)},u.m=e,u.c=n,u.d=function(e,t,r){u.o(e,t)||Object.defineProperty(e,t,{enumerable:!0,get:r})},u.r=function(e){"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},u.t=function(e,t){if(1&t&&(e=u(e)),8&t)return e;if(4&t&&"object"==typeof e&&e&&e.__esModule)return e;var r=Object.create(null);if(u.r(r),Object.defineProperty(r,"default",{enumerable:!0,value:e}),2&t&&"string"!=typeof e)for(var n in e)u.d(r,n,function(t){return e[t]}.bind(null,n));return r},u.n=function(e){var t=e&&e.__esModule?function(){return e.default}:function(){return e};return u.d(t,"a",t),t},u.o=function(e,t){return Object.prototype.hasOwnProperty.call(e,t)},u.p="https://poocoin.app/",u.oe=function(e){throw console.error(e),e};var i=this.webpackJsonpcharts=this.webpackJsonpcharts||[],c=i.push.bind(i);i.push=t,i=i.slice();for(var f=0;f<i.length;f++)t(i[f]);var l=c;r()}([])</script><script src="https://poocoin.app/static/js/3.722756de.chunk.js"></script><script src="https://poocoin.app/static/js/main.2267a016.chunk.js"></script></body></html>

这是代码

from bs4 import BeautifulSoup
from urllib.request import Request, urlopen
from selenium import webdriver
import requests

def scraping(url)
    browser = webdriver.PhantomJS()
    browser.get(url)
    html = browser.page_source
    return BeautifulSoup(html, 'lxml')

price = scraping("https://poocoin.app/tokens/0x78bc22a215c1ef8a2e41fa1c39cd7bdc09bd5174")
print(price)

我一无所获

scraping.findAll("span")

【问题讨论】：

标签： python selenium web-scraping beautifulsoup

【解决方案1】：

试试这个：

from bs4 import BeautifulSoup
from urllib.request import Request, urlopen
from selenium import webdriver
import requests
import time  # New import

def scraping(url)
    browser = webdriver.PhantomJS()
    browser.get(url)
    time.sleep(5)  # 5 seconds
    html = browser.page_source
    return BeautifulSoup(html, 'lxml')

# Get Html
page = scraping("https://poocoin.app/tokens/0x78bc22a215c1ef8a2e41fa1c39cd7bdc09bd5174")

# Extract price as str
price = page.find("span", class_="text-success").getText()

print(price)  # Outputs $3.74

【讨论】：

price = page.find("span", class_="text-success") 结果返回 None
这是因为页面加载需要时间，并且您获取 html 的速度太快。查看我更新的答案（我刚刚成功运行）。
它还能用吗？我试过了，它给了我 AttributeError: 'NoneType' object has no attribute 'getText' 我也延长了时间
是的，我再次运行它没有问题。

【解决方案2】：

PooCoin 限制了 phantomJS 视图，最好改用这个：

from bs4 import BeautifulSoup
from selenium import webdriver
import time
from pyvirtualdisplay import Display
import telegram_send
from api.constants import *

coin_address = '0x4e8a9d0bf525d78fd9e0c88710099f227f6924cf'


def scraping(url):
    display = Display(visible=0, size=(1200, 1200))
    display.start()
    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_argument('--disable-extensions')
    chrome_options.add_argument('--profile-directory=Default')
    chrome_options.add_argument("--incognito")
    chrome_options.add_argument("--disable-plugins-discovery")
    print(chrome_driver_dir)
    driver = webdriver.Chrome(executable_path=chrome_driver_dir, options=chrome_options)
    driver.delete_all_cookies()
    driver.get(url)

    time.sleep(5)  # 5 seconds
    html = driver.page_source
    display.stop()

    return BeautifulSoup(html, 'lxml')

# Get Html
page = scraping("https://poocoin.app/tokens/" + coin_address)

# Extract price as str
prices = page.find_all("span", class_="text-success")
# the element position always changes
price = prices[7].getText()
print(price)

【讨论】：