【问题标题】:Grabbing an item from a page returns None in python beautifulsoup从页面抓取项目在 python beautifulsoup 中返回 None
【发布时间】:2021-10-13 17:11:23
【问题描述】:

我正在尝试获取文本“一个解锁的钱包持有 100.00% 的 SMM/BNB LP(点击查看 LP 持有者)”但是我不确定我是否做得对。

import requests, re, random
from bs4 import BeautifulSoup
from urllib.request import Request, urlopen


user_agent_list = [
"header = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:86.0Gecko/20100101 Firefox/86.0'}",
"header = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36'}",
"header = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Safari/605.1.15'}",
"header = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36'}",
"header = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36'}",
"header = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36'}"
]

header = random.choice(user_agent_list)

tokenurl = "https://poocoin.app/tokens/0xb393d44b9aaecf946925c0198b71f2a5163ce78b"

page = requests.get(tokenurl,header)
token = BeautifulSoup(page.content, 'html.parser')
website = token.find('a', class_='text-danger bg-dark rounded p-2 mt-1 d-inline-block text-decoration-underline')
print (website)

电流输出:

None

【问题讨论】:

  • 我不是很喜欢bs4,但是我觉得这个需要.text,可能是print (website.text)
  • 点击 URL 时得到的响应是什么?我猜是 403,对吧?
  • 我在 python 中不是很好。我所看到的只是输出 None 这意味着有些事情是不正确的
  • 当我运行相同的代码时,它给了我一个 403。
  • 使用这个顺便说一句你想找到什么Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36

标签: python-3.x selenium web-scraping beautifulsoup


【解决方案1】:

当您使用BeautifulSoup 时,您需要了解当您给它一个html 文档时它可以给出什么输出。这是您在问题中使用代码获得的整个文档,即变量 token 中存在的内容。

<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!-->
<html class="no-js" lang="en-US">
 <!--<![endif]-->
 <head>
  <title>
   Access denied | poocoin.app used Cloudflare to restrict access
  </title>
  <meta charset="utf-8"/>
  <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
  <meta content="IE=Edge,chrome=1" http-equiv="X-UA-Compatible"/>
  <meta content="noindex, nofollow" name="robots"/>
  <meta content="width=device-width,initial-scale=1" name="viewport"/>
  <link href="/cdn-cgi/styles/main.css" id="cf_styles-css" media="screen,projection" rel="stylesheet" type="text/css"/>
  <script async="" src="/cdn-cgi/bm/cv/669835187/api.js">
  </script>
 </head>
 <body>
  <div id="cf-wrapper">
   <div class="cf-alert cf-alert-error cf-cookie-error hidden" data-translate="enable_cookies" id="cookie-alert">
    Please enable cookies.
   </div>
   <div class="p-0" id="cf-error-details">
    <header class="mx-auto pt-10 lg:pt-6 lg:px-8 w-240 lg:w-full mb-15 antialiased">
     <h1 class="inline-block md:block mr-2 md:mb-2 font-light text-60 md:text-3xl text-black-dark leading-tight">
      <span data-translate="error">
       Error
      </span>
      <span>
       1020
      </span>
     </h1>
     <span class="inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed">
      Ray ID: 67c75596dc9cdca5 •
     </span>
     <span class="inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed">
      2021-08-10 06:58:11 UTC
     </span>
     <h2 class="text-gray-600 leading-1.3 text-3xl lg:text-2xl font-light">
      Access denied
     </h2>
    </header>
    <section class="w-240 lg:w-full mx-auto mb-8 lg:px-8">
     <div class="w-1/2 md:w-full" id="what-happened-section">
      <h2 class="text-3xl leading-tight font-normal mb-4 text-black-dark antialiased" data-translate="what_happened">
       What happened?
      </h2>
      <p>
       This website is using a security service to protect itself from online attacks.
      </p>
     </div>
    </section>
    <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300">
     <p class="text-13">
      <span class="cf-footer-item sm:block sm:mb-1">
       Cloudflare Ray ID:
       <strong class="font-semibold">
        67c75596dc9cdca5
       </strong>
      </span>
      <span class="cf-footer-separator sm:hidden">
       •
      </span>
      <span class="cf-footer-item sm:block sm:mb-1">
       <span>
        Your IP
       </span>
       : 103.162.8.200
      </span>
      <span class="cf-footer-separator sm:hidden">
       •
      </span>
      <span class="cf-footer-item sm:block sm:mb-1">
       <span>
        Performance &amp; security by
       </span>
       <a href="https://www.cloudflare.com/5xx-error-landing" id="brand_link" rel="noopener noreferrer" target="_blank">
        Cloudflare
       </a>
      </span>
     </p>
    </div>
    <!-- /.error-footer -->
   </div>
   <!-- /#cf-error-details -->
  </div>
  <!-- /#cf-wrapper -->
  <script type="text/javascript">
   window._cf_translation = {};
  </script>
  <script type="text/javascript">
   (function(){window['__CF$cv$params']={r:'67c75596dc9cdca5',m:'7d5cb070f52bc2658c47cfc88a34832f4146bc1d-1628578691-1800-AR1RssYylCV1f2kIgs0CPVdq4ZoKz1sRl9UYE21Ha1aj3nICL+7XnFzztlixt0HmssNjZllMWXNfOzKg8lz3ARfY0damLC5V/icRyggdI3F8p8ff60JVxyOhbPddj1CHEN1tln1DXTGeD22FuS1QMeDRr71Zt702lXjmpb5+pIM7TRuE62iKa+JcTuP0oC3ajFUxmStjY0GhQ+8F1LcKVXuP54xCzOzN5t8t87Bx7KDyu6Bn9pOew7YNYJRrDxaFUW7UqbF3XNw6yb5HqE+LCikS1PxIUKa3s3SRSgsFiyzz884GhqB2dHZo474r19GcyY1JYGe2BohY8p4/oPnGO/mgGKJ6tlbBkuluL0lohBVDtbLAUo0atxlDUxIgooWbVZshvx8x0F99vHzlB0AriSihU/oGlATjttPFIcr+6gvk',s:[0x3f2b1d591e,0x8f03d56274],}})();
  </script>
  <script data-cf-beacon='{"rayId":"67c75596dc9cdca5","token":"b7a3c0b8fa4646c3a017dc0af118d444","version":"2021.7.0","si":10}' defer="" src="https://static.cloudflareinsights.com/beacon.min.js">
  </script>
 </body>
</html>

上面的整个文档中只有一个a 标签,如下所示:

<a href="https://www.cloudflare.com/5xx-error-landing" id="brand_link" rel="noopener noreferrer" target="_blank">
Cloudflare 
</a>

从而使您的输出None。您必须使用SeleniumScrapy 在这里获得您想要的输出,或者您可以提供一个输入,使BeautifulSoup 提供您想要的输出。

单独使用Selenium

from webdriver_manager.chrome import ChromeDriverManager
from selenium import webdriver

option = webdriver.ChromeOptions()
# option.add_argument('--headless')
option.add_argument("--log-level=3")
option.add_experimental_option('excludeSwitches', ['enable-logging'])


CDM = ChromeDriverManager(log_level='0')
driver = webdriver.Chrome(CDM.install(), options=option)

tokenurl = "https://poocoin.app/tokens/0xb393d44b9aaecf946925c0198b71f2a5163ce78b"
driver.get(tokenurl)
time.sleep(2)
a = driver.find_element_by_class_name('text-danger.bg-dark.rounded.p-2.mt-1.d-inline-block.text-decoration-underline').text
print(a)
driver.quit()

通过SeleniumBS 提供更好的输入:

from webdriver_manager.chrome import ChromeDriverManager
from selenium import webdriver
from bs4 import BeautifulSoup

option = webdriver.ChromeOptions()
# option.add_argument('--headless')
option.add_argument("--log-level=3")
option.add_experimental_option('excludeSwitches', ['enable-logging'])


CDM = ChromeDriverManager(log_level='0')
driver = webdriver.Chrome(CDM.install(), options=option)

tokenurl = "https://poocoin.app/tokens/0xb393d44b9aaecf946925c0198b71f2a5163ce78b"
driver.get(tokenurl)
time.sleep(2)
token = BeautifulSoup( driver.page_source, 'html.parser')
driver.quit()
website = token.find('a', class_='text-danger bg-dark rounded p-2 mt-1 d-inline-block text-decoration-underline')
print(website.text)

两种情况都会有相同的输出:

An unlocked wallet is holding 100.00% of the SMM/BNB LP (click to view LP holders)

因此,您得到 None 作为输出的原因是因为 token 变量中没有足够的数据。您可以将最后一个代码块中的 token 变量与原始代码中的变量进行比较。这是因为如今的网页是动态的,因此我们必须使用可以简化它们的工具。

【讨论】:

  • selenium 还是 scrappy 哪个更好?
  • 这完全取决于你在做什么项目。使用BeautifulSoupSelenium 可以完成简单和小型的项目。与Selenium 相比,使用Scrapy 可以更快地完成复杂和更大的任务。有时我们需要混合这两个库中的所有或任何一个。 `最好按以下顺序学习这些:BS、Selenium、Scrapy。
猜你喜欢
  • 1970-01-01
  • 2022-01-10
  • 2020-03-12
  • 1970-01-01
  • 1970-01-01
  • 2020-03-11
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
相关资源
最近更新 更多