【问题标题】:how i can scrape google link when make a request for get link its say please wait after 3 second link generate and its cant scrape [duplicate]我如何在请求获取链接时抓取谷歌链接,它说请在 3 秒链接生成后等待,并且无法抓取 [重复]
【发布时间】:2019-08-14 07:07:36
【问题描述】:

http://linkblend.icu/nAK5f 这是链接

这是代码

url='http://linkblend.icu/nAK5f'
req=requests.get(url).text
soup=BeautifulSoup(req,'html.parser')
data=soup.findAll('a')
print(data)

这是结果

[<a class="navbar-brand" href="/">Link Blend</a>, <a href="/advertising-rates">Advertising</a>, <a href="/payout-rates">Payout Rates</a>, <a class="btn btn-success btn-lg get-link disabled" href="javascript: void(0)">
            Please wait...        </a>, <a href="/pages/privacy">Privacy Policy</a>, <a href="/pages/terms">Terms of Use</a>, <a href="https://www.facebook.com/#"><i class="fa fa-facebook"></i></a>, <a href="https://twitter.com/#"><i class="fa fa-twitter"></i></a>, <a href="https://plus.google.com/#"><i class="fa fa-google-plus"></i></a>]

【问题讨论】:

  • 请你告诉我这个问题在哪里也提到了。?

标签: python web-scraping


【解决方案1】:

如果您想继续使用 BeautifulSoup 来解析 html 源代码,请在 Selenium 的帮助下使用以下代码

from bs4 import BeautifulSoup as bs
from selenium import webdriver as wd

""" Initiate the firefox driver """
driver = wd.Firefox()

""" Navigate to desired url """
driver.get('site.com')

""" retrieve source """
html = driver.page_source

""" Parse via BeautifulSoup """
soup = bs(html)

""" Locate tag """
data=soup.findAll('a')
print(data)

""" Close down browser """
driver.close()

【讨论】:

  • driver=wd.Firefox() Traceback (most recent call last): File "&lt;pyshell#5&gt;", line 1, in &lt;module&gt; driver=wd.Firefox() File "C:\Program Files (x86)\Python37-32\lib\site-packages\selenium\webdriver\firefox\webdriver.py", line 163, in __init__ log_path=service_log_path) File "C:\Program Files (x86)\Python37-32\lib\site-packages\selenium\webdriver\firefox\service.py", line 44, in __init__ log_file = open(log_path, "a+") if log_path is not None and log_path != "" else None PermissionError: [Errno 13] Permission denied: 'geckodriver.log'
  • 我收到此错误权限被拒绝:' geckodriver.log '
  • 您可能需要更新:pip install --upgrade selenium
  • 是的,先生,我已经升级了仍然无法正常工作
  • 确保您的系统上安装了 Firefox,并从以下位置安装 geckodriver:github.com/mozilla/geckodriver/releases/tag/v0.24.0
猜你喜欢
  • 2021-06-22
  • 2021-08-24
  • 2019-07-19
  • 1970-01-01
  • 2012-08-29
  • 1970-01-01
  • 2017-10-03
  • 1970-01-01
  • 2017-05-29
相关资源
最近更新 更多