报废问题答案

【问题标题】：Scrapping Issue报废问题
【发布时间】：2022-01-09 00:16:10
【问题描述】：

我正在尝试从以下https://www.podchaser.com/lists/best-liberal-politics-podcasts-107a9FVaYE获取数据

页面上没有验证码，但我无法抓取数据。这是我得到的汤汁。

<div class="cf-columns two">
<div class="cf-column">
<h2 data-translate="why_captcha_headline">Why do I have to complete a CAPTCHA?</h2>
<p data-translate="why_captcha_detail">Completing the CAPTCHA proves you are a human and gives you temporary access to the web property.</p>
</div>
<div class="cf-column">
<h2 data-translate="resolve_captcha_headline">What can I do to prevent this in the future?</h2>
<p data-translate="resolve_captcha_antivirus">If you are on a personal connection, like at home, you can run an anti-virus scan on your device to make sure it is not infected with malware.</p>
<p data-translate="resolve_captcha_network">If you are at an office or shared network, you can ask the network administrator to run a scan across the network looking for misconfigured or infected devices.</p>
</div>
</div>
</div>
<div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300">
<p class="text-13">
<span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">6ca95b9c3b3b7d82</strong></span>
<span class="cf-footer-separator sm:hidden">•</span>
<span class="cf-footer-item sm:block sm:mb-1"><span>Your IP</span>: 34.125.89.195</span>
<span class="cf-footer-separator sm:hidden">•</span>
<span class="cf-footer-item sm:block sm:mb-1"><span>Performance &amp; security by</span> <a href="https://www.cloudflare.com/5xx-error-landing" id="brand_link" rel="noopener noreferrer" target="_blank">Cloudflare</a></span>
</p>
</div><!-- /.error-footer -->
</div>
</div>

不知道如何克服这个问题。如果有可见的验证码，我知道如何处理。关于如何抓取此网页的任何想法。

【问题讨论】：

网站很明显不希望你这样做。他们提供API 并在terms of service 中设置限制。

标签： python web-scraping beautifulsoup

【解决方案1】：

您可以设置“用户代理”标头以通过该障碍。从实际浏览器复制用户代理值对我来说很有帮助。

如果你使用requests，你可以这样设置：

res = requests.get('https://www.podchaser.com/lists/best-liberal-politics-podcasts-107a9FVaYE',
                   headers={'User-Agent': 'Mozilla/5.0 ...'})

【讨论】：