网络爬虫使用 Python 提取 Href

Question

我编写了一个代码，它从指定的 url 中提取所有链接。我从在线视频教程中获得了这个想法。当我尝试使用 nytimes.com 时，如果成功了。但是当我尝试使用yell.com 时，我抛出了一个错误：“错误：HTTP 错误416：请求的范围不能满足-http://www.yell.com/”。我应该采用什么技术来绕过这个。

import urllib.parse;
import urllib;
from bs4 import BeautifulSoup;

##url = "http://nytimes.com";
url = "http://www.yell.com/";

urls = [url];   
visited = [url];

while(len(urls) > 0):

    try:
        htmltext = urllib.request.urlopen(urls[0]).read();

        soup = BeautifulSoup(htmltext);

        urls.pop(0);
        print(len(urls));

        for tag in soup.findAll('a',href=True) :
            tag['href'] = urllib.parse.urljoin(url,tag['href']);
            if(url in tag['href'] and tag['href'] not in visited) :
                urls.append(tag['href']);
                visited.append(tag['href']);

    except urllib.error.HTTPError as e:
        print("Error: " + str(e)
              + " - " + url);

print(visited);

Answer 1

这里发生的情况是，yell.com 正在检测异常活动。如果您尝试使用 selenium 直观地进行此抓取，它将加载 Javascript：

from selenium import webdriver
from bs4 import BeautifulSoup

driver = webdriver.Firefox()
driver.get(url)
driver.set_window_position(0, 0)
driver.set_window_size(100000, 200000)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(5) # wait to load

# at this point, if you see the Firefox window that opened you will see the message

# Anyway, if you manage to get pass trough that blocking, you could load BeautifulSoup this way: 
soup = BeautifulSoup(driver.page_source)