抓取 goodreads.com 时被阻止

Question

我正在尝试从“https://www.goodreads.com/book/show/”上获取大量书籍样本（100k+），但我一直被阻止。 到目前为止，我已经尝试在我的代码中实现以下解决方案：

• 检查 robots.txt 以查找哪些站点/元素无法访问

• 指定一个或多个随机变化的标题

• 使用多个工作代理以避免被阻止

• 在每次使用 10 个并发线程的抓取迭代之间设置最长 20 秒的延迟

这是一个简化版本的代码，在尝试仅抓取书名和作者时被阻止，而不使用多个同时线程：

import requests
from lxml import html
import random

proxies_list = ["http://89.71.193.86:8080", "http://178.77.206.21:59298", "http://79.106.37.70:48550",
                "http://41.190.128.82:47131", "http://159.224.109.140:38543", "http://94.28.90.214:37641",
                "http://46.10.241.140:53281", "http://82.147.120.30:56281", "http://41.215.32.86:55561"]
proxies = {"http": random.choice(proxies_list)}

# real header
# headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'}

# multiple headers
headers_list = ['Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36',
                'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36',
                'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36',
                'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.38 Safari/537.36',
                'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36',
                'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36',
                'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1623.0 Safari/537.36',
                'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36']
headers = {"user-agent": random.choice(headers_list)}

first_url = 1
last_url = 10000     # Last book is 8,630,000
sleep_time = 20

for book_reference_number in range(first_url, last_url):
    try:
        goodreads_html = requests.get("https://www.goodreads.com/book/show/" + str(book_reference_number), timeout=5, headers=headers, proxies=proxies)
        doc = html.fromstring(goodreads_html.text)
        book_title = doc.xpath('//div[@id="topcol"]//h1[@id="bookTitle"]')[0].text.strip(", \t\n\r")
        try:
            author_name = doc.xpath('//div[@id="topcol"]//a[@class="authorName"]//span')[0].text.strip(", \t\n\r")
        except:
            author_name = ""
        time.sleep(sleep_time)
        print(str(book_reference_number), book_title, author_name)
    except:
        print(str(book_reference_number) + " cannot be scraped.")
        pass

Answer 1

如果你真的想刮大数据库，那么我会推荐 selenium，被阻塞的机会会很低，而且很稳定。不需要time.sleep()（时间延迟，但您可以添加以使其更稳定）。检查下面的代码...

import time
from bs4 import BeautifulSoup
from selenium import webdriver
##copy chromedriver into python folder
driver = webdriver.Chrome()
#driver.set_window_position(-2000,0)#this function will minimize the window
first_url = 1
last_url = 10000     # Last book is 8,630,000

for book_reference_number in range(first_url, last_url):
    driver.get("https://www.goodreads.com/book/show/"+str(book_reference_number))
    #time.sleep(2)#optional
    soup = BeautifulSoup(driver.page_source, 'lxml')
    try:
        book_title = soup.select('.gr-h1.gr-h1--serif')[0].text.strip()
    except:
        book_title = ''
    try:
        author_name = soup.select('.authorName')[0].text.strip()
    except:
        author_name = ''

    print('NO.', book_reference_number, 'TITLE: ', book_title, 'AUTHOR: ', author_name)