如何用循环刮掉多个页面的麻烦？

Question

这是我只抓取一页的代码，但我有 11000 个。区别在于他们的身份。

https://www.rlsnet.ru/mkb_index_id_1.htm
https://www.rlsnet.ru/mkb_index_id_2.htm
https://www.rlsnet.ru/mkb_index_id_3.htm
....
https://www.rlsnet.ru/mkb_index_id_11000.htm

如何循环我的代码来抓取所有 11000 页？如此大量的页面甚至可能吗？可以将它们放入一个列表中，然后进行抓取，但如果有 11000 个，那将是很长的路要走。

import requests
from pandas import DataFrame
import numpy as np
import pandas as pd
from bs4 import BeautifulSoup

page_sc = requests.get('https://www.rlsnet.ru/mkb_index_id_1.htm')
soup_sc = BeautifulSoup(page_sc.content, 'html.parser')
items_sc = soup_sc.find_all(class_='subcatlist__item')
mkb_names_sc = [item_sc.find(class_='subcatlist__link').get_text() for item_sc in items_sc]
mkb_stuff_sce = pd.DataFrame(
    {
        'first': mkb_names_sc,
    })
mkb_stuff_sce.to_csv('/Users/gfidarov/Desktop/Python/MKB/mkb.csv')

Answer 1

您可以像这样动态创建 url 字符串。您可能还希望在循环的每个其他迭代中使用时间延迟，以免被服务器阻塞。

import requests
from pandas import DataFrame
import numpy as np
import pandas as pd
from bs4 import BeautifulSoup


path_of_csv = '/Users/gfidarov/Desktop/Python/MKB/mkb.csv'

first_string = 'https://www.rlsnet.ru/mkb_index_id_'
third_string = '.htm'

df = pd.DataFrame(columns=['scraping results'])

try:
    for second_string in range(1, 11001):
        second_string = str(second_string)
        url = first_string + second_string + third_string
        page_sc = requests.get(url)
        soup_sc = BeautifulSoup(page_sc.content, 'html.parser')
        items_sc = soup_sc.find_all(class_='subcatlist__item')
        mkb_names_sc = [item_sc.find(class_='subcatlist__link').get_text() for item_sc in items_sc]
        df.append({'scraping results': mkb_names_sc}, ignore_index=True)

    df.to_csv(
        path_or_buf=path_of_csv
    )

except:
    # If it fails in the middle of the process, the results won't be lost
    path_of_csv = 'backup_' + path_of_csv
    df.to_csv(
        path_or_buf=path_of_csv 
    )
    print('Failed at index ' + second_string + '. Please start from here again by setting the beginning of the range to this index. A backup was made of the results that were already scraped. You may want to rename the backup to avoid overwriting in the next run.')

Answer 2

我的方法很简单。我只是在循环上面的代码。

for i in range(1,11001):

    page_sc = requests.get('https://www.rlsnet.ru/mkb_index_id_{}.htm'.format(i))

    soup_sc = BeautifulSoup(page_sc.content, 'html.parser')
    items_sc = soup_sc.find_all(class_='subcatlist__item')
    mkb_names_sc = [item_sc.find(class_='subcatlist__link').get_text() for item_sc in items_sc]
    mkb_stuff_sce = pd.DataFrame(
        {
            'first': mkb_names_sc,
        })
    mkb_stuff_sce.to_csv('/Users/gfidarov/Desktop/Python/MKB/mkb.csv')

我所做的是，我使用 for 循环遍历代码，range() 函数正在生成 index 列表，我使用 format() 方法将其放入 url。

这应该像一个魅力。希望这会有所帮助:)