网络抓取时如何使用 \"for\" 来 \"循环\" 多个 url？ [关闭]

Question

当前的代码非常适合仅抓取一个 Url 的信息，我希望能够一次从多个 URL 抓取（可能使用 For url in Urls），这是下面仅一个 URL 的当前代码。请提供任何帮助或指导，我们将不胜感激

import datetime
from bs4 import BeautifulSoup
import requests

def get_url_data_from_url_request(url):
 print(">> get_url_data_from_url_request: "+str(url))
 url_data = None
 headers = {"user-agent": "Mozilla/5.0 (Windows NT 
 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like 
 Gecko) Chrome/90.0.4430.93 Safari/537.36"}
 s = requests.session()
 s.keep_alive = False
 request = s.get(url, proxies=None, headers=headers)
 print("request.status_code: ", request.status_code )
 url_data = request.text
 request.connection.close()
 s.close()
 return url_data

def main():
 print("bdr.sandbox")
 generated_on = datetime.datetime.now()
 print(generated_on)
 source_product_url = ""
 url_data = get_url_data_from_url_request(url=source_product_url)
 soup = BeautifulSoup(url_data, "lxml")
 id_element = soup.find('span', {"itemprop": "sku"}).text
 print(id_element)
 
if __name__ == '__main__':
main()

Answer 1

有无数种方法可以做到；这是一个示例：在 main 中参数化 source_product_url 并在循环中调用它。 （如果您想保存数据以供将来使用，请添加return id_element。）

def get_sku(source_product_url):
 print("bdr.sandbox")
 generated_on = datetime.datetime.now()
 print(generated_on) 
 url_data = get_url_data_from_url_request(url=source_product_url)
 soup = BeautifulSoup(url_data, "lxml")
 id_element = soup.find('span', {"itemprop": "sku"}).text
 print(id_element)
 return id_element

此时你可以只做for url in urlsList: get_sku(url)，但是如果你想收集一些数据[并且还要确保，即使在抓取某些页面时出现任何错误，程序仍然会尝试抓取其余部分]，然后你可以做类似的事情

def try_get_sku(srcProdUrl):
 try: return {'source_product_url': srcProdUrl, 'sku': get_sku(srcProdUrl)}
 except Exception as e:
  errorMsg = f'{type(e)} {e}'
  print(f'failed to scrape {srcProdUrl} 
 ', errorMsg)
  return {'source_product_url': srcProdUrl, 'errorMsg': errorMsg} 

（这也方便list comprehension）

def main():
 spUrls = [
  'https://www.6pm.com/p/easy-spirit-epic-gray/product/9450972/color/11',
  ## ADD ALL URLS TO SCRAPE ##
 ]
 productsInfo = [try_get_sku(spUrls) for url in spUrls]
 # do something with productsInfo if you want to, like
 # pandas.DataFrame(productsInfo).to_csv('productsInfo.csv') # saves in csv file [import pandas]