如何在 python-requests 中迭代多个链接？答案

【问题标题】：How to iterate multiple links in python-requests?如何在 python-requests 中迭代多个链接？
【发布时间】：2019-08-05 02:29:29
【问题描述】：

这里我只发送一个 url，page 的响应是200。并提取数据。

当我尝试使用文件发送多个链接时，response 是 400

单个网址的代码：

import requests
import lxml.html as lh
import pandas as pd
import html
from lxml import html
from bs4 import BeautifulSoup
import requests

from bs4 import BeautifulSoup
import requests
#cars = [] # gobal array for storing each car_data object
url='http://www.redbook.com.au/cars/research/used/details/1969-ford-mustang-mach-1-manual/SPOT-ITM-225034'
car_data={} # use it as a local variable
headers = {'User-Agent':'Mozilla/5.0'}
page = (requests.get(url, headers=headers))
tree = html.fromstring(page.content)
if tree.xpath('//tr[td="Badge"]//following-sibling::td[2]/text()'):
    badge = tree.xpath('//tr[td="Badge"]//following-sibling::td[2]/text()')[0]
    car_data["badge"] = badge   
if tree.xpath('//tr[td="Series"]//following-sibling::td[2]/text()'):
    car_data["series"] = tree.xpath('//tr[td="Series"]//following-sibling::td[2]/text()')[0]
if tree.xpath('//tr[td="Body"]//following-sibling::td[2]/text()'):
    car_data["body_small"] = tree.xpath('//tr[td="Body"]//following-sibling::td[2]/text()')[0]
#cars.append(car_data) #Append it to global array

输出：

{'badge': 'Mach 1', 'body_small': 'Fastback', 'series': '(No Series)'}

我尝试了多个链接的代码：

import requests
import lxml.html as lh
import pandas as pd
import html
from lxml import html
from bs4 import BeautifulSoup
import requests


cars = [] # gobal array for storing each car_data object
f = open("file.txt",'r') #file.txt would contain all the links that you wish to read
#This for loop will perform your thing for each url in the file
for url in f: 
    car_data={} # use it as a local variable
    headers = {'User-Agent':'Mozilla/5.0'}
    page = (requests.get(url, headers=headers))
    tree = html.fromstring(page.content)
    if tree.xpath('//tr[td="Badge"]//following-sibling::td[2]/text()'):
        badge = tree.xpath('//tr[td="Badge"]//following-sibling::td[2]/text()')[0]
        car_data["badge"] = badge
        print(car_data)
    if tree.xpath('//tr[td="Series"]//following-sibling::td[2]/text()'):
        car_data["series"] = tree.xpath('//tr[td="Series"]//following-sibling::td[2]/text()')[0]
    if tree.xpath('//tr[td="Body"]//following-sibling::td[2]/text()'):
        car_data["body_small"] = tree.xpath('//tr[td="Body"]//following-sibling::td[2]/text()')[0]
    cars.append(car_data) #Append it to global array

文件.txt：

http://www.redbook.com.au/cars/research/used/details/1969-ford-mustang-mach-1-manual/SPOT-ITM-225034
http://www.redbook.com.au/cars/research/used/details/1969-ford-falcon-gtho-phase-i-xw-manual/SPOT-ITM-222630
http://www.redbook.com.au/cars/research/used/details/1969-ford-falcon-xt-auto/SPOT-ITM-222613
http://www.redbook.com.au/cars/research/used/details/1969-ford-falcon-xt-manual/SPOT-ITM-222612

【问题讨论】：

标签： python html python-3.x python-requests

【解决方案1】：

您需要在列表中添加行，然后迭代此列表以实现循环。使用with open，读取完成后会自动关闭文件。

with open('file.txt') as f:
    #read file without newlines 
    urls = f.read().splitlines()
for url in urls: 
    ...

【讨论】：