bs4 python从<span></span>中提取值到.csv一遍又一遍地打印相同的结果

Question

我已经设法构建了一个非常原始的程序来从活塞头刮取车辆数据并将其打印到带有链接、品牌、型号的.csv 文件，并且正在努力获取价格，这是我遇到问题的地方。

我想将价格刮到我的.csv 文件 (Price) 的第四列，并正确打印网站上每辆车的价格。

我只是让它打印一辆车的价格，然后在.csv 文件中的每辆车旁边一次又一次地重复它。

我尝试过soup.findAll 和soup.find_all 来查看解析多个元素是否可行，但这只会造成更大的混乱。

有人可以帮忙吗？

我也在尝试抓取图像 src，并希望将其打印在名为 images 的另一列 (5) 上。

import csv ; import requests

from bs4 import BeautifulSoup

outfile = open('pistonheads.csv','w', newline='')
writer = csv.writer(outfile)
writer.writerow(["Link", "Make", "Model", "Price"])

url = 'https://www.pistonheads.com/classifieds?Category=used-cars&Page=1&ResultsPerPage=100'

get_url = requests.get(url)

get_text = get_url.text

soup = BeautifulSoup(get_text, 'html.parser')

car_link = soup.find_all('div', 'listing-headline', 'price')

for div in car_link:
   links = div.findAll('a')
      for a in links:
      link = ("https://www.pistonheads.com" + a['href'])
      make = (a['href'].split('/')[-4])
      model = (a['href'].split('/')[-3])
      price = soup.find('span')
      writer.writerow([link, make, model, price])
      print(link, make, model, price)
outfile.close()

Answer 1

你可以试试这个：

import csv, requests, re
from urllib.parse import urlparse
from bs4 import BeautifulSoup as soup
d = soup(requests.get('https://www.pistonheads.com/classifieds?Category=used-cars&ResultsPerPage=100').text, 'html.parser')
def extract_details(_s:soup) -> list:
  _link = _s.find('a', {'href':re.compile('/classifieds/used\-cars/')})['href']
  _, _, make, model, *_ = _link[1:].split('/')
  price, img = _s.find('div', {'class':'price'}).text, [i['src'] for i in _s.find_all('img')]
  return [_link, make, model, price, 'N/A' if not img else img[0]]

with open('filename.csv', 'w') as f:
  _listings = [extract_details(i) for i in d.find_all('div', {'class':'ad-listing'}) if i.find('div', {'class':'price'})]
  write = csv.writer(f)
  write.writerows([['make', 'model', 'price', 'img'], *_listings])

Answer 2

原因是因为price = soup.find('span')

.find() 将抓取它找到的第一个元素。你让它查看你的soup 对象。但是您希望它在哪里看到，在您的 a 内，因为这就是您使用 for a in links: 循环的内容

我还添加了.text，因为我假设您只想要文本，而不是整个标签元素。即price = a.find('span').text

import csv ; import requests

from bs4 import BeautifulSoup

outfile = open('pistonheads.csv','w', newline='')
writer = csv.writer(outfile)
writer.writerow(["Link", "Make", "Model", "Price", 'Images'])

url = 'https://www.pistonheads.com/classifieds?Category=used-cars&Page=1&ResultsPerPage=100'

get_url = requests.get(url)

get_text = get_url.text

soup = BeautifulSoup(get_text, 'html.parser')

car_link = soup.find_all('div', 'listing-headline', 'price')

for div in car_link:
   links = div.findAll('a')
   for a in links:
      link = ("https://www.pistonheads.com" + a['href'])
      make = (a['href'].split('/')[-4])
      model = (a['href'].split('/')[-3])
      price = a.find('span').text
      image_link = a.parent.parent.find('img')['src']
      image = link + image_link
      writer.writerow([link, make, model, price, image])
      print(link, make, model, price, image)



outfile.close()