【问题标题】:New to webscraper and looping data unexpectedlywebscraper 的新手和意外循环数据
【发布时间】:2020-03-28 18:30:34
【问题描述】:

我刚刚完成了一些编程入门课程,主要是 Python,我正在尝试编写一个基本的抓取工具来从特定站点提取使用过的特斯拉的数据(稍后添加差异站点并存储在数据库中并执行使用 SQL 进行一些数据分析等),我马上就遇到了问题。

我正在尝试提取 8 条信息,最后 3 条信息在同一个 div 中,而不是循环显示每辆车的信息(上市日期、位置和里程),而是在每个循环中将所有汽车的信息吐回.

我知道问题出在我的循环代码中,从“for listing_details”开始,但每当我尝试调整不同的元素时,我都会用回溯错误将其炸毁。

这是下面的代码,我真诚地希望有人能指出我明显的疏忽!

from bs4 import BeautifulSoup
import requests

source = requests.get('https://onlyusedtesla.com/listings/?_sft_model=model-s&_sfm_out_asking_price=699+150004&_sfm_state=Arizona').text

soup = BeautifulSoup(source, 'lxml')

for listing in soup.find_all('div', class_='out-list-item'):

    listing_info = listing.h2.a.text
    listing_separate = listing_info.split("/")

    for list in listing_separate:
        print(list)


    price_info = listing.find('div', class_='asking-price')
    price = price_info.text
    p = price.split(' ')
    print(p[3], p[10])


    for listing_details in soup.find_all('div', class_='out-summary-data small-12 medium-4 columns'):

        details = listing_details
        listing_date = details.find_all("p")[0]
        ld = listing_date.text
        print(ld)
        location = details.find_all("p")[1]
        loc = location.text
        print(loc)
        mileage = details.find_all("p")[3]
        mil = mileage.text
        print(mil)

【问题讨论】:

  • 你的预期输出是什么?

标签: python python-3.x loops web-scraping beautifulsoup


【解决方案1】:
import requests
from bs4 import BeautifulSoup

r = requests.get(
    'https://onlyusedtesla.com/listings/?_sft_model=model-s&_sfm_out_asking_price=699+150004&_sfm_state=Arizona')
soup = BeautifulSoup(r.text, 'html.parser')

h2 = []
asking = []
details = []
for item in soup.findAll('div', attrs={'class': 'small-12 large-8 columns'}):
    for name in item.findAll('h2'):
        h2.append(name.text)
for item in soup.findAll('div', attrs={'class': 'asking-price'}):
    item = item.text.replace("       ", "")
    asking.append(item[15:])
for item in soup.findAll('div', attrs={'class': 'out-summary-data small-12 medium-4 columns'}):
    item = [name.get_text(strip=True) for name in item.findAll('p')]
    details.append(item)

for a, b, c in zip(h2, asking, details):
    print(a, b, *c)

输出:

  2015 / Model S / P90DL / Dark Blue  55,000.00 Nov 21, 2019 Gilbert, Arizona Excellent 76,000 miles
  2015 / Model S / 85 / White  39,950.00 Nov 20, 2019 GOODYEAR, Arizona Excellent 100,500 miles
  2017 / Model S / 75 / Pearl White  63,000.00 Nov 11, 2019 Tempe, Arizona Excellent 20,436 miles
  2016 / Model S / 75 / Midnight Silver Metallic  48,000.00 Oct 6, 2019 Phoenix, Arizona Excellent 27,703 miles
  2017 / Model S / 90D / Metallic Blue  62,950.00 Oct 2, 2019 Chandler, Arizona Excellent 48,000 miles
  2017 / Model S / 75D / Midnight Silver  57,000.00 Aug 10, 2019 Phoenix, Arizona Excellent 26,500 miles
  2014 / Model S / 85 / Grey Metallic  39,950.00 Aug 3, 2019 Gilbert, Arizona Very Good 68,861 miles
  2017 / Model S / 100D / Deep Blue Metallic  72,500.00 Jul 1, 2019 Phoenix, Arizona Excellent 17,000 miles
  2016 / Model S / 75D / Midnight Silver  49,495.00 Feb 24, 2019 Gilbert, Arizona Excellent 20,157 miles
  2012 / Model S / P85 / Black  32,000.00 Feb 19, 2019 Mesa, Arizona Very Good 133,000 miles
  2018 / Model S / 75D / Red  74,950.00 Feb 5, 2019 Tucson, Arizona Excellent 8,964 miles
  2014 / Model S / P85D / Midnight Silver  70,950.00 Sep 13, 2018 Phoenix, Arizona Excellent 13,800 miles
  2013 / Model S / 60 / Red  37,500.00 Jun 27, 2018 Chandler, Arizona Good 73,536 miles
  2018 / Model S / 75D / Blue. EAP  72,950.00 Jun 20, 2018 Queen Creek, Arizona Excellent 2,279 miles
  2014 / Model S / P85+ / Black  69,900.00 Jun 18, 2018 Gilbert, Arizona Excellent 48,500 miles
  2014 / Model S / 85 / Blue  44,950.00 May 3, 2018 Tempe, Arizona Excellent 68,000 miles
  2013 / Model S / 85 / Silver  41,000.00 Apr 18, 2018 Chandler, Arizona Excellent 59,000 miles
  2013 / Model S / 85 / White  49,850.00 Apr 10, 2018 Queen Creek, Arizona Excellent 39,937 miles
  2014 / Model S / 85 / Dark Blue. Autopilot  57,000.00 Mar 28, 2018 Mesa, Arizona Excellent 59,077 miles
  2014 / Model S / P85+ / Blue  58,000.00 Mar 18, 2018 Phoenix, Arizona Excellent 51,309 miles

【讨论】:

  • 太棒了!非常感谢你的帮助!点赞!
  • 完成。再次感谢。对不起,这个网站是全新的。
  • 欢迎您!很高兴为您提供帮助;)欢迎来到我们的社区
猜你喜欢
  • 2021-06-19
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2021-12-07
  • 1970-01-01
  • 1970-01-01
相关资源
最近更新 更多