【发布时间】：2022-11-07 19:38:23
【问题描述】：
我需要在出租房屋的 Zillow 上对加利福尼亚州的 3 页进行网络抓取，并将所有数据放入熊猫数据框中。我需要提取每个列表的所有特征 - 地址、城市、卧室和浴室的数量、房子的大小、地块的大小、建造年份、租金价格、租金日期
我的代码：
from bs4 import BeautifulSoup
import requests
import time
import os
import random
import re
!pip install selenium
!pip install webdriver-manager
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument('--disable-blink-features=AutomationControlled')
import pandas as pd
import scipy as sc
import numpy as np
import sys
req_headers = {
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en;q=0.8',
'upgrade-insecure-requests': '1',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951 Safari/537.36'
}
response = requests.get("https://www.zillow.com/homes/for_rent/CA/house_type/",headers=req_headers)
print(response)
soup = BeautifulSoup(response.content, 'html.parser')
print(soup.prettify())
listing_urls = []
listings = soup.find_all("article", {"class": "list-card list-card-additional-attribution list-card_not-saved"})
for listing in listings:
listing_url = listing.find("a")["href"]
print(listing_url)
listing_urls.append(listing_url)
我被困在这里 - 我收到以下错误：
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_24224/2055957203.py in <module>
4
5 for listing in listings:
----> 6 listing_url = listing.find("a")["href"]
7
8 print(listing_url)
TypeError: 'NoneType' object is not subscriptable
此外，代码仅打印整个页面的 2 个链接（每页有 40 个待租房屋/公寓列表）
谢谢 ！ :)
【问题讨论】：
标签： python pandas selenium web-scraping beautifulsoup