从网站上抓取 Python

Question

我尝试为https://www.waug.com/area/?idx=15写一个网络爬虫：

#!/usr/bin/env python3
#_*_coding:utf8_*_

import requests
from bs4 import BeautifulSoup

url = requests.get('https://www.abcd.com/area/?abc=15')

html = url.text

soup = BeautifulSoup(html, 'html.parser')

count = 1

names = soup.select('#good_{} > div > div.class_name > div > div'.format(count))
prices = soup.select('#good_{} > div > div.class_name > div.class_name'.format(count))

for name in names:
    while count < 45:
        print(name.text)
        count = count + 1

for price in prices:
    while count < 45:
        print(price.text)
        count = count + 1

输出只有第一个项目名称的 45 倍，没有价格。如何获取所有商品名称和价格？我想在同一行获取商品名称和价格。 （为了以防万一，我更改了 url 和一些类名）

Answer 1

为了确保为正确的标题获得正确的名称，我将获得整个“item-good”类。

然后使用 for 循环可以让我确定我得到的标题与其价格相匹配。

这是一个如何使用 BeautifulSoup 解析网站的示例：

#!/usr/bin/env python3
#_*_coding:utf8_*_

import requests
from bs4 import BeautifulSoup

url = requests.get('https://www.waug.com/area/?idx=15')

html = url.text

soup = BeautifulSoup(html, 'html.parser')

count = 1

items = soup.findAll("div", {"class": "item-good"})

for item in items:
  item_title = item.find("div", {"class": "good-title-text"})
  item_price = item.find("div", {"class": "price-selling"})
  print item_title.text + " "  + item_price.text
  # If you get encoding errors delete the row above and uncomment the one below
  #print item_title.text.encode("utf-8") + " "  + item_price.text.encode("utf-8")

根据 OP 的要求，这还不够，因为有一个“更多”按钮可以在网页中按下以检索所有结果。

这可以使用 Selenium Webdriver 来完成。

=== 重要提示 ===

为了完成这项工作，您还需要在脚本文件夹中复制“chromedriver”文件。

您可以从this Google website下载。

这是脚本：

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys

browser = webdriver.Chrome()
browser.get('https://www.waug.com/area/?idx=15')



for number in range(10):
    try:
       WebDriverWait(browser, 60).until(EC.presence_of_element_located((By.ID, "more_good")))
       more_button = browser.find_element_by_id('more_good')
       more_button.click()
       time.sleep(10)
    except:
       print "Scrolling is now complete!"

source = browser.page_source

# This source variable should be used as input for BeautifulSoup
print source

现在将两个已解释的灵魂合并以获得最终请求的结果。

请记住，这只是一个快速的'n'dirty hack，需要适当的错误处理和完善，但它应该足以让你开始：

#!/usr/bin/env python3
#_*_coding:utf8_*_

from bs4 import BeautifulSoup
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys

browser = webdriver.Chrome()
browser.get('https://www.waug.com/area/?idx=15')

def is_page_load_complete():
    close_button = browser.find_element_by_id('close_good');
    return close_button.is_displayed();

while(True):
    WebDriverWait(browser, 60).until(EC.presence_of_element_located((By.ID, "more_good")))
    time.sleep(10)
    more_button = browser.find_element_by_id('more_good')
    if (more_button.is_displayed()):
        more_button.click()
    else:
        if (is_page_load_complete()):
            break

source = browser.page_source
soup = BeautifulSoup(source, 'html.parser')
items = soup.findAll("div", {"class": "item-good"})

for item in items:
  item_title = item.find("div", {"class": "good-title-text"})
  item_price = item.find("div", {"class": "price-selling"})
  print item_title.text + " "  + item_price.text
  # If you get encoding errors comment the row above and uncomment the one below
  #print item_title.text.encode("utf-8") + " "  + item_price.text.encode("utf-8")

print "Total items found: " + str(len(items))