BeautifulSoup 蒸汽市场网页抓取错误

Question

我正在尝试使用 python 和 BeautifulSoup4 编写一个程序，该程序查看某个游戏（在本例中为 Rust）的 Steam 市场首页，并查看每个项目并获取它们的名称和价格。到目前为止，我已经设法让这个在第一页上工作（因为每个页面只显示 10 个项目，但是当我更改第二页的网址时，我得到的输出与第一页完全相同。

我用于第一页的 URL 是：https://steamcommunity.com/market/search?appid=252490#p1_popular_desc

第二页是：https://steamcommunity.com/market/search?appid=252490#p2_popular_desc

代码是：

import bs4 as bs
import urllib.request

for web_page in range(1,3):
    print('webpage number is: '+ str(web_page))
    if web_page == 1:
        url = "https://steamcommunity.com/market/search?appid=252490#p1_popular_desc"
        print(url)
        sauce = urllib.request.urlopen(url).read()
        soup = bs.BeautifulSoup(sauce,'lxml')


    if web_page == 2:
        urlADD = '#p2_popular_desc'
        url ="https://steamcommunity.com/market/search?appid=252490#p2_popular_desc"
        print(url)

        sauce = urllib.request.urlopen(url).read()
        soup = bs.BeautifulSoup(sauce,'lxml')



    for div in soup.find_all('a',class_='market_listing_row_link'):
        span = div.find('span',class_='normal_price')
        span2 = div.find('span',class_='market_listing_item_name')
        print(span2.text)
        print(span.text)

我不确定这里有什么问题，欢迎提供帮助。

Answer 1

试试这个： 你需要为 Firefox 安装 selenium 和 geckodriver 你将需要这个pypi.python.org/pypi/selenium（快乐的脚本：>）

#Mossein~King(1m here to help)
import time
import selenium
import selenium.webdriver as webdriver
from BeautifulSoup import BeautifulSoup

#for.testing.purposes.only
driver = webdriver.Firefox()

url = ''
driver.get(url)

#pages you like to interact with
pages = 2
for x in xrange(pages):
    pagesource = driver.page_source
    soup = BeautifulSoup(pagesource)
    #do your stuff

    #go to next page
    #example if next button is <a class='MosseinKing Is Awesome'>
    driver.find_element_by_xpath("//span[@class='MosseinKing Is Awesome']").click()
    #wait for 2 seconds for page to load
    time.sleep(2)