使用 Scrapy 难以从网页中抓取所需数据答案

【问题标题】：Difficulty scraping needed data from webpage with Scrapy使用 Scrapy 难以从网页中抓取所需数据
【发布时间】：2019-06-04 20:03:17
【问题描述】：

我正在抓取以下网页，http://www.starcitygames.com/catalog/category/Duel%20Decks%20Venser%20vs%20Koth，我需要获取卡片名称、价格、库存和状态。好吧，我得到了四个工作中的三个，但我的病情有问题。无论我尝试什么，它要么只给我 NULL 要么给我其他不正确的东西。

部分 HTML 代码

<td class="deckdbbody search_results_7">
<a href="http://www.starcitygames.com/content/cardconditions">NM/M</a>
</td>

SplashSpider.py

import csv
from scrapy.spiders import Spider
from scrapy_splash import SplashRequest
from ..items import GameItem

# process the csv file so the url + ip address + useragent pairs are the same as defined in the file # returns a list of dictionaries, example:
# [ {'url': 'http://www.starcitygames.com/catalog/category/Rivals%20of%20Ixalan',
#    'ip': 'http://204.152.114.244:8050',
#    'ua': "Mozilla/5.0 (BlackBerry; U; BlackBerry 9320; en-GB) AppleWebKit/534.11"},
#    ...
# ]
def process_csv(csv_file):
    data = []
    reader = csv.reader(csv_file)
    next(reader)
    for fields in reader:
        if fields[0] != "":
            url = fields[0]
        else:
            continue # skip the whole row if the url column is empty
        if fields[1] != "":
            ip = "http://" + fields[1] + ":8050" # adding http and port because this is the needed scheme
        if fields[2] != "":
            useragent = fields[2]
        data.append({"url": url, "ip": ip, "ua": useragent})
    return data


class MySpider(Spider):
    name = 'splash_spider'  # Name of Spider

    # notice that we don't need to define start_urls
    # just make sure to get all the urls you want to scrape inside start_requests function

    # getting all the url + ip address + useragent pairs then request them
    def start_requests(self):

        # get the file path of the csv file that contains the pairs from the settings.py
        with open(self.settings["PROXY_CSV_FILE"], mode="r") as csv_file:
           # requests is a list of dictionaries like this -> {url: str, ua: str, ip: str}
            requests = process_csv(csv_file)

        for req in requests:
            # no need to create custom middlewares
            # just pass useragent using the headers param, and pass proxy using the meta param

            yield SplashRequest(url=req["url"], callback=self.parse, args={"wait": 3},
                    headers={"User-Agent": req["ua"]},
                    splash_url = req["ip"],
                    )
    # Scraping
    def parse(self, response):
        item = GameItem()
        for game in response.css("tr[class^=deckdbbody]"):
            # Card Name
            item["card_name"] = game.css("a.card_popup::text").extract_first()
            item["condition"] = game.css("a::text").extract_first() #Problem is here

            item["stock"] = game.css("td[class^=deckdbbody].search_results_8::text").extract_first()
            item["price"] = game.css("td[class^=deckdbbody].search_results_9::text").extract_first()

            yield item

【问题讨论】：

标签： python scrapy splash-screen scrapy-splash

【解决方案1】：

您需要在 CSS 表达式中指定目标单元格：

item["condition"] = game.css("td[class^=deckdbbody].search_results_7 a::text").get()

【讨论】：

【解决方案2】：

我认为使用这个选择器你没有得到正确的<a> 元素。您的条件的 css 表示要在 tr[class^=deckdbbody] 中获取第一个 <a>，但条件列不是 tr[class^=deckdbbody] 中的第一个 <a> 元素。

为了选择正确的元素，您可以使用 xpath contains() 来测试它是否是所需的链接。

>>> response.css("tr[class^=deckdbbody]").xpath(".//a[contains(@href, 'cardconditions')]/text()").extract()
['NM/M', 'PL', 'NM/M', 'NM/M', 'PL', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'PL', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'PL', 'NM/M', 'NM/M', 'PL', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'NM/M', 'PL', 'NM/M', 'NM/M', 'NM/M']

此外，我认为您不需要 Scrapy Splash 来抓取该站点，数据似乎可以从 scrapy shell 命令获得。

另外，值得一看https://stackoverflow.com/help/minimal-reproducible-example

【讨论】：