如何从网页中的多个表格中抓取内容

Question

我想从网页中的多个表格中抓取内容，HTML 代码如下所示：

<div class="fixtures-table full-table-medium" id="fixtures-data">             
    <h2 class="table-header"> Date 1    </h2>
    <table class="table-stats">
        <tbody>
            <tr class='preview' id='match-row-EFBO755307'>
                <td class='details'>
                    <p>
                        <span class='team-home teams'>
                            <a href='random_team'>team 1</a>                
                        </span>                 
                        <span class='team-away teams'>
                            <a href='random_team'>team 2</a>                
                        </span>
                    </p>
                </td>
            </tr>
            <tr class='preview' id='match-row-EFBO755307'>
                <td class='match-details'>
                    <p>
                        <span class='team-home teams'>
                            <a href='random_team'>team 3</a>                
                        </span>                 
                        <span class='team-away teams'>
                            <a href='random_team'>team 4</a>                
                        </span>
                    </p>
                </td>
            </tr>
        </tbody>
    </table>

    <h2 class="table-header"> Date 2    </h2>
    <table class="table-stats">
        <tbody>
            <tr class='preview' id='match-row-EFBO755307'>
                <td class='match-details'>
                    <p>
                        <span class='team-home teams'>
                            <a href='random_team'>team X</a>                
                        </span>                 
                        <span class='team-away teams'>
                            <a href='random_team'>team Y</a>                
                        </span>
                    </p>
                </td>
            </tr>
            <tr class='preview' id='match-row-EFBO755307'>
                <td class='match-details'>
                    <p>
                        <span class='team-home teams'>
                            <a href='random_team'>Team A</a>                
                        </span>                 
                        <span class='team-away teams'>
                            <a href='random_team'>Team B</a>                
                        </span>
                    </p>
                </td>
            </tr>
        </tbody>
    </table>
</div>

在日期下还有更多的比赛（9 场或 2 场或 1 场，取决于该日期所进行的比赛）和编号。表数为 63（等于天数）

我想提取每个日期的球队之间的比赛，以及哪支球队是主场，哪支球队是客场。

我使用的是scrapy shell并尝试了以下命令：

 title = sel.xpath("//td[@class = 'match-details']")[0] 
 l_home = title.xpath("//span[@class = 'team-home teams']/a/text()").extract()

这打印了主队的列表，这打印了所有客队的列表，

 l_Away = title.xpath("//span[@class = 'team-away teams']/a/text()").extract()

这给了我所有日期的清单：

sel.xpath("/html/body/div[3]/div/div/div/div[4]/div[2]/div/h2/text()").extract()

我想要的是所有日期都获得当天进行的比赛（以及哪支球队是主客场）

我的 items.py 应该如下所示：

date = Field()
home_team = Field()
away_team2 = Field()

请帮我写parse函数和Item类。

提前致谢。

Answer 1

这是来自scrapy shell 的示例逻辑：

>>> for table in response.xpath('//table[@class="table-stats"]'):
...     date = table.xpath('./preceding-sibling::h2[1]/text()').extract()[0]
...     print date
...     for match in table.xpath('.//tr[@class="preview" and @id]'):
...         home_team = match.xpath('.//span[@class="team-home teams"]/a/text()').extract()[0]
...         away_team = match.xpath('.//span[@class="team-away teams"]/a/text()').extract()[0]
...         print home_team, away_team
... 
 Date 1    
team 1 team 2
team 3 team 4
 Date 2    
team X team Y
Team A Team B

在parse() 方法中，您需要在内部循环中实例化一个Item 实例并在yield 它：

def parse(self, response):
    for table in response.xpath('//table[@class="table-stats"]'):
        date = table.xpath('./preceding-sibling::h2[1]/text()').extract()[0]
        for match in table.xpath('.//tr[@class="preview" and @id]'):
            home_team = match.xpath('.//span[@class="team-home teams"]/a/text()').extract()[0]
            away_team = match.xpath('.//span[@class="team-away teams"]/a/text()').extract()[0]

            item = MyItem()
            item['date'] = date
            item['home_team'] = home_team
            item['away_team'] = away_team
            yield item

Myitem 在哪里：

class MyItem(Item):
    date = Field()
    home_team = Field()
    away_team = Field()