使用多个 <table> 标记抓取 HTML 页面并从特定的 <a> 标记后代中提取文本答案

【问题标题】：Scrape HTML page with multiple <table> tags and extract text from specific <a> tag descendants使用多个 <table> 标记抓取 HTML 页面并从特定的 <a> 标记后代中提取文本
【发布时间】：2021-05-18 19:08:42
【问题描述】：

我在数据库字段中有这个 html 源代码。我想分析这段代码，特别是一些表格的字段，并将它们打印在屏幕上。这是关于表格的代码：

<table cellspacing="1" cellpadding="1" class="troop_details inReturn"
    >
        <thead>
            <tr>
                <td class="role">
                                            <a href="/karte.php?d=91628">01] #WorkInProgress</a>
                                    </td>
                <td colspan="11" class="troopHeadline">
                                                                <a href="/karte.php?d=91611">Return from 01-soldier</a>
                                    </td>
            </tr>
        </thead>
        <tbody class="units">
            <tr>
                <th class="coords">
                                            &#x202d;<span class="coordinates coordinatesWrapper coordinatesAligned coordinatesltr"><span class="coordinateX">(&#x202d;&minus;&#x202d;1&#x202c;&#x202c;</span><span class="coordinatePipe">|</span><span class="coordinateY">&#x202d;&minus;&#x202d;28&#x202c;&#x202c;)</span></span>&#x202c;                                    </th>
                                    <td class="uniticon">
                        <img class="unit u21" title="Phalanx: 1:12:51" alt="Phalanx" src="/img/x.gif" />                    </td>
                                    <td class="uniticon">
                        <img class="unit u22" title="Swordsman: 1:25:00" alt="Swordsman" src="/img/x.gif" />                    </td>
                                    <td class="uniticon">
                        <img class="unit u23" title="Pathfinder: 0:30:00" alt="Pathfinder" src="/img/x.gif" />                  </td>
                                    <td class="uniticon">
                        <img class="unit u24" title="Theutates Thunder: 0:26:51" alt="Theutates Thunder" src="/img/x.gif" />                    </td>
                                    <td class="uniticon">
                        <img class="unit u25" title="Druidrider: 0:31:53" alt="Druidrider" src="/img/x.gif" />                  </td>
                                    <td class="uniticon">
                        <img class="unit u26" title="Haeduan: 0:39:14" alt="Haeduan" src="/img/x.gif" />                    </td>
                                    <td class="uniticon">
                        <img class="unit u27" title="Ram: 2:07:30" alt="Ram" src="/img/x.gif" />                    </td>
                                    <td class="uniticon">
                        <img class="unit u28" title="Trebuchet: 2:50:00" alt="Trebuchet" src="/img/x.gif" />                    </td>
                                    <td class="uniticon">
                        <img class="unit u29" title="Chieftain: 1:42:00" alt="Chieftain" src="/img/x.gif" />                    </td>
                                    <td class="uniticon">
                        <img class="unit u30" title="Settler: 1:42:00" alt="Settler" src="/img/x.gif" />                    </td>
                                                    <td class="uniticon last">
                        <img class="unit uhero" title="Hero" alt="Hero" src="/img/x.gif" />                 </td>
                            </tr>
        </tbody>

        <tbody class="units last">
            <tr>
                <th>Troops</th>
                                                    <td class="unit none">
                                                    0                                           </td>
                                    <td class="unit none">
                                                    0                                           </td>
                                    <td class="unit none">
                                                    0                                           </td>
                                    <td class="unit">
                                                    500                                         </td>
                                    <td class="unit none">
                                                    0                                           </td>
                                    <td class="unit none">
                                                    0                                           </td>
                                    <td class="unit none">
                                                    0                                           </td>
                                    <td class="unit none">
                                                    0                                           </td>
                                    <td class="unit none">
                                                    0                                           </td>
                                    <td class="unit none">
                                                    0                                           </td>
                                    <td class="unit none last">
                                                    0                                           </td>
                            </tr>
        </tbody>

                    <tbody class="infos">
                <tr>
                    <th>Bounty</th>
                    <td colspan="11">
                        <div class="res">
                            <div class="inlineIconList resourceWrapper"><div class="inlineIcon resources" title="Lumber"><i class="r1"></i><span class="value ">6758</span></div><div class="inlineIcon resources" title="Clay"><i class="r2"></i><span class="value ">8093</span></div><div class="inlineIcon resources" title="Iron"><i class="r3"></i><span class="value ">6908</span></div><div class="inlineIcon resources" title="Crop"><i class="r4"></i><span class="value ">15741</span></div></div>                       </div>
                        <div class="carry">
                            <img class="carry full" title="carry"
                                 alt="carry"
                                 src="/img/x.gif"/> &#x202d;&#x202d;37500&#x202c;&nbsp;/&nbsp;&#x202d;37500&#x202c;&#x202c;                     </div>
                    </td>
                </tr>
            </tbody>
        
        <tbody class="infos">
            <tr>
                <th>Arrival</th>
                <td colspan="11">
                    <div class="in">in&nbsp;<span  class="timer" counting="down" value="85">0:01:25</span>&nbsp;hrs.</div>
                    <div class="at"><span>at&nbsp;00:43:10</span><span> </span></div>
                </td>
            </tr>
        </tbody>
    </table>
            <a name="at"></a>
    <table cellspacing="1" cellpadding="1" class="troop_details inReturn"
    >
        <thead>
            <tr>
                <td class="role">
                                            <a href="/karte.php?d=91628">01] #WorkInProgress</a>
                                    </td>
                <td colspan="11" class="troopHeadline">
                                                                <a href="/karte.php?d=94829">Return from 0-New Hulk</a>
                                    </td>
            </tr>
        </thead>
        <tbody class="units">
            <tr>
                <th class="coords">
                                            &#x202d;<span class="coordinates coordinatesWrapper coordinatesAligned coordinatesltr"><span class="coordinateX">(&#x202d;&minus;&#x202d;1&#x202c;&#x202c;</span><span class="coordinatePipe">|</span><span class="coordinateY">&#x202d;&minus;&#x202d;28&#x202c;&#x202c;)</span></span>&#x202c;                                    </th>
                                    <td class="uniticon">
                        <img class="unit u21" title="Phalanx: 0:45:33" alt="Phalanx" src="/img/x.gif" />                    </td>
                                    <td class="uniticon">
                        <img class="unit u22" title="Swordsman: 0:53:09" alt="Swordsman" src="/img/x.gif" />                    </td>
                                    <td class="uniticon">
                        <img class="unit u23" title="Pathfinder: 0:18:46" alt="Pathfinder" src="/img/x.gif" />                  </td>
                                    <td class="uniticon">
                        <img class="unit u24" title="Theutates Thunder: 0:16:47" alt="Theutates Thunder" src="/img/x.gif" />                    </td>
                                    <td class="uniticon">
                        <img class="unit u25" title="Druidrider: 0:19:56" alt="Druidrider" src="/img/x.gif" />                  </td>
                                    <td class="uniticon">
                        <img class="unit u26" title="Haeduan: 0:24:32" alt="Haeduan" src="/img/x.gif" />                    </td>
                                    <td class="uniticon">
                        <img class="unit u27" title="Ram: 1:19:44" alt="Ram" src="/img/x.gif" />                    </td>
                                    <td class="uniticon">
                        <img class="unit u28" title="Trebuchet: 1:46:18" alt="Trebuchet" src="/img/x.gif" />                    </td>
                                    <td class="uniticon">
                        <img class="unit u29" title="Chieftain: 1:03:47" alt="Chieftain" src="/img/x.gif" />                    </td>
                                    <td class="uniticon">
                        <img class="unit u30" title="Settler: 1:03:47" alt="Settler" src="/img/x.gif" />                    </td>
                                                    <td class="uniticon last">
                        <img class="unit uhero" title="Hero" alt="Hero" src="/img/x.gif" />                 </td>
                            </tr>
        </tbody>

        <tbody class="units last">
            <tr>
                <th>Troops</th>
                                                    <td class="unit none">
                                                    0                                           </td>
                                    <td class="unit none">
                                                    0                                           </td>
                                    <td class="unit none">
                                                    0                                           </td>
                                    <td class="unit">
                                                    400                                         </td>
                                    <td class="unit none">
                                                    0                                           </td>
                                    <td class="unit none">
                                                    0                                           </td>
                                    <td class="unit none">
                                                    0                                           </td>
                                    <td class="unit none">
                                                    0                                           </td>
                                    <td class="unit none">
                                                    0                                           </td>
                                    <td class="unit none">
                                                    0                                           </td>
                                    <td class="unit none last">
                                                    0                                           </td>
                            </tr>
        </tbody>

                    <tbody class="infos">
                <tr>
                    <th>Bounty</th>
                    <td colspan="11">
                        <div class="res">
                            <div class="inlineIconList resourceWrapper"><div class="inlineIcon resources" title="Lumber"><i class="r1"></i><span class="value ">6130</span></div><div class="inlineIcon resources" title="Clay"><i class="r2"></i><span class="value ">5835</span></div><div class="inlineIcon resources" title="Iron"><i class="r3"></i><span class="value ">5638</span></div><div class="inlineIcon resources" title="Crop"><i class="r4"></i><span class="value ">12397</span></div></div>                       </div>
                        <div class="carry">
                            <img class="carry full" title="carry"
                                 alt="carry"
                                 src="/img/x.gif"/> &#x202d;&#x202d;30000&#x202c;&nbsp;/&nbsp;&#x202d;30000&#x202c;&#x202c;                     </div>
                    </td>
                </tr>
            </tbody>
        
        <tbody class="infos">
            <tr>
                <th>Arrival</th>
                <td colspan="11">
                    <div class="in">in&nbsp;<span  class="timer" counting="down" value="920">0:15:20</span>&nbsp;hrs.</div>
                    <div class="at"><span>at&nbsp;00:57:05</span><span> </span></div>
                </td>
            </tr>
        </tbody>
    </table>

我感兴趣的数据如下：

01-士兵归来00:43:10
0-新绿巨人归来00:57:05

感谢您的建议，这是我目前的代码：

  <?php include 'database.php' ?>
<?php session_start(); ?>

<?php
include_once('simple_html_dom.php');
$caserma = $_SESSION["caserma"];
    
$dom = new DOMDocument;
libxml_use_internal_errors(true);
$dom->loadHTML($_SESSION["caserma"], LIBXML_HTML_NOIMPLIED | LIBXML_HTML_NODEFDTD);
$xpath = new DOMXPath($dom);
$texts = [];
foreach ($xpath->query("//table[contains(@class, 'troop_details') and contains(@class, 'inReturn')]//td[@class='troopHeadline']//a[@href]/text()") as $textNode) {
    $texts[] = $textNode->nodeValue;
}
var_export($texts);
 ?>

但作为输出，它给了我数组 ( )

【问题讨论】：

标签： php dom web-scraping xpath domdocument

【解决方案1】：

假设 $_SESSION["caserma"] 包含您的完整 html 文档的代码：(Demo)

$dom = new DOMDocument;
libxml_use_internal_errors(true);
$dom->loadHTML($_SESSION["caserma"]);
$xpath = new DOMXPath($dom);
$texts = [];
foreach ($xpath->query("//table[contains(@class, 'troop_details') and contains(@class, 'inReturn')]//td[@class='troopHeadline']//a[@href]/text()") as $textNode) {
    $texts[] = $textNode->nodeValue;
}
var_export($texts);

样本输入的输出：

array (
  0 => 'Return from 01-soldier',
  1 => 'Return from 0-New Hulk',
)

XPath 分解：

//                                                                         # search to any depth in the document
table[contains(@class, 'troop_details') and contains(@class, 'inReturn')]  # find all table tags with both `troop_details` and `inReturn` classes
//                                                                         # continue searching any descendants of any matches
td[@class='troopHeadline']                                                 # match all td tags with `troopHeadline` as its class
//                                                                         # continue searching anydescendants of any matches
a[@href]                                                                   # match all a tags with an href attribute
/                                                                          # search the immediate descendant (any first generation child)
text()                                                                     # match the text of the parent a tag

libxml_use_internal_errors(true) 用于消除“无效”文档中的任何潜在错误。
在 xpath 中使用 contains(...) and contains() 非常重要，这样即使类属性更改了它们的顺序或将新类添加到元素中，xpath 仍然会正确匹配。
foreach() 循环将迭代所有符合条件的文本节点。
提取nodeValue 并将其推送到结果数组中。

【讨论】：

谢谢，这段代码运行良好，但出现了另一个问题，我为此打开了另一个问题。作为输出它给了我空数组（），我认为输入代码可能不是有效的xml/html。你能帮帮我吗？