当只有 $eval 给出结果时如何获取所有 innerHTML（$$ 返回未定义）

Question

我正在尝试从每行中提取 3 个信息。完成后，它会滚动到页面底部单击“加载更多”，然后抓取新数据，依此类推，直到没有更多“加载更多”按钮。

为了从表中提取所有数据，我使用了 $$eval 但这会导致未定义。但是，如果我改用 $eval，我会得到数据，但这只会从表的第一行中提取数据。为什么 $$eval 返回“未定义”，如果我不能使用它，如何循环遍历表以使用 $eval 获取所有值？

    const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({ headless: false }); // default is true
  const page = await browser.newPage();
  await page.goto('someexamplesite.com', {
    waitUntil: 'domcontentloaded',
  });

  const ExerciseName = await page.$$eval(
    '.ExCategory-results > .ExResult-row:nth-child(2) > .ExResult-cell > .ExHeading > a',
    (e) => e.innerText
  );

  const muscleTargeted = await page.$$eval(
    ' .ExCategory-results > .ExResult-row:nth-child(2) > .ExResult-cell > .ExResult-muscleTargeted > a',
    (e) => e.innerText
  );

  const equipmentType = await page.$$eval(
    '.ExCategory-results > .ExResult-row:nth-child(2) > .ExResult-cell > .ExResult-equipmentType > a',
    (e) => e.innerText
  );

  //click on load more
  await page.waitForSelector(
    '#js-ex-content > #js-ex-category-body > .ExCategory-results > .ExLoadMore > .bb-flat-btn'
  );

  console.log({ ExerciseName, muscleTargeted, equipmentType });

    await browser.close();
})().catch((e) => {
  console.error(e);
});

我试图抓取的代码

<div class="ExCategory-results">
    <div class="ExCategory-resultsLoadIndicator" id="js-ex-finder-load-indicator">
      <div class="ExCategory-resultsLoadIndicatorBox">
        <div class="ExCategory-resultsLoadIndicatorSpinner bb-spinner-btn__spinner"></div>
      </div>
    </div>
        
          <div class="ExResult-row  flexo-container flexo-between" itemscope="" itemtype="http://schema.org/ExerciseAction">
            <div class="ExResult-cell ">
                <!-- using male photos -->
                <img class="ExImg ExResult-img  ls-is-cached lazyloaded" width="70" height="70" onerror="if (window._E_) _E_(this)" alt=" thumbnail image" src="https://www.websites.com/exercises/exerciseImages/sequences/742/Male/m/742_1.jpg" data-src="https://www.websites.com/exercises/exerciseImages/sequences/742/Male/m/742_1.jpg" itemprop="image">
            </div>
            <div class="ExResult-cell ExResult-cell--nameEtc">
              <h3 class="ExHeading ExResult-resultsHeading">
                <a href="/exercises/rickshaw-carry" itemprop="name">
                  Rickshaw Carry
                </a>
              </h3>
              <div class="ExResult-details ExResult-muscleTargeted">
                Muscle Targeted:
                <a href="/exercises/muscle/forearms">
                  Forearms
                </a>
              </div>
              <div class="ExResult-details ExResult-equipmentType">
                Equipment Type:
                <a href="/exercises/equipment/other">
                  Other
                </a>
              </div>
            </div>
            <div class="ExResult-cell ExResult-cell--rating">
              <div class="ExRating">
                <div class="ExRating-badge">
                  9.6
                </div>
                <div class="ExRating-description ExRating-description--Average">
                  Average
                </div>
              </div>
            </div>
          </div>        
        
          <div class="ExResult-row  flexo-container flexo-between" itemscope="" itemtype="http://schema.org/ExerciseAction">
            <div class="ExResult-cell ">
                <!-- using male photos -->
                <img class="ExImg ExResult-img  ls-is-cached lazyloaded" width="70" height="70" onerror="if (window._E_) _E_(this)" alt=" thumbnail image" src="https://www.websites.com/images/2020/xdb/cropped/xdb-50m-single-leg-leg-press-m1-square-600x600.jpg" data-src="https://www.websites.com/images/2020/xdb/cropped/xdb-50m-single-leg-leg-press-m1-square-600x600.jpg" itemprop="image">
            </div>
            <div class="ExResult-cell ExResult-cell--nameEtc">
              <h3 class="ExHeading ExResult-resultsHeading">
                <a href="/exercises/single-leg-press" itemprop="name">
                  Single-Leg Press
                </a>
              </h3>
              <div class="ExResult-details ExResult-muscleTargeted">
                Muscle Targeted:
                <a href="/exercises/muscle/quadriceps">
                  Quadriceps
                </a>
              </div>
              <div class="ExResult-details ExResult-equipmentType">
                Equipment Type:
                <a href="/exercises/equipment/machine">
                  Machine
                </a>
              </div>
            </div>
            <div class="ExResult-cell ExResult-cell--rating">
              <div class="ExRating">
                <div class="ExRating-badge">
                  9.6
                </div>
                <div class="ExRating-description ExRating-description--Average">
                  Average
                </div>
              </div>
            </div>
          </div>        

Answer 1

page.$$eval 方法在后台运行Array.from(document.querySelectorAll(selector))，所以你得到的是一个数组。您不能直接在数组上应用(e) => e.innerText（即使它的长度为1），而不对其进行迭代或通过正确的索引获取正确的元素（例如：e[0].innerText），否则您将得到@987654330 @。

您可以使用Array.map 迭代匹配的元素并将每个元素的innerText 收集到一个数组中。

const exerciseName = await page.$$eval(
    '.ExCategory-results > .ExResult-row:nth-child(2) > .ExResult-cell > .ExHeading > a',
    elements => elements.map(el => el.innerText)
  )

输出：

[ 'Rickshaw Carry' ]

---

编辑：

您可以通过 (1) 计算具有相同类名的元素，使用带索引的循环（最容易使用常规 for 循环）来迭代行类：

const rowsCounts = await page.$$eval('.ExCategory-results > .ExResult-row', rows => rows.length)

然后 (2) 遍历子 .ExResult-row:nth-child(n) ...，并将 innerTexts 收集到一个数组 (exerciseNames)：

const exerciseNames = []
for (let i = 1; i < rowsCounts + 1; i++) { // you mignt need i = 2
  const exerciseName = await page.$eval(
    `.ExCategory-results > .ExResult-row:nth-child(${i}) > .ExResult-cell > .ExHeading > a`,
    el => el.innerText)
  exerciseNames.push(exerciseName)
}

输出：

[
  'Rickshaw Carry',
  'Single-Leg Press',
  'Landmine twist',
  'Weighted pull-up',
  'T-Bar Row with Handle',
  'Palms-down wrist curl over bench'
]

注意：在这种情况下，循环应该从1 而不是0 开始，因为没有“nth-child(0)”。在您的示例中，第一个也丢失了，因此您可能需要从 2 开始迭代。