使用 Puppeteer 捕获可滚动页面中的所有链接 - 无限滚动

Question

此脚本以无限滚动的方式滚动页面并捕获所有链接。

每次都向底部移动，重复加载新内容

1. 如何返回结果？
2. 此外，如何以块的形式返回结果，避免将部分结果附加到同一个数组中？

脚本：

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
                                            headless: false,
                                            userDataDir: "C:\\Users\\johndoe\\AppData\\Local\\Google\\Chrome\\User Data\\Default"
                                        });
  const page = await browser.newPage();
  await page.setViewport({
    width: 1920,
    height: 1080,
    deviceScaleFactor: 1,
  });
  await page.goto('https://www.facebook.com/groups/000000000000/members',{waitUntil: 'networkidle0'});
  page.on('console', msg => console.log('PAGE LOG:', msg.text()));  //sottoscrivo l'evento console e lo recupero nell'evaluate

  let rawMembers = await page.evaluate(() => { 

    const intervall = 3000;
    let stop = false;
    document.addEventListener('keypress', e => stop = true);  //press a key to exit

    let results = [];

    let pageHeigth = 0;
    let timerId = setTimeout(function tick() {

      if ((stop === false) && (document.body.scrollHeight > pageHeigth)){

        pageHeigth = document.body.scrollHeight  //save the current page heigth
        document.scrollingElement.scrollTop = pageHeigth;  //move the scroll to the end of the page (page visible size), this will couse new content to be loaded - virtula scroll)

        console.log('PAGE HEIGTH: ', pageHeigth);

        //do the work (capture what i need, all the links in my case)
        const anchors = Array.from(document.querySelectorAll('a'));
        const serializableLinks = anchors.map(x => x.getAttribute("href"));   //convert to something serializable (string)
        results.concat(serializableLinks);

        timerId = setTimeout(tick, intervall);  //schedule a new timeout to repeat the function
      } 
      else
      {
        clearTimeout(timerId)
        console.log('Exit');
        return results;
      }

    }, intervall);
  });

  //await browser.close();
})();

Answer 1

你可以使用

await page.evaluate(() => {
  document.scrollingElement.scrollTop = document.body.scrollHeight
})

这将滚动到页面底部。如果你想滚动一个 DOM 元素，你可以简单地

await page.evaluate(() => {
  let domElement = document.querySelector(YOUR DOM ELEMENT)
  domElement.scrollTop = domElement.scrollHeight   
})