【发布时间】：2023-01-09 10:59:18
【问题描述】：
我正在使用下面的代码一直滚动到 YouTube 页面的底部并且它有效。我的问题是，在网站向下滚动到底部后，我怎样才能在 console.log 中看到已到达底部？
笔记：该解决方案应该适用于 youtube.com。我已经尝试获取文档高度并将其与滚动高度进行比较，但这没有用！
const puppeteer = require('puppeteer');
let thumbArr = []
const scrapeInfiniteScrollItems = async(page) => {
while (true) {
const previousHeight = await page.evaluate(
"document.querySelector('ytd-app').scrollHeight"
);
await page.evaluate(() => {
const youtubeScrollHeight =
document.querySelector("ytd-app").scrollHeight;
window.scrollTo(0, youtubeScrollHeight);
});
await page.waitForFunction(
`document.querySelector('ytd-app').scrollHeight > ${previousHeight}`, {
timeout: 0
}
);
const thumbnailLength = (await page.$$('ytd-grid-video-renderer')).length
//this logs the amount of thumbnails every loop but once bottom scroll has been reached it stops logging (obviously) but the question is how am I supposed to compare the last amount of thumbnail's found with total thumbnails once the loop has stopped running. Take a look below to better understand my question.
thumbArr.push(thumbnailLength)
if (thumbnailLength == thumbArr.at(-1)) {
console.log('bottom has been reached')
}
await page.waitForTimeout(1000)
}
};
(async() => {
const browser = await puppeteer.launch({
headless: false
});
const page = await browser.newPage();
await page.goto('https://www.youtube.com', {
waitUntil: 'networkidle2',
});
await scrapeInfiniteScrollItems(page)
})();
更新：
let clientHeightArr = []
let clientHeightArrTracker = []
const scrapeInfiniteScrollItems = async(browser, page) => {
var infiniteScrollTrackerInterval = setInterval(async() => {
clientHeightArrTracker.push(clientHeightArr.length)
if (clientHeightArrTracker.some((e, i, arr) => arr.indexOf(e) !== i) == true) {
clearInterval(infiniteScrollTrackerInterval)
console.log('Bottom is reached')
//causes error "ProtocolError: Protocol error (Runtime.callFunctionOn): Target closed."
await browser.close()
}
}, 2000)
while (true) {
const previousHeight = await page.evaluate(
"document.querySelector('ytd-app').scrollHeight"
);
await page.evaluate(() => {
const youtubeScrollHeight =
document.querySelector("ytd-app").scrollHeight;
window.scrollTo(0, youtubeScrollHeight);
});
await page.waitForFunction(
`document.querySelector('ytd-app').scrollHeight > ${previousHeight}`, {
timeout: 0
},
);
const clientHeight = await page.$$eval("ytd-app", el => el.map(x => x.clientHeight));
clientHeightArr.push(clientHeight[0])
await page.waitForTimeout(1000)
}
};
(async() => {
const browser = await puppeteer.launch({
headless: false
});
const page = await browser.newPage();
await page.goto('https://www.youtube.com/c/mkbhd/videos', {
waitUntil: 'networkidle2',
});
await scrapeInfiniteScrollItems(browser, page)
})();
【问题讨论】：
-
你说的支票在哪里做？它应该可以工作，也许有一个增量，以防万一差异很小。打印这两个值来调试它并调试为什么它没有检测到结束。您还可以计算迭代之间页面上视频元素缩略图（或其他）的数量，如果它停止更改，您就完成了。
await new Promise((resolve) => setTimeout(resolve, 1000));应该是await page.waitForTimeout(1000)虽然几乎总是有一个page.waitForFunction更精确（可能是卡片/缩略图再次计数）。 -
顺便说一句，根据您要获取的数据，您可能根本不需要滚动，所以整个事情通常是一个 xy problem 如果您首先提供为什么需要滚动的上下文，则可以解决.通常，数据位于网络请求或静态 HTML 中，您可以毫不费力地获取它。
-
@ggorlen 这是我试图获取的数据
const title = await page.$$eval(".ytd-grid-video-renderer #video-title", el => el.map(x => x.getAttribute("title"))); -
这是在哪个页面？
-
@ggorlen 这个例如
https://www.youtube.com/c/mkbhd/videos
标签： javascript puppeteer