Puppeteer 浏览器用户代理列表

Question

我想从我的默认位置路径加载随机的用户代理列表，例如：'agents.json'，而不是直接添加仅 1 个用户代理。

agents.json

["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36", "Mozilla/5.0 (iPad; CPU OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/79.0.3945.73 Mobile/15E148 Safari/604.1"]

我的代码：

const puppeteer = require('../core/puppeteer');
const referers = require('../core/referers.json');
const referers = require('../core/agents.json');
const viewVideosInBatch = async ({ targetUrls, durationInSeconds, port }) => {
  let browser;
  try {
    browser = await puppeteer.getBrowserInstance(port);
    const randomReferer = referers[Math.floor(Math.random() * referers.length)];
    const randomAgents = agents[Math.floor(Math.random() * agents.length)];
    const page = await browser.newPage();
    browser = await page.setUserAgent({ agents: randomAgents });
    page.setDefaultTimeout(PAGE_DEFAULT_TIMEOUT * 1000);
    page.on('error', handlePageCrash(page));
    page.on('pageerror', handlePageCrash(page));
    page.setExtraHTTPHeaders({ referer: randomReferer });

    await page.setViewport({
      width: 640,
      height: 480,
      deviceScaleFactor: 1,
    });
    const ipAddr = await getCurrentIP(page);
    const targetUrlsForAction = _take(_shuffle(targetUrls), VIEW_ACTION_COUNT);
    await watchVideosInSequence(page, ipAddr, targetUrlsForAction, durationInSeconds);
    await page.close();
  } catch (error) {
    logger.warn('Entire view action in a batch failed.');
    logger.debug(error);
  } finally {
    await browser.close();
  }
};

我得到的错误：

(node:1) UnhandledPromiseRejectionWarning: Error: Protocol error (Network.setUserAgentOverride): Target closed.
view_4  |     at /app/node_modules/puppeteer/lib/Connection.js:183:56
view_4  |     at new Promise (<anonymous>)
view_4  |     at CDPSession.send (/app/node_modules/puppeteer/lib/Connection.js:182:12)
view_4  |     at next (/app/node_modules/puppeteer-extra-plugin-stealth/evasions/sourceurl/index.js:30:43)
view_4  |     at CDPSession.send (/app/node_modules/puppeteer-extra-plugin-stealth/evasions/sourceurl/index.js:46:18)
view_4  |     at Plugin.onPageCreated (/app/node_modules/puppeteer-extra-plugin-stealth/evasions/user-agent-override/index.js:75:18)
view_4  |     at runMicrotasks (<anonymous>)
view_4  |     at processTicksAndRejections (internal/process/task_queues.js:97:5)
view_4  |     at async Plugin._onTargetCreated (/app/node_modules/puppeteer-extra-plugin/dist/index.cjs.js:491:17)
view_4  | (node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 79)

Answer 1

您似乎在设置 UA 时重新声明了浏览器，但您不应该这样做。此外，您不需要将其设置为对象：page.setUserAgent 接受字符串：page.setUserAgent(userAgent)。

await page.setUserAgent(randomAgent); 会很好。

const browser = await puppeteer.getBrowserInstance(port);
const randomReferer = referers[Math.floor(Math.random() * referers.length)];
const randomAgent = agents[Math.floor(Math.random() * agents.length)]; // made it singular instead of plural
const page = await browser.newPage();
await page.setUserAgent(randomAgent); // like this
page.setDefaultTimeout(PAGE_DEFAULT_TIMEOUT * 1000);
page.on('error', handlePageCrash(page));
page.on('pageerror', handlePageCrash(page));
page.setExtraHTTPHeaders({ referer: randomReferer });

注意：我认为这只是你的帖子中的一个错误，但你还有两个名为referers的const，这个应该是agents：

const referers = require('../core/agents.json');