在`forEach`循环中中断`request`以提高效率

Question

我正在构建一个简单的网络爬虫来自动化新闻通讯，这意味着我只需要对一定数量的页面进行翻页。在这个例子中，这没什么大不了的，因为脚本只会爬取 3 个额外的页面。但是对于不同的情况，这将是非常低效的。

所以我的问题是，有没有办法在这个 forEach 循环中停止执行 request()？

或者我是否需要改变我的方法来逐个抓取页面，如outlined in this guide.

脚本

'use strict';
var request = require('request');
var cheerio = require('cheerio');
var BASEURL = 'https://jobsite.procore.com';

scrape(BASEURL, getMeta);

function scrape(url, callback) {
  var pages = [];
  request(url, function(error, response, body) {
    if(!error && response.statusCode == 200) {

      var $ = cheerio.load(body);

      $('.left-sidebar .article-title').each(function(index) {
        var link = $(this).find('a').attr('href');
        pages[index] = BASEURL + link;
      });
      callback(pages, log);
    }
  });
}

function getMeta(pages, callback) {
  var meta = [];
  // using forEach's index does not work, it will loop through the array before the first request can execute
  var i = 0;
  // using a for loop does not work here
  pages.forEach(function(url) {
    request(url, function(error, response, body) {
      if(error) {
        console.log('Error: ' + error);
      }

      var $ = cheerio.load(body);

      var desc = $('meta[name="description"]').attr('content');
      meta[i] = desc.trim();

      i++;

      // Limit
      if (i == 6) callback(meta);
      console.log(i);
    });
  });
}

function log(arr) {
  console.log(arr);
}

输出

$ node crawl.js 
1
2
3
4
5
6
[ 'Find out why fall protection (or lack thereof) lands on the Occupational Safety and Health Administration (OSHA) list of top violations year after year.',
  'noneChances are you won’t be seeing any scented candles on the jobsite anytime soon, but what if it came in a different form? The allure of smell has conjured up some interesting scent technology in recent years. Take for example the Cyrano, a brushed-aluminum cylinder that fits in a cup holder. It’s Bluetooth-enabled and emits up to 12 scents or smelltracks that can be controlled using a smartphone app. Among the smelltracks: “Thai Beach Vacation.”',
  'The premise behind the hazard communication standard is that employees have a right to know the toxic substances and chemical hazards they could encounter while working. They also need to know the protective things they can do to prevent adverse effects of working with those substances. Here are the steps to comply with the standard.',
  'The Weitz Company has been using Procore on its projects for just under two years. Within that time frame, the national general contractor partnered with Procore to implement one of the largest technological advancements in its 163-year history.  Click here to learn more about their story and their journey with Procore.',
  'MGM Resorts International is now targeting Aug. 24 as the new opening date for the $960 million hotel and casino complex it has been building in downtown Springfield, Massachusetts.',
  'So, what trends are taking center stage this year? Below are six of the most prominent. Some of them are new, and some of them are continuations of current trends, but they are all having a substantial impact on construction and the structures people live and work in.' ]
7
8
9

Answer 1

除了使用slice 限制您的选择之外，您还可以重构代码以重用某些功能。

对不起，我想了一会儿就忍不住了。

我们可以从重构开始：

const rp = require('request-promise-native');
const {load} = require('cheerio');

function scrape(uri, transform) {
  const options = {
    uri,
    transform: load
  };

  return rp(options).then(transform);
}

scrape(
  'https://jobsite.procore.com',
  ($) => $('.left-sidebar .article-title a').toArray().slice(0,6).map((linkEl) => linkEl.attribs.href)
).then((links) => Promise.all(
  links.map(
    (link) => scrape(
      `https://jobsite.procore.com/${link}`,
      ($) => $('meta[name="description"]').attr('content').trim()
    )
  )
)).then(console.log).catch(console.error);

虽然这确实使代码更加干练和简洁，但它指出了可能需要改进的部分：链接的请求。

目前，它将几乎同时触发对原始页面上所有（或最多）6 个链接的请求。这可能是您想要的，也可能不是您想要的，这取决于在您提到的其他某个时间点将请求多少链接。

另一个潜在的问题是错误管理。就重构而言，如果任何一个请求失败，那么所有请求都将被丢弃。

如果您喜欢这种方法，只需考虑几点。 两者都可以通过多种方式解决。

Answer 2

没有办法阻止forEach。您可以通过检查forEach 中的标志来模拟停止，但这仍会循环遍历所有元素。顺便说一句，对 io 操作使用循环并不是最佳选择。

正如您所说，处理一组递增数据以进行处理的最佳方法是一个接一个地进行，但我会添加一个转折点：一个接一个地线程化。

注意：线程不是指实际线程。多拿一点 “多行工作”的定义。由于 IO 操作不锁定 主线程，当一个或多个请求正在等待数据时， 其他“工作线”可以运行 JavaScript 来处理数据 收到，因为 JavaScript 是单线程的（不谈论 WebWorkers)。

就像拥有一个页面数组一样简单，它接收要即时抓取的页面，以及一个读取该数组的一页，处理结果然后返回起点的函数（加载下一页数组并处理结果）。

现在您只需将该函数调用为您想要运行的线程数量，然后完成。伪代码：

var pages = [];

function loadNextPage() {
    if (pages.length == 0) {
        console.log("Thread ended");
        return;
    }
    var page = shift(); // get the first element
    loadAndProcessPage(page, loadNextPage);
}

loadAndProcessPage(page, callback) {
    requestOrWhatever(page, (error, data) => {
        if (error) {
            // retry or whatever
        } else {
            processData(data);
            callback();
        }
    });
}

function processData(data) {
    // Process the data and push new links to the pages array
    pages.push(data.link1);
    pages.push(data.link2);
    pages.push(data.link3);
}

console.log("Start new thread");
loadNextPage();

console.log("And another one");
loadNextPage();

console.log("And another one");
loadNextPage();

console.log("And another thread");
loadNextPage();

当数组中没有更多页面时，此代码将停止，并且如果在某个时候页面恰好少于线程数量，则线程将关闭。需要在这里和那里进行一些调整，但你明白了。

Answer 3

我假设您尝试在一些页面后停止执行（在您的示例中看起来像六个）。正如其他一些回复所述，您无法阻止从 Array.prototype.forEach() 执行回调，但是在每次执行时，您都可以阻止运行请求调用。

function getMeta(pages, callback) {
    var meta = []
    var i = 0
    pages.forEach(url => {
        // MaxPages you were looking for
        if(i <= maxPages)
            request((err, res, body) => {
                // ... Request logic
            })
    })

您还可以使用 while 循环来换行以遍历每个页面，一旦 i 达到您想要的值，循环将退出并且不会在其他页面上运行