取消 Abot 上的站点抓取答案

【问题标题】：Cancel Site Crawl on Abot取消 Abot 上的站点抓取
【发布时间】：2019-12-06 14:59:59
【问题描述】：

我有一个使用 Abot 抓取的域列表，目的是当它在其中一个站点上找到亚马逊链接时，它会退出，然后转到下一个站点。但我似乎看不到谁能退出页面抓取。例如

static Main(string[] args)
{
    var domains= new List<string> { "http://domain1", "http://domain2" };

    foreach (string domain in domains)
    {
        var config = new CrawlConfiguration
        {
            MaxPagesToCrawl = 100,
            MinCrawlDelayPerDomainMilliSeconds = 3000
        };

        var crawler = new PoliteWebCrawler(config);

        crawler.PageCrawlCompleted += PageCrawlCompleted;
        var uri = new Uri(domain);
        var crawlResult = crawler.Crawl(uri);
    }
}

private static void PageCrawlCompleted(object sender, PageCrawlCompletedArgs e)
{
    var crawledPage = e.CrawledPage;
    var crawlContext = e.CrawlContext;

    var document = crawledPage.AngleSharpHtmlDocument;
    var anchors = document.QuerySelectorAll("a").OfType<IHtmlAnchorElement>();
    var hrefs = anchors.Select(x => x.Href).ToList();

    var regEx= new Regex(@"https?:\/\/(www|smile)\.amazon(\.co\.uk|\.com).*");
    var resultList = hrefs.Where(f => regEx.IsMatch(f)).ToList();

    if (resultList.Any())
    {
        //NEED TO EXIT THE SITE CRAWL HERE
    }

}

【问题讨论】：

问题 - 我认为 - 是您试图从 void 方法控制循环流。我不太确定这个库，但根据抓取结果来查找链接可能会更好。
是的，问题是这个库在那个事件处理程序中执行页面爬取，它只能返回 void。我想我会进入源代码，看看它在做什么，然后覆盖它或创建一个新方法。

标签： c# web-crawler abot

【解决方案1】：

我建议以下...

var myCancellationToken = new CancellationTokenSource();
crawler.CrawlAsync(someUri, myCancellationToken);

private static void PageCrawlCompleted(object sender, PageCrawlCompletedArgs e)
{
    //More performant (since the parsing has already been done by Abot)
    var hasAmazonLinks = e.CrawledPage.ParsedLinks
      .Any(hl => hl.HrefValue.AbsoluteUri
         .ToLower()
         .Contains("amazon.com"));

    if (hasAmazonLinks)
    {
        //LOG SOMETHING BEFORE YOU STOP THE CRAWL!!!!!

        //Option A: Preferred method, Will clear all scheduled pages and cancel any threads that are currently crawling
        myCancellationToken.Cancel();

        //Option B: Same result as option A but no need to do anything with tokens. Not the preferred method. 
        e.CrawlContext.IsCrawlHardStopRequested = true;

        //Option C: Will clear all scheduled pages but will allow any threads that are currently crawling to complete. No cancellation tokens needed. Consider it a soft stop to the crawl.
        e.CrawlContext.IsCrawlStopRequested = true;
    }
}

【讨论】：

【解决方案2】：

PoliteWebCrawler 旨在开始抓取并深入挖掘网站 URL。如果您只想获取 URL 的内容（例如网站的第一页），您可以使用专为此类工作设计的 PageRequester。

var pageRequester = new PageRequester(new CrawlConfiguration(), new WebContentExtractor());

var crawledPage = await pageRequester.MakeRequestAsync(new Uri("http://google.com"));
Log.Logger.Information("{result}", new
{
    url = crawledPage.Uri,
    status = Convert.ToInt32(crawledPage.HttpResponseMessage.StatusCode)
});

顺便说一句，如果你想在这个过程中停止爬虫，你可以使用以下两种方法之一：

//1. hard crawl stop
crawlContext.CancellationTokenSource.Cancel();
//2. soft stop
crawlContext.IsCrawlStopRequested = true;

【讨论】：

感谢您的回复，但我需要爬取该站点直到找到亚马逊 URL，然后退出爬取该站点并转到下一个站点。