crawler4j 只抓取种子 URL答案

【问题标题】：crawler4j crawls only seed URLscrawler4j 只抓取种子 URL
【发布时间】：2016-05-12 21:04:31
【问题描述】：

为什么下面的代码基于crawler4j 只爬取给定的种子URL，而不开始爬取其他链接？

public static void main( String[] args )
{
      String crawlStorageFolder = "F:\\crawl";
      int numberOfCrawlers = 7;

      CrawlConfig config = new CrawlConfig();
      config.setCrawlStorageFolder(crawlStorageFolder);
      config.setMaxDepthOfCrawling(4);
      /*
       * Instantiate the controller for this crawl.
       */
      PageFetcher pageFetcher = new PageFetcher(config);

      RobotstxtConfig robotstxtConfig = new RobotstxtConfig();
      robotstxtConfig.setEnabled(false);

      RobotstxtServer robotstxtServer = new RobotstxtServer(robotstxtConfig, pageFetcher);
      CrawlController controller = null;
        try {
            controller = new CrawlController(config, pageFetcher, robotstxtServer);
        } catch (Exception e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

      /*
       * For each crawl, you need to add some seed urls. These are the first
       * URLs that are fetched and then the crawler starts following links
       * which are found in these pages
       */
      controller.addSeed("http://edition.cnn.com/2016/05/11/politics/paul-ryan-donald-trump-meeting/index.html");        

      /*
       * Start the crawl. This is a blocking operation, meaning that your code
       * will reach the line after this only when crawling is finished.
       */
      controller.start(MyCrawler.class, numberOfCrawlers);

  }

【问题讨论】：

你能把shouldVisit方法的代码贴在MyCrawler.class中吗？
谢谢，我的错。在示例中 shouldVisit 包含他们的网站作为“必须拥有”域 - 错过了 - 现在一切正常:)
我就是这么想的 :)

标签： web-crawler crawler4j

【解决方案1】：

官方示例仅限于www.ics.uci.edu域。因此，需要对扩展Crawler类中的shouldVisit方法进行适配。

 /**
   * You should implement this function to specify whether the given url
   * should be crawled or not (based on your crawling logic).
   */
  @Override
  public boolean shouldVisit(Page referringPage, WebURL url) {
    String href = url.getURL().toLowerCase();
    // Ignore the url if it has an extension that matches our defined set of image extensions.
    if (IMAGE_EXTENSIONS.matcher(href).matches()) {
      return false;
    }

    // Only accept the url if it is in the "www.ics.uci.edu" domain and protocol is "http".
    return href.startsWith("http://www.ics.uci.edu/");
  }

【讨论】：