Stormcrawler 没有从网页中检索所有文本内容答案

【问题标题】：Stormcrawler not retrieving all text content from web pageStormcrawler 没有从网页中检索所有文本内容
【发布时间】：2021-07-11 17:22:24
【问题描述】：

我正在尝试使用 Stormcrawler 来抓取我们网站上的一组页面，虽然它能够检索和索引页面的一些文本，但它无法捕获页面上的大量其他文本。

我已经在运行 Ubuntu 18.04 的服务器上安装了 Zookeeper、Apache Storm 和 Stormcrawler，并使用了 here 提供的 Ansible playbooks（谢谢你！），以及 Elasticsearch 和 Kibana。在大多数情况下，我使用配置默认值，但进行了以下更改：

对于弹性索引映射，我启用了_source: true，并为所有属性（内容、主机、标题、网址）启用了索引和存储
在crawler-conf.yaml 配置中，我已注释掉所有textextractor.include.pattern 和textextractor.exclude.tags 设置，以强制捕获整个页面

在重新创建新的 ES 索引，运行 mvn clean package，然后启动爬虫拓扑之后，stormcrawler 开始做它的事情，内容开始出现在 Elasticsearch 中。但是，对于许多页面，检索和索引的内容只是页面上所有文本的子集，通常不包括我们感兴趣的主页文本。

例如，以下 XML 路径中的文本没有返回/索引：

<html> <body> <div#maincontentcontainer.container> <div#docs-container> <div> <div.row> <div.col-lg-9.col-md-8.col-sm-12.content-item> <div> <div> <p> (text)

虽然此路径中的文本返回：

<html> <body> <div> <div.container> <div.row> <p> (text)

除了注释掉所有特定的标记包含和排除模式之外，是否需要进行任何其他配置更改？根据我对文档的理解，这些选项的默认设置是强制对整个页面进行索引。

我将非常感谢任何帮助。感谢您提供出色的软件。

以下是我的配置文件：

crawler-conf.yaml

config:
  topology.workers: 3
  topology.message.timeout.secs: 1000
  topology.max.spout.pending: 100
  topology.debug: false

  fetcher.threads.number: 100

  # override the JVM parameters for the workers
  topology.worker.childopts: "-Xmx2g -Djava.net.preferIPv4Stack=true"

  # mandatory when using Flux
  topology.kryo.register:
    - com.digitalpebble.stormcrawler.Metadata

  # metadata to transfer to the outlinks
  # metadata.transfer:
  # - customMetadataName

  # lists the metadata to persist to storage
  metadata.persist:
   - _redirTo
   - error.cause
   - error.source
   - isSitemap
   - isFeed
   
  http.agent.name: "My crawler"
  http.agent.version: "1.0"
  http.agent.description: ""
  http.agent.url: ""
  http.agent.email: ""

  # The maximum number of bytes for returned HTTP response bodies.
  http.content.limit: -1

  # FetcherBolt queue dump => comment out to activate
  # fetcherbolt.queue.debug.filepath: "/tmp/fetcher-dump-{port}"

  parsefilters.config.file: "parsefilters.json"
  urlfilters.config.file: "urlfilters.json"

  # revisit a page daily (value in minutes)
  fetchInterval.default: 1440

  # revisit a page with a fetch error after 2 hours (value in minutes)
  fetchInterval.fetch.error: 120

  # never revisit a page with an error (or set a value in minutes)
  fetchInterval.error: -1

  # text extraction for JSoupParserBolt
  # textextractor.include.pattern:
  #  - DIV[id="maincontent"]
  #  - DIV[itemprop="articleBody"]
  #  - ARTICLE

  # textextractor.exclude.tags:
  #  - STYLE
  #  - SCRIPT

  # configuration for the classes extending AbstractIndexerBolt
  # indexer.md.filter: "someKey=aValue"
  indexer.url.fieldname: "url"
  indexer.text.fieldname: "content"
  indexer.canonical.name: "canonical"
  indexer.md.mapping:
  - parse.title=title
  - parse.keywords=keywords
  - parse.description=description
  - domain=domain

  # Metrics consumers:
  topology.metrics.consumer.register:
     - class: "org.apache.storm.metric.LoggingMetricsConsumer"
       parallelism.hint: 1

  http.protocol.implementation: "com.digitalpebble.stormcrawler.protocol.selenium.RemoteDriverProtocol"
  https.protocol.implementation: "com.digitalpebble.stormcrawler.protocol.selenium.RemoteDriverProtocol"
  selenium.addresses: "http://localhost:9515"

es-conf.yaml

config:
  # ES indexer bolt
  es.indexer.addresses: "localhost"
  es.indexer.index.name: "content"
  # es.indexer.pipeline: "_PIPELINE_"
  es.indexer.create: false
  es.indexer.bulkActions: 100
  es.indexer.flushInterval: "2s"
  es.indexer.concurrentRequests: 1
  
  # ES metricsConsumer
  es.metrics.addresses: "http://localhost:9200"
  es.metrics.index.name: "metrics"
  
  # ES spout and persistence bolt
  es.status.addresses: "http://localhost:9200"
  es.status.index.name: "status"
  es.status.routing: true
  es.status.routing.fieldname: "key"
  es.status.bulkActions: 500
  es.status.flushInterval: "5s"
  es.status.concurrentRequests: 1
  
    # spout config #
    
  # positive or negative filters parsable by the Lucene Query Parser
  # es.status.filterQuery: 
  #  - "-(key:stormcrawler.net)"
  #  - "-(key:digitalpebble.com)"

  # time in secs for which the URLs will be considered for fetching after a ack of fail
  spout.ttl.purgatory: 30
  
  # Min time (in msecs) to allow between 2 successive queries to ES
  spout.min.delay.queries: 2000

  # Delay since previous query date (in secs) after which the nextFetchDate value will be reset to the current time
  spout.reset.fetchdate.after: 120

  es.status.max.buckets: 50
  es.status.max.urls.per.bucket: 2
  # field to group the URLs into buckets
  es.status.bucket.field: "key"
  # fields to sort the URLs within a bucket
  es.status.bucket.sort.field: 
   - "nextFetchDate"
   - "url"
  # field to sort the buckets
  es.status.global.sort.field: "nextFetchDate"

  # CollapsingSpout : limits the deep paging by resetting the start offset for the ES query 
  es.status.max.start.offset: 500
  
  # AggregationSpout : sampling improves the performance on large crawls
  es.status.sample: false

  # max allowed duration of a query in sec 
  es.status.query.timeout: -1

  # AggregationSpout (expert): adds this value in mins to the latest date returned in the results and
  # use it as nextFetchDate
  es.status.recentDate.increase: -1
  es.status.recentDate.min.gap: -1

  topology.metrics.consumer.register:
       - class: "com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer"
         parallelism.hint: 1
         #whitelist:
         #  - "fetcher_counter"
         #  - "fetcher_average.bytes_fetched"
         #blacklist:
         #  - "__receive.*"

es-crawler.flux

name: "crawler"

includes:
    - resource: true
      file: "/crawler-default.yaml"
      override: false

    - resource: false
      file: "crawler-conf.yaml"
      override: true

    - resource: false
      file: "es-conf.yaml"
      override: true

spouts:
  - id: "spout"
    className: "com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout"
    parallelism: 10

  - id: "filespout"
    className: "com.digitalpebble.stormcrawler.spout.FileSpout"
    parallelism: 1
    constructorArgs:
      - "."
      - "seeds.txt"
      - true

bolts:
  - id: "filter"
    className: "com.digitalpebble.stormcrawler.bolt.URLFilterBolt"
    parallelism: 3
  - id: "partitioner"
    className: "com.digitalpebble.stormcrawler.bolt.URLPartitionerBolt"
    parallelism: 3
  - id: "fetcher"
    className: "com.digitalpebble.stormcrawler.bolt.FetcherBolt"
    parallelism: 3
  - id: "sitemap"
    className: "com.digitalpebble.stormcrawler.bolt.SiteMapParserBolt"
    parallelism: 3
  - id: "parse"
    className: "com.digitalpebble.stormcrawler.bolt.JSoupParserBolt"
    parallelism: 12
  - id: "index"
    className: "com.digitalpebble.stormcrawler.elasticsearch.bolt.IndexerBolt"
    parallelism: 3
  - id: "status"
    className: "com.digitalpebble.stormcrawler.elasticsearch.persistence.StatusUpdaterBolt"
    parallelism: 3
  - id: "status_metrics"
    className: "com.digitalpebble.stormcrawler.elasticsearch.metrics.StatusMetricsBolt"
    parallelism: 3

streams:
  - from: "spout"
    to: "partitioner"
    grouping:
      type: SHUFFLE
      
  - from: "spout"
    to: "status_metrics"
    grouping:
      type: SHUFFLE     

  - from: "partitioner"
    to: "fetcher"
    grouping:
      type: FIELDS
      args: ["key"]

  - from: "fetcher"
    to: "sitemap"
    grouping:
      type: LOCAL_OR_SHUFFLE

  - from: "sitemap"
    to: "parse"
    grouping:
      type: LOCAL_OR_SHUFFLE

  - from: "parse"
    to: "index"
    grouping:
      type: LOCAL_OR_SHUFFLE

  - from: "fetcher"
    to: "status"
    grouping:
      type: FIELDS
      args: ["url"]
      streamId: "status"

  - from: "sitemap"
    to: "status"
    grouping:
      type: FIELDS
      args: ["url"]
      streamId: "status"

  - from: "parse"
    to: "status"
    grouping:
      type: FIELDS
      args: ["url"]
      streamId: "status"

  - from: "index"
    to: "status"
    grouping:
      type: FIELDS
      args: ["url"]
      streamId: "status"

  - from: "filespout"
    to: "filter"
    grouping:
      type: FIELDS
      args: ["url"]
      streamId: "status"

  - from: "filter"
    to: "status"
    grouping:
      streamId: "status"
      type: CUSTOM
      customClass:
        className: "com.digitalpebble.stormcrawler.util.URLStreamGrouping"
        constructorArgs:
          - "byDomain"

parsefilters.json

{
  "com.digitalpebble.stormcrawler.parse.ParseFilters": [
    {
      "class": "com.digitalpebble.stormcrawler.parse.filter.XPathFilter",
      "name": "XPathFilter",
      "params": {
        "canonical": "//*[@rel=\"canonical\"]/@href",
        "parse.description": [
            "//*[@name=\"description\"]/@content",
            "//*[@name=\"Description\"]/@content"
         ],
        "parse.title": [
            "//TITLE",
            "//META[@name=\"title\"]/@content"
         ],
         "parse.keywords": "//META[@name=\"keywords\"]/@content"
      }
    },
    {
      "class": "com.digitalpebble.stormcrawler.parse.filter.LinkParseFilter",
      "name": "LinkParseFilter",
      "params": {
         "pattern": "//FRAME/@src"
       }
    },
    {
      "class": "com.digitalpebble.stormcrawler.parse.filter.DomainParseFilter",
      "name": "DomainParseFilter",
      "params": {
        "key": "domain",
        "byHost": false
       }
    },
    {
      "class": "com.digitalpebble.stormcrawler.parse.filter.CommaSeparatedToMultivaluedMetadata",
      "name": "CommaSeparatedToMultivaluedMetadata",
      "params": {
        "keys": ["parse.keywords"]
       }
    }
  ]
}

尝试使用 Chromedriver

我为 Ubuntu 安装了最新版本的 Chromedriver 和 Google Chrome。

首先我以stormcrawler用户身份在localhost:9515以headless模式启动chromedriver（通过一个单独的python shell，如下所示），然后我重新启动stormcrawler拓扑（也作为stormcrawler用户），但最终得到一堆与 Chrome 相关的错误。然而奇怪的是，我可以直接在 Python shell 中确认 chromedriver 运行正常，并且我可以通过ps -ef 确认驱动程序和浏览器都在积极运行。当我尝试从命令行简单地启动 chromedriver（即chromedriver --headless &）时，也会出现同样的错误堆栈。

以无头模式启动 chromedriver（在 python3 shell 中）

from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--no-sandbox')
options.add_argument('--headless')
options.add_argument('--window-size=1200x600')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--disable-setuid-sandbox')
options.add_argument('--disable-extensions')
options.add_argument('--disable-infobars')
options.add_argument('--remote-debugging-port=9222')
options.add_argument('--user-data-dir=/home/stormcrawler/cache/google/chrome')
options.add_argument('--disable-gpu')
options.add_argument('--profile-directory=Default')
options.binary_location = '/usr/bin/google-chrome'
driver = webdriver.Chrome(chrome_options=options, port=9515, executable_path=r'/usr/bin/chromedriver')

从启动stormcrawler拓扑的堆栈跟踪 运行命令：storm jar target/stormcrawler-1.0-SNAPSHOT.jar org.apache.storm.flux.Flux --local es-crawler.flux --sleep 60000

9486 [Thread-26-fetcher-executor[3 3]] ERROR o.a.s.util - Async loop died!
java.lang.RuntimeException: org.openqa.selenium.WebDriverException: unknown error: Chrome failed to start: exited abnormally.
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Build info: version: '4.0.0-alpha-6', revision: '5f43a29cfc'
System info: host: 'stormcrawler-dev', ip: '127.0.0.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.15.0-33-generic', java.version: '1.8.0_282'
Driver info: driver.version: RemoteWebDriver
remote stacktrace: #0 0x55d590b21e89 <unknown>

    at com.digitalpebble.stormcrawler.protocol.selenium.RemoteDriverProtocol.configure(RemoteDriverProtocol.java:101) ~[stormcrawler-1.0-SNAPSHOT.jar:?]
    at com.digitalpebble.stormcrawler.protocol.ProtocolFactory.<init>(ProtocolFactory.java:69) ~[stormcrawler-1.0-SNAPSHOT.jar:?]
    at com.digitalpebble.stormcrawler.bolt.FetcherBolt.prepare(FetcherBolt.java:818) ~[stormcrawler-1.0-SNAPSHOT.jar:?]
    at org.apache.storm.daemon.executor$fn__10180$fn__10193.invoke(executor.clj:803) ~[storm-core-1.2.3.jar:1.2.3]
    at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
    at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: org.openqa.selenium.WebDriverException: unknown error: Chrome failed to start: exited abnormally.
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
...

确认 chromedriver 和 chrome 都在运行且可访问

~/stormcrawler$ ps -ef | grep -i 'driver'
stormcr+ 18862 18857  0 14:28 pts/0    00:00:00 /usr/bin/chromedriver --port=9515
stormcr+ 18868 18862  0 14:28 pts/0    00:00:00 /usr/bin/google-chrome --disable-background-networking --disable-client-side-phishing-detection --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-gpu --disable-hang-monitor --disable-infobars --disable-popup-blocking --disable-prompt-on-repost --disable-setuid-sandbox --disable-sync --enable-automation --enable-blink-features=ShadowDOMV0 --enable-logging --headless --log-level=0 --no-first-run --no-sandbox --no-service-autorun --password-store=basic --profile-directory=Default --remote-debugging-port=9222 --test-type=webdriver --use-mock-keychain --user-data-dir=/home/stormcrawler/cache/google/chrome --window-size=1200x600
stormcr+ 18899 18877  0 14:28 pts/0    00:00:00 /opt/google/chrome/chrome --type=renderer --no-sandbox --disable-dev-shm-usage --enable-automation --enable-logging --log-level=0 --remote-debugging-port=9222 --test-type=webdriver --allow-pre-commit-input --ozone-platform=headless --field-trial-handle=17069524199442920904,10206176048672570859,131072 --disable-gpu-compositing --enable-blink-features=ShadowDOMV0 --lang=en-US --headless --enable-crash-reporter --lang=en-US --num-raster-threads=1 --renderer-client-id=4 --shared-files=v8_context_snapshot_data:100

~/stormcrawler$ sudo netstat -lp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 localhost:9222          0.0.0.0:*               LISTEN      18026/google-chrome 
tcp        0      0 localhost:9515          0.0.0.0:*               LISTEN      18020/chromedriver

【问题讨论】：

嗨，丹尼斯，您要获取的页面的 URL 是否公开可用？我想看看我是否可以重现这个问题。对于弹性索引映射，如果源设置为 true，则您无需显式设置要存储和索引的字段。
我忘记了两个重要问题：您使用哪个版本的 SC？你能分享你的 parsefilter.json 文件吗？
@JulienNioche 当然，我已将 parsefilters.json 文件添加到我的帖子中。我正在使用 Storm 1.2.3、Zookeeper 3.5.6 和 Storm-crawler-elasticsearch-archetype 1.17。
@JulienNioche，哎呀，我很抱歉。这似乎只是一个javascript问题。我会尝试在 Stormcrawler 中使用 Selenium 组件...
github.com/DigitalPebble/storm-crawler/blob/master/core/src/… 接受地址和端口 - 这应该适用于 Chromedriver

标签： stormcrawler

【解决方案1】：

IIRC 您需要设置一些additional config 才能使用 ChomeDriver。

或者（尚未尝试）https://hub.docker.com/r/browserless/chrome 将是在 Docker 容器中处理 Chrome 的好方法。

【讨论】：

谢谢，我最终能够使用您共享的其他配置设置使其正常工作。但是，这最终导致我发现了一个单独但相关的错误，我应该向您指出 - selenium.pageLoadTimeout 的默认设置为 -1 存在问题，导致以下错误：java.lang.RuntimeException: org.openqa.selenium.InvalidArgumentException: invalid argument: value must be a non-negative integer。我需要在我的 crawler-conf.yaml 文件中将此参数显式设置为 >0 以使拓扑工作。
另外，由于我最初的问题最终变成了一个不同的问题，我计划将 chromedriver 设置作为一个单独的问题重新发布，并可以自己回答我最终设置的详细信息。
很高兴你让它工作。如果你认为 selenium.pageLoadTimeout 是一个错误，请在 Github 上打开一个问题。谢谢
当然，完成。刚刚在此处发布了 chromedriver 设置的问答：stackoverflow.com/questions/67320758/…