带有 Apache Beam 的 Google Cloud Dataflow 不显示日志

Question

我的问题是数据流上的日志没有显示任何内容（启用了监控 api），我不知道为什么。

使用以下 Apache Beam 代码（采用自 https://cloud.google.com/dataflow/docs/guides/logging），

import argparse
import logging
import re
from apache_beam.io import ReadFromText
from apache_beam.options.pipeline_options import PipelineOptions, SetupOptions
from apache_beam import FlatMap, Map, Pipeline


def run(argv=None):
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--input",
        dest="input",
        default="gs://dataflow-samples/shakespeare/kinglear.txt",
        help="Input file to process.",
    )
    known_args, pipeline_args = parser.parse_known_args(argv)

    pipeline_options = PipelineOptions(pipeline_args)
    pipeline_options.view_as(SetupOptions).save_main_session = True

    with Pipeline(options=pipeline_options) as p:
        filtered_words = (
            p
            | "Read" >> ReadFromText(known_args.input)
            | "Split" >> FlatMap(lambda x: re.findall(r"[A-Za-z\']+", x))
            | "Log" >> Map(lambda x: logging.info(f"x: {x}"))
        )


if __name__ == "__main__":
    logging.getLogger().setLevel(logging.INFO)
    run()

使用直接运行器在本地运行，

...
INFO:root:x: his
INFO:root:x: enemy
INFO:root:x: king
INFO:root:x: and
INFO:root:x: did
INFO:root:x: him
...

在 Google Cloud Dataflow 上运行时不会产生任何结果。

这里是依赖，

python = "^3.8"
apache-beam = {extras = ["gcp"], version = "^2.28.0"}

Answer 1

原来日志路由器中的默认接收器排除了数据流日志。

使用包含过滤器resource.type="dataflow_step" 在日志路由器中创建新接收器可解决此问题。