如何在 AppEngine MapReduce 中获得计数器结果？答案

【问题标题】：How to get counter results in AppEngine MapReduce?如何在 AppEngine MapReduce 中获得计数器结果？
【发布时间】：2014-05-21 05:09:46
【问题描述】：

我正在使用谷歌mapreduce lib 来处理我的数据。在处理数据时，在映射器函数中使用计数器。但我不知道如何在 finalized 方法中获得计数器结果。

def mapper(obj):
    yield obj
    yield operation.counters.Increment("process-obj")


class Test(base_handler.PipelineBase):
    """A pipeline to ingest log as CSV in Google Storage
    """

    def run(self, setting_id):
        filepath = yield mapreduce_pipeline.MapperPipeline(
            "test",
            "mapper",
            "mapreduce.input_readers.DatastoreInputReader",
            output_writer_spec="mapreduce.output_writers.FileOutputWriter",
            params={

            },
            shards=10
        )
    def finalized(self):
        # how to read the counter process-obj
        # how to get the setting_id
        pass

【问题讨论】：

标签： python google-app-engine mapreduce

【解决方案1】：

命名输出可能是您正在寻找的。您可以找到更多详细信息here。

这是您的代码，使用命名输出来取回各种计数器，包括您定义的计数器：

def mapper(obj):
    yield obj
    yield operation.counters.Increment("process-obj")


class Test(base_handler.PipelineBase):
    """A pipeline to ingest log as CSV in Google Storage
    """

    output_names = ['counters']

    def run(self, setting_id):
        results = yield mapreduce_pipeline.MapperPipeline(
          "test",
          "mapper",
          "mapreduce.input_readers.DatastoreInputReader",
          output_writer_spec="mapreduce.output_writers.FileOutputWriter",
          params={

          },
          shards=10
        )
        yield MapreduceResult(results.counters)

    def finalized(self):
        print 'Counters here: ', self.outputs.counters


class MapreduceResult(base_handler.PipelineBase):

    def run(self, counters):
        self.fill(self.outputs.counters, counters)

【讨论】：