Go prometheus 客户端使用 Python prometheus 客户端获得不同的延迟答案

【问题标题】：Go prometheus client gets different latency with Python prometheus clientGo prometheus 客户端使用 Python prometheus 客户端获得不同的延迟
【发布时间】：2020-06-06 01:00:11
【问题描述】：

我正在阅读latency measurement codes，因为我发现测量的延迟高于 python 代码测量的端到端延迟。 下面InstrumentRoundTripperDuration的用法对吗？我在网上没有找到类似的例子。

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)


// metrics client
func NewClientMetrics(spec *v1.PredictorSpec, deploymentName string, modelName string) *ClientMetrics {
    histogram := prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    ClientRequestsMetricName,
            Help:    "A histogram of latencies for client calls from executor",
            Buckets: prometheus.DefBuckets,
        },
        []string{DeploymentNameMetric, PredictorNameMetric, PredictorVersionMetric, ServiceMetric, ModelNameMetric, ModelImageMetric, ModelVersionMetric, "method", "code"},
    )

    err := prometheus.Register(histogram)
    if err != nil {
        prometheus.Unregister(histogram)
        prometheus.Register(histogram)
    }
    ...

    return &ClientMetrics{
        ClientHandledHistogram: histogram,
        Predictor:              spec,
        DeploymentName:         deploymentName,
        ModelName:              modelName,
        ImageName:              imageName,
        ImageVersion:           imageVersion,
    }
}

// In json rest client, construct metrics client
func NewJSONRestClient(protocol string, deploymentName string, predictor *v1.PredictorSpec, annotations map[string]string, options ...BytesRestClientOption) (client.SeldonApiClient, error) {

    httpClient := http.DefaultClient
...

    client := JSONRestClient{
        httpClient,
        logf.Log.WithName("JSONRestClient"),
        protocol,
        deploymentName,
        predictor,
        metric.NewClientMetrics(predictor, deploymentName, ""),
    }
    ...

    return &client, nil
}

// In Json rest client, create a function to measure roundtrip latency
func (smc *JSONRestClient) getMetricsRoundTripper(modelName string, service string) http.RoundTripper {
...
    return promhttp.InstrumentRoundTripperDuration(smc.metrics.ClientHandledHistogram.MustCurryWith(prometheus.Labels{
        metric.DeploymentNameMetric:   smc.DeploymentName,
        metric.PredictorNameMetric:    smc.predictor.Name,
        metric.PredictorVersionMetric: smc.predictor.Annotations["version"],
        metric.ServiceMetric:          service,
        metric.ModelNameMetric:        modelName,
        metric.ModelImageMetric:       imageName,
        metric.ModelVersionMetric:     imageVersion,
    }), http.DefaultTransport)
}

// http call (getMetricsRoundTripper is called middleware)
client := smc.httpClient
client.Transport = smc.getMetricsRoundTripper(modelName, method)

源代码：

https://github.com/SeldonIO/seldon-core/blob/master/executor/api/rest/client.go#L88 和第 148、118 行

https://github.com/SeldonIO/seldon-core/blob/master/executor/api/metric/client.go#L22

谢谢

【问题讨论】：

标签： go prometheus latency prometheus-alertmanager

【解决方案1】：

Python prometheus 客户端与 go prometheus 客户端有不同的存储桶。前者使用Buckets: []float64{0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10}。

但后者使用的是[]float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10}（code）。

他们使用不同的存储桶，因此指标可能看起来不同。

【讨论】：