如何在 google cloudbuild 步骤中保留变量？答案

【问题标题】：How to persist variables in google cloudbuild steps?如何在 google cloudbuild 步骤中保留变量？
【发布时间】：2020-12-29 23:03:21
【问题描述】：

我有一个 cloudbuild.json，用于将管道上传到 gcp kubeflow。现在我想添加另一个步骤，我想在其中获取最新的管道 ID，然后将管道作为实验运行。所以我的主要问题是我应该如何在后续步骤中获取管道 ID。我编写了一个小脚本来获取最新的管道 ID，并将其添加为从 docker 运行的步骤，但现在我不确定如何获取此管道 ID。

这是我的 coudbuild.json

{
    "steps": [
        {
            "name": "gcr.io/cloud-builders/docker",
            "args": [
                "build",
                "-t",
                "trainer_image",
                "."
            ],
            "dir": "./trainer_image/"
        },
        {
            "name": "gcr.io/cloud-builders/docker",
            "args": [
                "build",
                "-t",
                "base_image",
                "."
            ],
            "dir": "./base_image/"
        },
        {
            "name": "gcr.io/dmgcp-pkg-internal-poc-oct-04/kfp-cli",
            "args": [
                "-c",
                "dsl-compile --py covertype_training_pipeline.py --output covertype_training_pipeline.yaml"
            ],
            "env": [
                "BASE_IMAGE=gcr.io/dmgcp-pkg-internal-poc-oct-04/base_image:test",
                "TRAINER_IMAGE=gcr.io/dmgcp-pkg-internal-poc-oct-04/trainer_image:test",
                "RUNTIME_VERSION=1.15",
                "PYTHON_VERSION=3.7",
                "COMPONENT_URL_SEARCH_PREFIX=https://raw.githubusercontent.com/kubeflow/pipelines/0.2.5/components/gcp/",
                "USE_KFP_SA=False"
            ],
            "dir": "./pipeline/"
        },
        {
            "name": "gcr.io/dmgcp-pkg-internal-poc-oct-04/kfp-cli",
            "args": [
                "-c",
                "kfp --endpoint 66df1d31e46e6510-dot-us-central2.pipelines.googleusercontent.com pipeline upload -p credit_fraud_training_test covertype_training_pipeline.yaml"
            ],
            "dir": "./pipeline/"
        },
        {
            "name": "gcr.io/cloud-builders/docker",
            "args": [
                "build",
                "-t",
                "id_image",
                "."
            ],
            "dir": "./id_image/"
        }
    ],
    "images": [
        "gcr.io/dmgcp-pkg-internal-poc-oct-04/trainer_image:test",
        "gcr.io/dmgcp-pkg-internal-poc-oct-04/base_image:test"
    ]
}

这是我获取最新管道 ID 的 python 脚本

import kfp
client = kfp.Client(host='66df1d31e46e6510-dot-us-central2.pipelines.googleusercontent.com')
pipelines = client.list_pipelines()
total_pipeline = len(pipelines.pipelines)
latest_pipeline_id = pipelines.pipelines[total_pipeline-1].id
with open('/workspace/pipeline.txt') as file:
    file.write(latest_pipeline_id)

【问题讨论】：

标签： python docker google-cloud-platform kubeflow-pipelines google-ai-platform

【解决方案1】：

您不能在步骤之间保留变量。您只能持久化文件。但是在一个步骤中，您可以重用局部变量。

- name: "gcr.io/cloud-builders/docker",
  entrypoint: "bash"
  args: 
   - "-c"
   - |
      export DOCKER_VERSION=$(docker version)
      echo $$DOCKER_VERSION #Reuse local variable
      echo $$DOCKER_VERSION > DOCKER_VERSION.txt #Save the variable for the next step
- name: "gcr.io/cloud-builders/docker",
  entrypoint: "bash"
  args: 
   - "-c"
   - |
      export DOCKER_VERSION=$(cat DOCKER_VERSION.txt) #Get the value saved in file
      echo $$DOCKER_VERSION #Reuse local variable

【讨论】：