【问题标题】:Generate templates in yaml from a CSV file从 CSV 文件在 yaml 中生成模板
【发布时间】:2021-11-23 20:35:06
【问题描述】:

我正在尝试使用我的变量从模板创建 yaml 文件。 我的 yaml 模板是这样的

number: {{NUMBER}}
  name: {{NAME}}
  region: {{REGION}}
  storenum: {{STORENUM}}
  clients: {{CLIENTS}}
  tags: {{TAGS}}


storename: {{STORENAME}}
employee: {{EMPLOYEE}}
products: {{PRODUCTS}}

但我的变量在 CSV 文件中,结构就是变量。

Number - Name - Region - Storenum  
StoreX - StoreX - New York - 30  

我现在有一个小脚本,可以从带有可变参数的模板创建 像这样的模板 script.sh template.yml -f variables.txt。 我的结果是这样的

number: 37579922
  name: Store1
  region: New York
  storenum: 32
  clients: 100
  tags: stores


storename: Store newyork
employee: 10
products: 200

但我只能一一做。有什么方法可以读取 CSV 参数并发送到程序并从 CSV 参数生成例如 Template1、Template2、..?任何帮助

#!/bin/bash
readonly PROGNAME=$(basename $0)

config_file="<none>"
print_only="false"
silent="false"

usage="${PROGNAME} [-h] [-d] [-f] [-s] -- 

where:
    -h, --help
        Show this help text
    -p, --print
        Don't do anything, just print the result of the variable expansion(s)
    -f, --file
        Specify a file to read variables from
    -s, --silent
        Don't print warning messages (for example if no variables are found)

examples:
    VAR1=Something VAR2=1.2.3 ${PROGNAME} test.txt 
    ${PROGNAME} test.txt -f my-variables.txt
    ${PROGNAME} test.txt -f my-variables.txt > new-test.txt"

if [ $# -eq 0 ]; then
  echo "$usage"
  exit 1    
fi

if [[ ! -f "${1}" ]]; then
    echo "You need to specify a template file" >&2
    echo "$usage"
    exit 1
fi

template="${1}"

if [ "$#" -ne 0 ]; then
    while [ "$#" -gt 0 ]
    do
        case "$1" in
        -h|--help)
            echo "$usage"
            exit 0
            ;;        
        -p|--print)
            print_only="true"
            ;;
        -f|--file)
            config_file="$2"
            ;;
        -s|--silent)
            silent="true"
            ;;
        --)
            break
            ;;
        -*)
            echo "Invalid option '$1'. Use --help to see the valid options" >&2
            exit 1
            ;;
        # an option argument, continue
        *)  ;;
        esac
        shift
    done
fi

vars=$(grep -oE '\{\{[A-Za-z0-9_]+\}\}' "${template}" | sort | uniq | sed -e 's/^{{//' -e 's/}}$//')

if [[ -z "$vars" ]]; then
    if [ "$silent" == "false" ]; then
        echo "Warning: No variable was found in ${template}, syntax is {{VAR}}" >&2
    fi
fi

# Load variables from file if needed
if [ "${config_file}" != "<none>" ]; then
    if [[ ! -f "${config_file}" ]]; then
      echo "The file ${config_file} does not exists" >&2
      echo "$usage"      
      exit 1
    fi

    source "${config_file}"
fi    

var_value() {
    eval echo \$$1
}

replaces=""

# Reads default values defined as {{VAR=value}} and delete those lines
# There are evaluated, so you can do {{PATH=$HOME}} or {{PATH=`pwd`}}
# You can even reference variables defined in the template before
defaults=$(grep -oE '^\{\{[A-Za-z0-9_]+=.+\}\}' "${template}" | sed -e 's/^{{//' -e 's/}}$//')

for default in $defaults; do
    var=$(echo "$default" | grep -oE "^[A-Za-z0-9_]+")
    current=`var_value $var`

    # Replace only if var is not set
    if [[ -z "$current" ]]; then
        eval $default
    fi

    # remove define line
    replaces="-e '/^{{$var=/d' $replaces"
    vars="$vars
$current"
done

vars=$(echo $vars | sort | uniq)

if [[ "$print_only" == "true" ]]; then
    for var in $vars; do
        value=`var_value $var`
        echo "$var = $value"
    done
    exit 0
fi

# Replace all {{VAR}} by $VAR value
for var in $vars; do
    value=$(var_value $var | sed -e "s;\&;\\\&;g" -e "s;\ ;\\\ ;g") # '&' and <space> is escaped 
    if [[ -z "$value" ]]; then
        if [ $silent == "false" ]; then
            echo "Warning: $var is not defined and no default is set, replacing by empty" >&2
        fi
    fi

    # Escape slashes
    value=$(echo "$value" | sed 's/\//\\\//g');
    replaces="-e 's/{{$var}}/${value}/g' $replaces"    
done

escaped_template_path=$(echo $template | sed 's/ /\\ /g')
eval sed $replaces "$escaped_template_path"

【问题讨论】:

  • 请使用 CSV 文件中的一些(完整)示例行更新问题; CSV 文件中的值是否始终位于同一位置,或者我们是否需要解析标题行以确定哪个值在哪个字段中?还使用预期输出更新问题(匹配 CSV 文件中的示例行);最后,考虑将代码块缩减为只需要解析 CSV 文件并生成输出的代码(即,我们不需要查看 usage 也不需要查看所有命令行选项处理)
  • 我没有尝试破译所有代码,但对defaults 的引用让我产生疑问......如果 CSV 文件没有模板字段的值,我们该怎么办?我们如何知道会发生这种情况以及我们应该使用哪些“默认”值(即显示包含“默认”值的文件和/或结构)
  • 你好@markp-fuso,CSV中的所有字段都相同,结构相同。 Number - Name - Region - Storenum - ..

标签: bash csv templates yaml


【解决方案1】:

但我只能一一做。有什么方法可以读取 CSV 参数并发送到程序并从 CSV 参数生成例如 Template1、Template2、..?任何帮助

我想不出一个完整的解决方案,但我有一个可以创建多个文件的解决方案。只需将其应用到您的脚本中即可。


file.csv

Number,Name,Region,Storenum,Clients,Tags,Storename,Employee,Products
88899223,Store1,New York,30,100,stores,Store newyork,10,200
37579922,Store2,Chicago,30,1000,stores,Store Chicago,10,200
77777777,Store3,New Orleans,309,100,stores,Store New Orleans,10,200
55555555,Store4,New Jersey,50,100,stores,Store Jersey,10,200
44444444,Store5,Connecticut,90,100,stores,Store Connecticut,10,200
33333333,Store6,Michigan,,900,stores,Store Michigan,10,200
22222222,Store7,Texas,30,200,stores,,10,200

Template.yaml

number: {{NUMBER}}
  name: {{NAME}}
  region: {{REGION}}
  storenum: {{STORENUM}}
  clients: {{CLIENTS}}
  tags: {{TAGS}}


storename: {{STORENAME}}
employee: {{EMPLOYEE}}
products: {{PRODUCTS}}

#!/usr/bin/env bash

n=1
skip=0
start=-1

while IFS=, read -r number name region storenum clients tags storename employee products; do
  if ((start++ >= skip)); then
    export NUMBER="${number:-none}" NAME="${name:-none}" REGION="${region:-none}" \
      STORENUM="${storenum:-none}" CLIENTS="${clients:-none}" TAGS="${tags:-none}" \
      STORENAME="${storename:-none}" EMPLOYEE="${employee:-none}" PRODUCTS="${products:-none}"
   sed 's/{{/$/;s/}}//g' Template.yaml |
   envsubst '$NUMBER $NAME $REGION $STORENUM $CLIENTS $TAGS $STORENAME $EMPLOYEE $PRODUCTS' > "Template$n"
  ((n++))
  fi
done < file.csv

通过运行检查创建的文件:

tail -n+1 Template[0-9]*

输出是:

==> Template1 <==
number: 88899223
  name: Store1
  region: New York
  storenum: 30
  clients: 100
  tags: stores


storename: Store newyork
employee: 10
products: 200


==> Template2 <==
number: 37579922
  name: Store2
  region: Chicago
  storenum: 30
  clients: 1000
  tags: stores


storename: Store Chicago
employee: 10
products: 200


==> Template3 <==
number: 77777777
  name: Store3
  region: New Orleans
  storenum: 309
  clients: 100
  tags: stores


storename: Store New Orleans
employee: 10
products: 200


==> Template4 <==
number: 55555555
  name: Store4
  region: New Jersey
  storenum: 50
  clients: 100
  tags: stores


storename: Store Jersey
employee: 10
products: 200


==> Template5 <==
number: 44444444
  name: Store5
  region: Connecticut
  storenum: 90
  clients: 100
  tags: stores


storename: Store Connecticut
employee: 10
products: 200


==> Template6 <==
number: 33333333
  name: Store6
  region: Michigan
  storenum: none
  clients: 900
  tags: stores


storename: Store Michigan
employee: 10
products: 200


==> Template7 <==
number: 22222222
  name: Store7
  region: Texas
  storenum: 30
  clients: 200
  tags: stores


storename: none
employee: 10
products: 200


==> Template8 <==
number: 22222222
  name: Store7
  region: Texas
  storenum: 30
  clients: 200
  tags: stores


storename: none
employee: 10
products: 200

  • 上述脚本需要/需要envsubstsed,因为exportbash shell 的内置函数。

  • 如果字段中没有值,则默认为none

  • 警告:对于大文件/数据大小,shell 本身就很慢。

【讨论】:

    猜你喜欢
    • 1970-01-01
    • 2019-05-29
    • 2023-03-15
    • 1970-01-01
    • 2019-05-31
    • 2014-08-23
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多