如何将现有的 dynamodb 表模式导出到 json？

Question

我想将一些 dynamodb 表（仅模式）复制到我的本地环境中以进行测试。首先我试过了：

aws dynamodb describe-table --table-name Foo > FooTable.json

但很明显，输出架构与create-table 命令的输入架构不兼容：

aws dynamodb create-table --cli-input-json file://FooTable.json --endpoint=http://localhost:8000

我要避免的是使用aws dynamodb create-table --generate-cli-skeleton 生成数十个骨架并手动填充它们：/

有没有办法以对娱乐“有用”的格式获取表模式？在听说他们的服务有多“好”之后，我发现没有直接的方法可以通过 Web 图形界面或标准的 aws 命令行来实现这一点，这令人难以置信。

Answer 1

我刚刚设法使用 bchew/dynamodump 进行了完整的转储和“恢复”：

git clone git@github.com:bchew/dynamodump.git

请注意文档https://github.com/bchew/dynamodump 中的--schemaOnly 选项。命令是：

./dynamodump.py -m backup --schemaOnly --region foo-region --host localhost --srcTable '*' --port 8000 --accessKey fooKey --secretKey barKey

然后您可以使用-m restore 模式将数据或架构放回本地 dynamodb 或任何需要的地方:)

话虽如此，我仍然觉得亚马逊 dynamodb 工具链的糟糕程度令人难以置信。来吧，伙计们。

Answer 2

DynamoDB 是一个 NoSQL 数据库，除了主键和二级索引之外没有架构。您可能会在 describe-table 中注意到它仅显示键和索引，而不显示任何其他列。

我通常在某种初始化代码中编写代码来创建表和键，然后开始使用表。如果桌子已经在那里，它不会受到伤害。稍后，如果我尝试插入数据并且“列”不存在也没关系，因为 DynamoDB 会自动创建它。

如果您需要导出数据，there are tools for that 将允许您从 DynamoDB 导出（有些还允许导入）。

Answer 3

这需要aws dynamodb describe-table 输出，并将其转换为aws dynamodb create-table --cli-input-json 的输入格式：

AWS_PROFILE=xyz aws dynamodb describe-table --table-name MyTable > mytable_full.json

# Pull out just what is minimally needed for table-creation:
#
#  TableName
#  KeySchema
#  AttributeDefinitions (must only contain attributes used in keys)
#  Global/Local Secondary Indexes
#  Defaults BillingMode to PAY_PER_REQUEST
#   (any provisioning can be set up manually based on load)

jq <mytable_full.json '.Table | {TableName, KeySchema, AttributeDefinitions} + (try {LocalSecondaryIndexes: [ .LocalSecondaryIndexes[] | {IndexName, KeySchema, Projection} ]} // {}) + (try {GlobalSecondaryIndexes: [ .GlobalSecondaryIndexes[] | {IndexName, KeySchema, Projection} ]} // {}) + {BillingMode: "PAY_PER_REQUEST"}' ​>mytable.json

AWS_PROFILE=xyz aws dynamodb create-table --cli-input-json file://mytable.json

您也可以将 json 粘贴到 python 中（python dict 语法与 json 非常匹配）例如

import boto3

dynamodb = boto3.resource("dynamodb")

tabledef = {
    "TableName": "MyTable",
    "KeySchema": [
...
}

table = dynamodb.create_table(**tabledef)
print("Table status: ", table.table_status)

参考资料：

https://docs.aws.amazon.com/cli/latest/reference/dynamodb/describe-table.html https://docs.aws.amazon.com/cli/latest/reference/dynamodb/create-table.html https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_CreateTable.html https://boto3.amazonaws.com/v1/documentation/api/latest/guide/dynamodb.html#creating-a-new-table

Answer 4

这是在 Windows 上使用 C#、AWS CLI 和 Newtonsoft JSON 的版本。首先运行以下命令：-

aws dynamodb describe-table --table-name TheTable --profile SourceAWSCredsProfile > TheTable.json

拿起文件，反序列化并序列化到--cli-input-json友好类：-

TableContainer tableContainer;

string sourceFile = "TheTable.json";
string destFile = "TheTable.cli-input.json";

using (StreamReader file = File.OpenText(sourceFile))
{
    JsonSerializer serializer = new JsonSerializer();
    tableContainer = (TableContainer)serializer.Deserialize(file, typeof(TableContainer));
}

File.WriteAllText(destFile, JsonConvert.SerializeObject(tableContainer.Table, Formatting.Indented, new JsonSerializerSettings
    {
        NullValueHandling = NullValueHandling.Ignore
    }
));

现在运行这个命令来导入表定义：-

aws dynamodb create-table --cli-input-json file://TheTable.cli-input.json --profile DestinationAWSCredsProfile

TableContainer 类定义如下。缺少某些属性会清除 --cli-input-json 参数不需要的所有内容。您可以随时通过运行重新创建此类：-

aws dynamodb create-table --generate-cli-skeleton

然后使用 Visual Studio 中非常方便的 Paste Special... Paste JSON as Classes 功能将输出复制并粘贴到新的类文件中。

public class TableContainer
{
    public DynamoTableCLIJSON Table { get; set; }
}

public class DynamoTableCLIJSON
{
    public Attributedefinition[] AttributeDefinitions { get; set; }
    public string TableName { get; set; }
    public Keyschema[] KeySchema { get; set; }
    public Localsecondaryindex[] LocalSecondaryIndexes { get; set; }
    public Globalsecondaryindex[] GlobalSecondaryIndexes { get; set; }
    public string BillingMode { get; set; }
    public Provisionedthroughput ProvisionedThroughput { get; set; }
    public Streamspecification StreamSpecification { get; set; }
    public Ssespecification SSESpecification { get; set; }
    public Tag[] Tags { get; set; }
}

public class Provisionedthroughput
{
    public int ReadCapacityUnits { get; set; }
    public int WriteCapacityUnits { get; set; }
}

public class Streamspecification
{
    public bool StreamEnabled { get; set; }
    public string StreamViewType { get; set; }
}

public class Ssespecification
{
    public bool Enabled { get; set; }
    public string SSEType { get; set; }
    public string KMSMasterKeyId { get; set; }
}

public class Attributedefinition
{
    public string AttributeName { get; set; }
    public string AttributeType { get; set; }
}

public class Keyschema
{
    public string AttributeName { get; set; }
    public string KeyType { get; set; }
}

public class Localsecondaryindex
{
    public string IndexName { get; set; }
    public Keyschema1[] KeySchema { get; set; }
    public Projection Projection { get; set; }
}

public class Projection
{
    public string ProjectionType { get; set; }
    public string[] NonKeyAttributes { get; set; }
}

public class Keyschema1
{
    public string AttributeName { get; set; }
    public string KeyType { get; set; }
}

public class Globalsecondaryindex
{
    public string IndexName { get; set; }
    public Keyschema2[] KeySchema { get; set; }
    public Projection1 Projection { get; set; }
    public Provisionedthroughput1 ProvisionedThroughput { get; set; }
}

public class Projection1
{
    public string ProjectionType { get; set; }
    public string[] NonKeyAttributes { get; set; }
}

public class Provisionedthroughput1
{
    public int ReadCapacityUnits { get; set; }
    public int WriteCapacityUnits { get; set; }
}

public class Keyschema2
{
    public string AttributeName { get; set; }
    public string KeyType { get; set; }
}

public class Tag
{
    public string Key { get; set; }
    public string Value { get; set; }
}