【问题标题】：How retrieve all of records in Azure storage tables using PowerShell or Azure CLI如何使用 PowerShell 或 Azure CLI 检索 Azure 存储表中的所有记录
【发布时间】：2018-09-15 18:39:03
【问题描述】：

我正在使用 Azure CLI，我可以连接到存储表并使用以下代码在我的存储中的每个表的循环中从表中获取项目：

"get metrics from $table"
$temp_result = (az storage entity query --table-name $table --connection-string $connection_string --accept "none" --filter "CounterName eq '\Memory\PercentUsedMemory' or CounterName eq '\NetworkInterface\BytesReceived' or CounterName eq '\NetworkInterface\BytesTransmitted' or CounterName eq '\Processor\PercentProcessorTime'" --select "TIMESTAMP" "Average" "CounterName" | ConvertFrom-Json)
($temp_result.items).Length

结果是这样的：

get metrics from WADMetricsPT1HP10DV2S20171007
88
get metrics from WADMetricsPT1HP10DV2S20171017
688
get metrics from WADMetricsPT1HP10DV2S20171027
464
get metrics from WADMetricsPT1HP10DV2S20180326
88
get metrics from WADMetricsPT1HP10DV2S20180405
72
get metrics from WADMetricsPT1MP10DV2S20171007
1000
get metrics from WADMetricsPT1MP10DV2S20171017
1000
get metrics from WADMetricsPT1MP10DV2S20171027
1000
get metrics from WADMetricsPT1MP10DV2S20180326
1000
get metrics from WADMetricsPT1MP10DV2S20180405
1000

但在我的一张表中，我有超过 37000 条记录，我想获取所有这些记录。我试着像下面这样设置--num-results 9999：

"get metrics from $table"
$temp_result = (az storage entity query --table-name $table --connection-string $connection_string --accept "none" --filter "CounterName eq '\Memory\PercentUsedMemory' or CounterName eq '\NetworkInterface\BytesReceived' or CounterName eq '\NetworkInterface\BytesTransmitted' or CounterName eq '\Processor\PercentProcessorTime'" --select "TIMESTAMP" "Average" "CounterName" --num-results 9999 | ConvertFrom-Json)
($temp_result.items).Length

但它会返回此错误：

{"odata.error":{"code":"InvalidInput","message":{"lang":"en-US","value":"One of the request inputs is not valid.\nRequestId:d8ccda32-f002-0034-2619-cd42aa000000\nTime:2018-04-05T20:07:56.7891454Z"}}}

您能否指导我出了什么问题以及如何使用 Powershell 或 Azure CLI 获取 Azure 存储表中的所有记录？

更新：

正如卢兆兴在this 回答中所说，我尝试使用--marker。但是有一些问题。

首先，这是我当前的代码：

$connection_string = (az storage account show-connection-string --resource-group $resourceGroup --name $storageAccount | ConvertFrom-Json).connectionString
$table_list = (az storage table list --connection-string $connection_string | ConvertFrom-Json).name

if ($table_list.Length -eq 0)
{
    "There is no table in '$storageAccount' storage"
}
else
{
    "There is "+$table_list.Length+" tables stored in $storageAccount."
    ###########################################
    ## Find those tables that stored metrics ##
    ###########################################
    $tables = @()
    Foreach($temp_table_name in $table_list)
    {
        if ($temp_table_name.StartsWith("WADMetrics"))
        {
            $tables += $temp_table_name
        }
    }

    if ($tables.Length -eq 0)
    {
        "There is no table starting with 'WADMetrics' as prefix in storage ($storageAccount) tables, then we can't detect any Metric."
    }
    else
    {
        ($tables.Length).tostring()+" tables stored metrics"
        Foreach($table in $tables)
        {
            "get metrics from $table"
            $is_more_results = 0
            $nextMarker = @{}
            $nextpartitionkey = ""
            $nextrowkey = ""

            Do
            {
                $filter_str = "CounterName eq '\Memory\PercentUsedMemory' or CounterName eq '\NetworkInterface\BytesReceived' or CounterName eq '\NetworkInterface\BytesTransmitted' or CounterName eq '\Processor\PercentProcessorTime'"
                $temp_result

                if ($nextpartitionkey -ne "")
                {
                    "Call with marker"
                    $temp_result = az storage entity query --table-name $table --connection-string $connection_string --accept "minimal" --filter "CounterName eq '\Memory\PercentUsedMemory' or CounterName eq '\NetworkInterface\BytesReceived' or CounterName eq '\NetworkInterface\BytesTransmitted' or CounterName eq '\Processor\PercentProcessorTime'" --select "TIMESTAMP" "Average" "CounterName" --num-results 50 --marker $nextMarker
                }
                else
                {
                    "Call without marker"
                    $temp_result = az storage entity query --table-name $table --connection-string $connection_string --accept "minimal" --filter "CounterName eq '\Memory\PercentUsedMemory' or CounterName eq '\NetworkInterface\BytesReceived' or CounterName eq '\NetworkInterface\BytesTransmitted' or CounterName eq '\Processor\PercentProcessorTime'" --select "TIMESTAMP" "Average" "CounterName" --num-results 50
                }

                $temp_result = [string]$temp_result
                $temp_result = $temp_result | ConvertFrom-Json
                ($temp_result.items).Length

                if (($temp_result.nextMarker.nextpartitionkey).Length -gt 0)
                {
                    #there is more items in the requested query
                    $nextpartitionkey = $temp_result.nextMarker.nextpartitionkey
                    $nextrowkey = $temp_result.nextMarker.nextrowkey
                    $nextMarker["nextpartitionkey"] = $nextpartitionkey
                    $nextMarker["nextrowkey"] = $nextrowkey
                    $is_more_results = 1
                }
                else
                {
                    $is_more_results = 0
                }
            } While ($is_more_results -ne 0)
        }
    }
}

以上代码将返回如下输出：

There is 16 tables stored in testresource********.
10 tables stored metrics
get metrics from WADMetricsPT1HP10DV2S20171007
Call without marker
50

items                                                                                                                                                        
-----                                                                                                                                                        
{@{Average=10.095833333333333; CounterName=\Memory\PercentUsedMemory; TIMESTAMP=2017-10-16T22:00:00+00:00; etag=W/"datetime'2017-10-16T23%3A00%3A05.500086...
Call with marker
az : ERROR: 'str' object has no attribute 'get'
At line:66 char:36
+ ... mp_result = az storage entity query --table-name $table --connection- ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (ERROR: 'str' ob...attribute 'get':String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

Traceback (most recent call last):
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\knack\cli.py", line 197, in invoke
    cmd_result = self.invocation.execute(args)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\commands\__init__.py", line 347, in execute
    six.reraise(*sys.exc_info())
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\six.py", line 693, in reraise
    raise value
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\commands\__init__.py", line 319, in execute
    result = cmd(params)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\commands\__init__.py", line 180, in __call__
    return super(AzCliCommand, self).__call__(*args, **kwargs)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\knack\commands.py", line 109, in __call__
    return self.handler(*args, **kwargs)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\__init__.py", line 420, in default_command_handler
    result = op(**command_args)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\multiapi\cosmosdb\v2017_04_17\table\tableservice.py", line 730, in 
query_entities
    resp = self._query_entities(*args, **kwargs)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\multiapi\cosmosdb\v2017_04_17\table\tableservice.py", line 776, in 
_query_entities
    next_partition_key = None if marker is None else marker.get('nextpartitionkey')
AttributeError: 'str' object has no attribute 'get'
ConvertFrom-Json : Cannot bind argument to parameter 'InputObject' because it is null.
At line:75 char:47
+                 $temp_result = $temp_result | ConvertFrom-Json
+                                               ~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidData: (:) [ConvertFrom-Json], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.ConvertFromJsonCommand

0

我尝试使用以下代码直接传递--marker：

if (($temp_result.nextMarker.nextpartitionkey).Length -gt 0)
{
    #there is more items in the requested query
    $nextpartitionkey = $temp_result.nextMarker.nextpartitionkey

    $nextMarker = $temp_result.nextMarker
    $is_more_results = 1
}

但错误是一样的：

There is 16 tables stored in testresource********.
10 tables stored metrics
get metrics from WADMetricsPT1HP10DV2S20171007
Call without marker
50

items                                                                                                                                                        
-----                                                                                                                                                        
{@{Average=10.095833333333333; CounterName=\Memory\PercentUsedMemory; TIMESTAMP=2017-10-16T22:00:00+00:00; etag=W/"datetime'2017-10-16T23%3A00%3A05.500086...
Call with marker
az : ERROR: 'str' object has no attribute 'get'
At line:66 char:36
+ ... mp_result = az storage entity query --table-name $table --connection- ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (ERROR: 'str' ob...attribute 'get':String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

Traceback (most recent call last):
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\knack\cli.py", line 197, in invoke
    cmd_result = self.invocation.execute(args)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\commands\__init__.py", line 347, in execute
    six.reraise(*sys.exc_info())
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\six.py", line 693, in reraise
    raise value
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\commands\__init__.py", line 319, in execute
    result = cmd(params)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\commands\__init__.py", line 180, in __call__
    return super(AzCliCommand, self).__call__(*args, **kwargs)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\knack\commands.py", line 109, in __call__
    return self.handler(*args, **kwargs)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\__init__.py", line 420, in default_command_handler
    result = op(**command_args)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\multiapi\cosmosdb\v2017_04_17\table\tableservice.py", line 730, in 
query_entities
    resp = self._query_entities(*args, **kwargs)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\multiapi\cosmosdb\v2017_04_17\table\tableservice.py", line 776, in 
_query_entities
    next_partition_key = None if marker is None else marker.get('nextpartitionkey')
AttributeError: 'str' object has no attribute 'get'
ConvertFrom-Json : Cannot bind argument to parameter 'InputObject' because it is null.
At line:75 char:47
+                 $temp_result = $temp_result | ConvertFrom-Json
+                                               ~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidData: (:) [ConvertFrom-Json], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.ConvertFromJsonCommand

0

我什至尝试删除 --marker 并通过查询传递 nextpartitionkey 和 nextrowkey！，它没有返回任何错误，但它会始终返回相同的并且循环将永远继续。

if ($nextpartitionkey -ne "")
{
    $filter_str = $filter_str + " and PartitionKey eq '$nextpartitionkey' and RowKey eq '$nextrowkey'"
}
$temp_result = az storage entity query --table-name $table --connection-string $connection_string --accept "minimal" --filter $filter_str --select "TIMESTAMP" "Average" "CounterName" --num-results 50

更新 2：

我尝试将标记作为搅拌。波纹管字符串是我用作字符串的标记示例：

"nextMarker": {
    "nextpartitionkey":  "1!260!OjAwMkZzdWJzY3JpcHRpb25zOjAwMkYzMTM1Nzc1YTowMDJEMTA2ZDowMDJENGVmODowMDJEODlhYTowMDJEN2VkYjM2YjRjMzU2OjAwMkZyZXNvdXJjZUdyb3Vwcz
owMDJGVGVzdFJlc291cmNlR3JvdXAyOjAwMkZwcm92aWRlcnM6MDAyRk1pY3Jvc29mdDowMDJFQ29tcHV0ZTowMDJGdmlydHVhbE1hY2hpbmVzOjAwMkZjcHV1c2FnZXRlc3Q-",
    "nextrowkey":  "1!72!OjAwNUNNZW1vcnk6MDA1Q1BlcmNlbnRVc2VkTWVtb3J5X18yNTE4OTQxMzExOTk5OTk5OTk5"
}

我也试过这个作为字符串：

{
    "nextpartitionkey":  "1!260!OjAwMkZzdWJzY3JpcHRpb25zOjAwMkYzMTM1Nzc1YTowMDJEMTA2ZDowMDJENGVmODowMDJEODlhYTowMDJEN2VkYjM2YjRjMzU2OjAwMkZyZXNvdXJjZUdyb3Vwcz
owMDJGVGVzdFJlc291cmNlR3JvdXAyOjAwMkZwcm92aWRlcnM6MDAyRk1pY3Jvc29mdDowMDJFQ29tcHV0ZTowMDJGdmlydHVhbE1hY2hpbmVzOjAwMkZjcHV1c2FnZXRlc3Q-",
    "nextrowkey":  "1!72!OjAwNUNNZW1vcnk6MDA1Q1BlcmNlbnRVc2VkTWVtb3J5X18yNTE4OTQxMzExOTk5OTk5OTk5"
}

在这两个中我都得到了这个错误：

az : ERROR: 'str' object has no attribute 'get'
At C:\Users\Reza\Desktop\ndbench\Azure\Automation\01_get_metrics\add_target_to_tables - runbook.ps1:87 char:36
+ ... mp_result = az storage entity query --table-name $table --connection- ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (ERROR: 'str' ob...attribute 'get':String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

Traceback (most recent call last):
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\knack\cli.py", line 197, in invoke
    cmd_result = self.invocation.execute(args)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\commands\__init__.py", line 347, in execute
    six.reraise(*sys.exc_info())
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\six.py", line 693, in reraise
    raise value
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\commands\__init__.py", line 319, in execute
    result = cmd(params)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\commands\__init__.py", line 180, in __call__
    return super(AzCliCommand, self).__call__(*args, **kwargs)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\knack\commands.py", line 109, in __call__
    return self.handler(*args, **kwargs)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\__init__.py", line 420, in default_command_handler
    result = op(**command_args)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\multiapi\cosmosdb\v2017_04_17\table\tableservice.py", line 730, in 
query_entities
    resp = self._query_entities(*args, **kwargs)
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\multiapi\cosmosdb\v2017_04_17\table\tableservice.py", line 776, in 
_query_entities
    next_partition_key = None if marker is None else marker.get('nextpartitionkey')
AttributeError: 'str' object has no attribute 'get'
ConvertFrom-Json : Cannot bind argument to parameter 'InputObject' because it is null.
At C:\Users\Reza\Desktop\ndbench\Azure\Automation\01_get_metrics\add_target_to_tables - runbook.ps1:112 char:47
+                 $temp_result = $temp_result | ConvertFrom-Json
+                                               ~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidData: (:) [ConvertFrom-Json], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.ConvertFromJsonCommand

那么在夏天，问题是我应该如何通过--marker？

【问题讨论】：

标签： azure azure-storage azure-powershell azure-cli

【解决方案1】：

您收到错误是因为一次无法返回 9999 个结果。正如Query Entities 文档所示，-num-results 的最大值为 1000。

为了获取表中的所有实体，您需要通过延续对象循环 az storage entity query。要获取下一页的查询结果，请将--marker设置为上一次迭代中$temp_result.next_marker的值。

【讨论】：

我已经编辑了这个问题。你能阅读更新部分吗？看来你是对的，但还有更多问题
您能否尝试将 $nextMarker 转换回字符串，然后再将其传递给 Azure CLI？
我做到了，我把结果写在“UPDATE 2”部分。
@Zhaixing Lu 你能看看更新2部分吗？

【解决方案2】：

这是 CLI 中的一个错误。我们将在下一个版本 v2.0.32 中进行修复。基本上没有解析出--marker 参数的属性。

阅读此处了解更多信息：https://github.com/Azure/azure-cli/issues/6194

【讨论】：