C# REST API GET 方法：无法处理大量记录答案

【问题标题】：C# REST API GET method: Unable to handle large amount of recordsC# REST API GET 方法：无法处理大量记录
【发布时间】：2019-11-24 04:27:10
【问题描述】：

我正在使用 VS 2017 重建 C# 应用程序。但是我无法检索超过 3993 条记录，我需要获得 ~180K 才能将它们插入到平面表（SQL db）中

我认为分页方法设计不正确，我不太确定如何修复它，如果我添加大于 4K 的数字，它会崩溃。解决此问题的最佳方法是什么？

这是我们目前所拥有的：

Helper.cs：

{
    ItemsDto itemsCollection = null;
    HttpClient httpClient = new HttpClient();


    httpClient.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", token);


    var uri = string.Format(@"{0}pageSize={1}&page=1", Constants.REQUEST_ENDPOINT, Constants.MAXVALUE);//Constants.MAXVALUE); 
    uri = uri + "&advancedFilter=__Status__~eq~'Active'";
    uri = uri + @"&dynamicColumns=Name,ID,Date,Status,Version,Name,Division";

    uri = uri + "&orderDirection=DESC";

    HttpResponseMessage response = await httpClient.GetAsync(uri);


    if (response == null)
    {
        throw new Exception("There is no returned request from API");  

    }

    if (response.IsSuccessStatusCode)
    {
        var responseAsString = await response.Content.ReadAsStringAsync();
        itemsCollection = JsonConvert.DeserializeObject<ItemsDto>(responseAsString) ?? new ItemsDto();
    }
    else
    {
        throw new Exception("An error request has ocurred: " + response.RequestMessage);  
    }

    if (itemsCollection.Items == null)
    {
        log.Info("There is no files available");
        itemsCollection.Items = new List<Document>();
    }

    return itemsCollection.Items;

}

App.config：

  <add key="IntMaxValue" value="4000" /> //Change pageSize Here

Constants.cs：

public static readonly int MAXVALUE = int.Parse(ConfigurationManager.AppSettings["IntMaxValue"]);

【问题讨论】：

也许你没有足够的内存来一次处理那么多？
当它崩溃时它给出了什么错误？
谢谢@Steven B，这可能是超时问题，我会稍微调查一下，看看如何配置它。
@Mikael 错误是“任务被取消。内部异常”我将尝试逐行调试代码，看看是否能看到其他内容。
@tana 检查关于该错误的答案here。应该是超时了

标签： c# rest api get

【解决方案1】：

最终的解决方案是使用“for”来正确使用 api 分页。在这种情况下，如果我想检索 150K 记录，则 page=10 和 pageSize=15000（每页 15k）

for(int pagina=1; pagina<=10; pagina++)
            {
                HttpClient httpClient = new HttpClient();
                httpClient.Timeout = TimeSpan.FromMinutes(30);
                httpClient.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", token);

                var uri = string.Format(@"{0}pageSize={1}&page={2}", Constants.REQUEST_ENDPOINT, 15000,pagina);//Constants.MAXVALUE); 
uri = uri + "&advancedFilter=__Status__~eq~'Active'";
uri = uri + @"&dynamicColumns=Name,ID,Date,Status,Version,Name,Division";

                uri = uri + "&orderDirection=DESC";

                HttpResponseMessage response = await httpClient.GetAsync(uri);


                if (response == null)
                {
                    throw new Exception("There is no returned request from TAP API");

                }

                if (response.IsSuccessStatusCode)
                {
                    var responseAsString = await response.Content.ReadAsStringAsync();
                    itemsCollection = JsonConvert.DeserializeObject<ItemsDto>(responseAsString) ?? new ItemsDto();
                    itemsCollectionTotal.Add(itemsCollection);
                }
                else
                {
                    throw new Exception("An error request has ocurred: " + response.RequestMessage);
                }

                if (itemsCollection.Items == null)
                {
                    log.Info("There is no documents available from API");
                    itemsCollection.Items = new List<Document>();
                }

            }

            return itemsCollectionTotal;//itemsCollectionTotal.Item;

        }

【讨论】：