Azure Data Lake Store - 从文件读取时出错

Question

我使用返回 Stream 的 FileSystemOperationsExtensions.Open 方法，我可以从中读取。 有时，当服务从流（~150-300 Mb）中读取大文件时，服务会出现以下异常：

System.IO.IOException: The read operation failed, see inner exception. ---> System.Net.WebException: The request was aborted: The request was canceled.
at System.Net.ConnectStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.Http.HttpClientHandler.WebExceptionWrapperStream.Read(Byte[] buffer, Int32 offset, Int32 count)

"ClassName": "System.IO.IOException",
"Message": "Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host."
 at System.Net.ConnectStream.Read(Byte[] buffer, Int32 offset, Int32 size)\r\n   
 at System.Net.Http.HttpClientHandler.WebExceptionWrapperStream.Read(Byte[] buffer, Int32 offset, Int32 count)

而且它是随机发生的。 此外，我创建了一个具有 60 分钟超时时间的 DataLakeStoreFileSystemManagementClient 类对象，但这些错误发生在它之前。 它可能需要 3、10、20 或任何分钟。 当然，我可以重新读取带有偏移量的流，但这需要额外的开发时间。 也许还有另一种方法可以避免这些异常。 有人可以帮我吗？

Answer 1

我用 270M+ 大小的文件做了 3 次演示测试，它对我来说总是正确的。请尝试使用以下代码进行测试。我们也可以从data lake store get started net sdk获得更多的datalake store演示代码。

演示代码：

var applicationId = "Application Id";
                var secretKey = "secretkey";
                var tenantId = "tenant id";
                var adlsAccountName = "Account name";
                var creds = ApplicationTokenProvider.LoginSilentAsync(tenantId, applicationId, secretKey).Result;
                var adlsFileSystemClient = new DataLakeStoreFileSystemManagementClient(creds,clientTimeoutInMinutes:60);
                var srcPath = "/mytempdir/ForDemoCode.zip";
                var destPath = @"c:\tom\ForDemoCode1.zip";

                Stopwatch stopWatch = new Stopwatch();
                stopWatch.Start();
                using (var stream = adlsFileSystemClient.FileSystem.Open(adlsAccountName, srcPath))
                using (var fileStream = new FileStream(destPath, FileMode.Create))
                {
                    stream.CopyTo(fileStream);
                }
                var file = new FileInfo(destPath);
                Console.WriteLine($"File size :{file.Length}");
                stopWatch.Stop();
                // Get the elapsed time as a TimeSpan value.
                TimeSpan ts = stopWatch.Elapsed;
                // Format and display the TimeSpan value.
                string elapsedTime = $"{ts.Hours:00}:{ts.Minutes:00}:{ts.Seconds:00}.{ts.Milliseconds/10:00}";
                Console.WriteLine("RunTime " + elapsedTime);
                Console.ReadKey();

打包配置文件：

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Microsoft.Azure.Management.DataLake.Store" version="2.1.1-preview" targetFramework="net452" />
  <package id="Microsoft.Azure.Management.DataLake.StoreUploader" version="1.0.0-preview" targetFramework="net452" />
  <package id="Microsoft.IdentityModel.Clients.ActiveDirectory" version="3.13.8" targetFramework="net452" />
  <package id="Microsoft.Rest.ClientRuntime" version="2.3.5" targetFramework="net452" />
  <package id="Microsoft.Rest.ClientRuntime.Azure" version="3.3.5" targetFramework="net452" />
  <package id="Microsoft.Rest.ClientRuntime.Azure.Authentication" version="2.2.0-preview" targetFramework="net452" />
  <package id="Newtonsoft.Json" version="9.0.2-beta1" targetFramework="net452" />
</packages>