GCP Dataflow shuffler 日志：调用 url= 失败，卷曲状态未知答案

【问题标题】：GCP Dataflow shuffler logs: Call to url= failed with curl status UNKNOWNGCP Dataflow shuffler 日志：调用 url= 失败，卷曲状态未知
【发布时间】：2020-06-27 19:53:53
【问题描述】：

我在“logName”中发现了很多与 GCP Dataflow 作业相关的奇怪错误日志：“...shuffler”和“...shuffler-startup”

数据流作业正常工作，“logName”中没有错误：“...worker”

以下是“logName”中带有 INFO 级别的完整错误行：“...shuffler-startup”

E0627 19:09:03.584041 487 curl_http_transport.cc:411] Call to url=<SUBSCRIPTION_NAME>:pull failed with curl status UNKNOWN: . curl=16

E0624 12:56:32.971646 195 curl_http_transport.cc:411] Call to url=<JOB_ID>/workItems:reportStatus failed with curl status UNKNOWN: . curl=16

E0613 09:23:08.731521 205 curl_http_transport.cc:410] Call to url=<JOB_ID>/workItems:reportStatus failed with curl status UNKNOWN: . curl=65: necessary data rewind wasn't possible

E0616 17:11:53.604693 481 curl_http_transport.cc:410] Call to url=<SUBSCRIPTION_NAME>:pull failed with curl status DEADLINE_EXCEEDED: Timed out. curl=28: Operation timed out after 120000 milliseconds with 0 bytes received

E0529 02:48:14.860606 55 curl_http_transport.cc:410] Call to url=<SUBSCRIPTION_NAME>:pull failed with curl status UNKNOWN: . curl=56: BoringSSL SSL_read: SSL_ERROR_SYSCALL, errno 104

E0627 19:51:37.833993 38041 curl_http_transport.cc:411] Call to url=<SUBSCRIPTION_NAME>:acknowledge failed with curl status UNKNOWN: . curl=16

和

以下是“logName”中具有 ERROR 级别的错误行示例：“...shuffler”

Call to url=<SUBSCRIPTION_NAME>:pull failed with curl status UNKNOWN: . curl=56: BoringSSL SSL_read: Unknown error 104, errno 104

Call to url=<SUBSCRIPTION_NAME>:pull failed with curl status UNKNOWN: . curl=65: necessary data rewind wasn't possible

Call to url=<SUBSCRIPTION_NAME>:pull failed with curl status UNKNOWN: . curl=16

【问题讨论】：

此 Dataflow 作业旨在执行什么？这些错误是否会在每次作业执行时永久发生？是否有更长的处理时间或其他可衡量的影响？

标签： google-cloud-dataflow google-cloud-pubsub google-api-cpp-client

【解决方案1】：

Apache Beam 以及大多数分布式数据处理库和引擎有时可能会触发错误。这是因为这些系统很复杂，涉及数百台机器和数千个线程（或更多！）。

如果您的管道成功，那么您不必担心。这些引擎和库旨在承受这些错误并从中恢复。

此特定错误是 Dataflow 服务的临时问题。

如果这是导致您的管道失败，请立即尝试再次运行它，因为该问题现在应该已经得到缓解。

【讨论】：