将left_joint与sparklyr一起使用时删除的表答案

【问题标题】：deleted tables when using left_joint with sparklyr将left_joint与sparklyr一起使用时删除的表
【发布时间】：2017-10-09 06:45:14
【问题描述】：

我正在处理一些我想加入的表，因为我使用 sparklyr（由于表大小）和 dplyr 的 left_joint。这是代码示例：

query.1 <- left_join(pa11, pa12, by = c("CODIGO_HAB_D","ID_EST","ID_ME","ID_PARTE_D","ID_PAR", "ID_REP")) %>% left_join(., pa13, by = c("ID_SINI" = "ID_SINI"))

query.1 <- left_join(query.1, a14, by = "ID_REP" )
query.1 <-left_join(query.1, a16, by = c("ID_MEJ" = "ID_ME"))
query.1 <-left_join(query.1, a17, by = c("ID_EST"  = "ID_ESTE"))
query.1 <-left_join(query.1, a18, by = "ID_PARTE_D" )
query.1 <-left_join(query.1, a19, by = "CODI" )
query.1 <-left_join(query.1, a110, by = c("ID_PROF.x" = "ID_PROF" ))
query.1 <-left_join(query.1, a111, by = c("ID_COM.x" = "ID_COM" ))
query.1 <-left_join(query.1, a113, by = c("ID_GRANDES.x" = "ID_GRANDES"))

当我 left_join 前 5 个表时，一切都按预期进行。当我对更多表重复此操作时，我收到此错误

Error in as.vector(x, "character") : 
cannot coerce type 'environment' to vector of type 'character'

然后，当我尝试查看 Spark 表时，我在 Rstudio 中遇到错误。

【问题讨论】：

标签： r dplyr rstudio sparklyr

【解决方案1】：

由于其他一些原因，我有时会收到这些错误。

根据我的经验，增加 Sparklyr 内存和执行程序开销内存会有所帮助

    config <- spark_config()
    config$`sparklyr.shell.driver-memory` <- "8G"
    config$`sparklyr.shell.executor-memory` <- "8G"
    config$spark.yarn.executor.memoryOverhead <- "2g"

【讨论】：