如何在 Scala 中将长 SQL 查询分解为多行？答案

【问题标题】：How do you break up long SQL query into multiple lines in Scala?如何在 Scala 中将长 SQL 查询分解为多行？
【发布时间】：2016-11-09 03:22:09
【问题描述】：

鉴于以下情况：

val sqlTransformer_labeled = new SQLTransformer().setStatement("""SELECT *, CAST(var1 AS DOUBLE) AS label, var2 - var3 - var4 AS newvar1, var5 - var6 AS new var2""")

如何将一个长 SQL 查询分成多行？为了便于阅读，我想要以下内容：

val sqlTransformer_labeled = new SQLTransformer().setStatement("""SELECT *, 
    CAST(var1 AS DOUBLE) AS label, 
    var2 - var3 - var4 AS newvar1, 
    var5 - var6 AS new var2""")

可读的代码将在 6 个月后对我自己、合作者以及我们的迭代建模工作流程都有帮助。

但是，尝试运行上面代码的后一个更清晰的版本时，会给出奇怪的反馈，如下所示：

scala> val sqlTransformer_labeled = new SQLTransformer().setStatement("""SELECT *,
     |
<init>                           DRIVER_IDENTIFIER                LEGACY_DRIVER_IDENTIFIER         Pipeline                         PipelineModel                    RDD_SCOPE_KEY                    RDD_SCOPE_NO_OVERRIDE_KEY
RandomForestClassifier           SPARK_JOB_DESCRIPTION            SPARK_JOB_GROUP_ID               SPARK_JOB_INTERRUPT_ON_CANCEL    SQLTransformer                   StringIndexer                    StringToColumn
VectorAssembler                  _sqlContext                      abs                              acos                             add_months                       approxCountDistinct              array
array_contains                   asc                              ascii                            asin                             atan                             atan2                            avg
base64                           bin                              bitwiseNOT                       boolToBoolWritable               booleanWritableConverter         broadcast                        bytesToBytesWritable
bytesWritableConverter           callUDF                          callUdf                          cbrt                             ceil                             classOf                          clearActiveContext
clone                            coalesce                         col                              collect_list                     collect_set                      column                           concat
concat_ws                        conv                             corr                             cos                              cosh                             count                            countDistinct
crc32                            cumeDist                         cume_dist                        current_date                     current_timestamp                date_add                         date_format
date_sub                         datediff                         dayofmonth                       dayofyear                        decode                           denseRank                        dense_rank
desc                             doubleRDDToDoubleRDDFunctions    doubleToDoubleWritable           doubleWritableConverter          encode                           eq                               equals
exp                              explode                          expm1                            expr                             factorial                        finalize                         first
floatToFloatWritable             floatWritableConverter           floor                            format_number                    format_string                    from_unixtime                    from_utc_timestamp
getClass                         getOrCreate                      get_json_object                  greatest                         hashCode                         hex                              hh_features
hh_split                         hh_timeline_pair_luid            hour                             hypot                            initcap                          inputFileName                    input_file_name
instr                            intRddToDataFrameHolder          intToIntWritable                 intWritableConverter             isNaN                            isTraceEnabled                   isnan
isnull                           jarOfClass                       jarOfObject                      json_tuple                       kurtosis                         lag                              last
last_day                         lead                             least                            length                           levenshtein                      lit                              localSeqToDataFrameHolder
localSeqToDatasetHolder          locate                           log                              log10                            log1p                            log2                             logDebug
logError                         logInfo                          logName                          logTrace                         logWarning                       longRddToDataFrameHolder         longToLongWritable
longWritableConverter            lower                            lpad                             ltrim                            markPartiallyConstructed         max                              md5
mean                             min                              minute                           model                            modeling_stack                   monotonicallyIncreasingId        monotonically_increasing_id
month                            months_between                   nanvl                            ne                               negate                           newBooleanEncoder                newByteEncoder
newDoubleEncoder                 newFloatEncoder                  newIntEncoder                    newLongEncoder                   newProductEncoder                newShortEncoder                  newStringEncoder
next_day                         not                              notify                           notifyAll                        ntile                            numDriverCores                   numericRDDToDoubleRDDFunctions
percentRank                      percent_rank                     pipeline                         pmod                             pow                              prediction                       quarter
rand                             randn                            rank                             rddToAsyncRDDActions             rddToDataFrameHolder             rddToDatasetHolder               rddToOrderedRDDFunctions
rddToPairRDDFunctions            rddToSequenceFileRDDFunctions    regexp_extract                   regexp_replace                   repeat                           reverse                          rf
rint                             round                            rowNumber                        row_number                       rpad                             rtrim                            sc
second                           setActiveContext                 sha1                             sha2                             shiftLeft                        shiftRight                       shiftRightUnsigned
signum                           sin                              sinh                             size                             skewness                         sort_array                       soundex
sparkPartitionId                 spark_partition_id               split                            sql                              sqlContext                       sqlTransformer                   sqlTransformer_labeled
sqrt                             stddev                           stddev_pop                       stddev_samp                      stringIndexer                    stringRddToDataFrameHolder       stringToText
stringWritableConverter          struct                           substring                        substring_index                  sum                              sumDistinct                      symbolToColumn
synchronized                     tan                              tanh                             toDegrees                        toRadians                        toString                         to_date
to_utc_timestamp                 translate                        trim                             trunc                            tstack                           udf                              unbase64
unhex                            unix_timestamp                   updatedConf                      upper                            var_pop                          var_samp                         variance
wait                             weekofyear                       when                             writableWritableConverter        year

     | cast(same_variable as double) as label,
     |
<init>                           DRIVER_IDENTIFIER                LEGACY_DRIVER_IDENTIFIER         Pipeline                         PipelineModel                    RDD_SCOPE_KEY                    RDD_SCOPE_NO_OVERRIDE_KEY
RandomForestClassifier           SPARK_JOB_DESCRIPTION            SPARK_JOB_GROUP_ID               SPARK_JOB_INTERRUPT_ON_CANCEL    SQLTransformer                   StringIndexer                    StringToColumn
VectorAssembler                  _sqlContext                      abs                              acos                             add_months                       approxCountDistinct              array
array_contains                   asc                              ascii                            asin                             atan                             atan2                            avg
base64                           bin                              bitwiseNOT                       boolToBoolWritable               booleanWritableConverter         broadcast                        bytesToBytesWritable
bytesWritableConverter           callUDF                          callUdf                          cbrt                             ceil                             classOf                          clearActiveContext
clone                            coalesce                         col                              collect_list                     collect_set                      column                           concat
concat_ws                        conv                             corr                             cos                              cosh                             count                            countDistinct
crc32                            cumeDist                         cume_dist                        current_date                     current_timestamp                date_add                         date_format
date_sub                         datediff                         dayofmonth                       dayofyear                        decode                           denseRank                        dense_rank
desc                             doubleRDDToDoubleRDDFunctions    doubleToDoubleWritable           doubleWritableConverter          encode                           eq                               equals
exp                              explode                          expm1                            expr                             factorial                        finalize                         first
floatToFloatWritable             floatWritableConverter           floor                            format_number                    format_string                    from_unixtime                    from_utc_timestamp
getClass                         getOrCreate                      get_json_object                  greatest                         hashCode                         hex                              hh_features
hh_split                         hh_timeline_pair_luid            hour                             hypot                            initcap                          inputFileName                    input_file_name
instr                            intRddToDataFrameHolder          intToIntWritable                 intWritableConverter             isNaN                            isTraceEnabled                   isnan
isnull                           jarOfClass                       jarOfObject                      json_tuple                       kurtosis                         lag                              last
last_day                         lead                             least                            length                           levenshtein                      lit                              localSeqToDataFrameHolder
localSeqToDatasetHolder          locate                           log                              log10                            log1p                            log2                             logDebug
logError                         logInfo                          logName                          logTrace                         logWarning                       longRddToDataFrameHolder         longToLongWritable
longWritableConverter            lower                            lpad                             ltrim                            markPartiallyConstructed         max                              md5
mean                             min                              minute                           model                            modeling_stack                   monotonicallyIncreasingId        monotonically_increasing_id
month                            months_between                   nanvl                            ne                               negate                           newBooleanEncoder                newByteEncoder
newDoubleEncoder                 newFloatEncoder                  newIntEncoder                    newLongEncoder                   newProductEncoder                newShortEncoder                  newStringEncoder
next_day                         not                              notify                           notifyAll                        ntile                            numDriverCores                   numericRDDToDoubleRDDFunctions
percentRank                      percent_rank                     pipeline                         pmod                             pow                              prediction                       quarter
rand                             randn                            rank                             rddToAsyncRDDActions             rddToDataFrameHolder             rddToDatasetHolder               rddToOrderedRDDFunctions
rddToPairRDDFunctions            rddToSequenceFileRDDFunctions    regexp_extract                   regexp_replace                   repeat                           reverse                          rf
rint                             round                            rowNumber                        row_number                       rpad                             rtrim                            sc
second                           setActiveContext                 sha1                             sha2                             shiftLeft                        shiftRight                       shiftRightUnsigned
signum                           sin                              sinh                             size                             skewness                         sort_array                       soundex
sparkPartitionId                 spark_partition_id               split                            sql                              sqlContext                       sqlTransformer                   sqlTransformer_labeled
sqrt                             stddev                           stddev_pop                       stddev_samp                      stringIndexer                    stringRddToDataFrameHolder       stringToText
stringWritableConverter          struct                           substring                        substring_index                  sum                              sumDistinct                      symbolToColumn
synchronized                     tan                              tanh                             toDegrees                        toRadians                        toString                         to_date
to_utc_timestamp                 translate                        trim                             trunc                            tstack                           udf                              unbase64
unhex                            unix_timestamp                   updatedConf                      upper                            var_pop                          var_samp                         variance
wait                             weekofyear                       when                             writableWritableConverter        year

等等

【问题讨论】：

它应该可以工作。你得到什么错误？
@talex 更新以反映结果的开头
尝试在控制台中使用:pa 粘贴模式和Ctrl + D 退出。您会看到所有这些输出，因为它向您展示了选项卡完成的可能性。
@Eric 没用。结果与上述相同。
@user2205916 尝试用空格替换你的制表符。

标签： scala

【解决方案1】：

这里的问题是您的 SQL 脚本包含选项卡，这会触发 REPL 中的选项卡完成。从脚本中删除选项卡，它应该可以工作。

或者，如果您不需要制表符完成，您可以尝试使用this answer 完全禁用它。

【讨论】：

【解决方案2】：

scala> """very long
     |   |sql query""".stripMargin
res2: String =
very long
sql query

【讨论】：

.stripMargin 这样做，但不允许我按照我在原始帖子中描述的方式编写代码