Spark - Snappy 库不可用答案

【问题标题】：Spark - Snappy library not availableSpark - Snappy 库不可用
【发布时间】：2017-07-31 08:54:18
【问题描述】：

我试图保存一个 Mllib 模型。

我在 Spark 中运行的代码：

model = RandomForest.trainClassifier(train_data, 
                                 numClasses=2, categoricalFeaturesInfo=categoricalFeaturesInfo,
                                 numTrees=numTrees, featureSubsetStrategy="auto",
                                 impurity=impurity, maxDepth=maxDepth, maxBins=maxBins)

model.save(sc, "file:///path/to/models/model_name")

错误信息是：

native snappy library not available: 
this version of libhadoop was built without snappy support

Spark 的信息

Spark 版本：

1.6.1

启动 Spark 的代码：

pyspark --packages com.databricks:spark-csv_2.11:1.5.0 --master "local[8]" --driver-memory 6G --executor-memory 6G --jars /usr/local/path/to/hadoop/lib/snappy-java-1.0.4.1.jar

环境变量（spark_env.sh）：

HADOOP_HOME=/usr/local/path/to/hadoop 
SPARK_HOME=/usr/local/path/to/spark
HADOOP_CONF_DIR=/usr/local/path/to/hadoop/etc/hadoop
SPARK_CONF_DIR=/usr/local/path/to/spark/conf
HADOOP_LZO_DIR=/usr/local/path/to/hadoop/lib
HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/usr/local/path/to/spark/*:/usr/local/path/to/spark/lib/*:/usr/local/path/to/hadoop/lib/*:/usr/local/path/to/hadoop/lib/native/*
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/path/to/hadoop/lib/native
SPARK_CLASSPATH=$SPARK_CLASSPATH:/usr/local/path/to/hadoop/lib/native/*:/usr/local/path/to/hadoop/lib/snappy-java-1.0.4.1.jar:/usr.local/path/to/spark/lib/*:$CLASSPATH

我故意在 SPARK_CLASSPATH 中添加了/usr/local/path/to/hadoop/lib/snappy-java-1.0.4.1.jar 无济于事。

【问题讨论】：

标签： hadoop pyspark apache-spark-mllib snappy

【解决方案1】：

接下来的步骤帮助我解决了类似的问题：

从https://hadoop.apache.org/releases.html下载hadoop库，本地解压，设置spark.driver.extraLibraryPath为$HADOOP_PATH/lib/native。

例子：

pyspark --packages com.databricks:spark-csv_2.11:1.5.0 --master "local[8]" --conf "spark.driver.extraLibraryPath=/home/hadoop/hadoop-2.8.1/lib/native" --driver-memory 6G --executor-memory 6G --jars /usr/local/path/to/hadoop/lib/snappy-java-1.0.4.1.jar

如果不适合您，您可以尝试将spark.executor.extraLibraryPath 设置为相同的值。

【讨论】：