如何在使用 SparkConf 连接到远程 Cassandra 集群时通过“需要身份验证”？答案

【问题标题】：How to get pass "requires authentication" while connecting to remote Cassandra cluster using SparkConf?如何在使用 SparkConf 连接到远程 Cassandra 集群时通过“需要身份验证”？
【发布时间】：2015-03-28 05:38:22
【问题描述】：

我正在尝试使用 apache spark 和 cassandra 进行数据分析。所以我写了一个java代码来访问在远程机器上运行的cassandra。我使用了以下java代码。

public class JavaDemo implements Serializable {
private transient SparkConf conf;

private JavaDemo(SparkConf conf) {
    this.conf = conf;
}

private void run() {
    JavaSparkContext sc = new JavaSparkContext(conf);
    generateData(sc);
    compute(sc);
    showResults(sc);
    sc.stop();
}

private void generateData(JavaSparkContext sc) {
    CassandraConnector connector = CassandraConnector.apply(sc.getConf());
    Session session = connector.openSession();

    // Prepare the schema

        session.execute("DROP KEYSPACE IF EXISTS java_api");
        session.execute("CREATE KEYSPACE java_api WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1}");
        session.execute("CREATE TABLE java_api.products (id INT PRIMARY KEY, name TEXT, parents LIST<INT>)");
        session.execute("CREATE TABLE java_api.sales (id UUID PRIMARY KEY, product INT, price DECIMAL)");
        session.execute("CREATE TABLE java_api.summaries (product INT PRIMARY KEY, summary DECIMAL)");

}

private void compute(JavaSparkContext sc) {
    System.out.println("IN compute");
}

private void showResults(JavaSparkContext sc) {
    System.out.println("IN showResults");
}

public static void main(String[] args) {


    SparkConf conf = new SparkConf();
    conf.setAppName("Java API demo");
    conf.setMaster("local[1]");
    System.out.println("---------------------------------");
    conf.set("spark.cassandra.connection.host", "192.168.1.219");


    JavaDemo app = new JavaDemo(conf);
    app.run();
}

}

192.168.1.219 是我运行 cassandra 的远程主机。默认端口是 9160。当我运行程序时，我收到以下错误。

    15/01/29 10:14:26 INFO ui.SparkUI: Started Spark Web UI at http://Justin:4040
15/01/29 10:14:27 WARN core.FrameCompressor: Cannot find LZ4 class, you should make sure the LZ4 library is in the classpath if you intend to use it. LZ4 compression will not be available for the protocol.
Exception in thread "main" com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /192.168.1.219:9042: Host /192.168.1.219:9042 requires authentication, but no authenticator found in Cluster configuration
    at com.datastax.driver.core.AuthProvider$1.newAuthenticator(AuthProvider.java:38)
    at com.datastax.driver.core.Connection.initializeTransport(Connection.java:139)
    at com.datastax.driver.core.Connection.<init>(Connection.java:111)
    at com.datastax.driver.core.Connection$Factory.open(Connection.java:445)
    at com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:216)
    at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:172)
    at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:80)
    at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1145)
    at com.datastax.driver.core.Cluster.getMetadata(Cluster.java:313)
    at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:166)
    at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$4.apply(CassandraConnector.scala:151)
    at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$4.apply(CassandraConnector.scala:151)
    at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:36)
    at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:61)
    at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:72)
    at com.datastax.spark.demo.JavaDemo.generateData(JavaDemo.java:42)
    at com.datastax.spark.demo.JavaDemo.run(JavaDemo.java:34)
    at com.datastax.spark.demo.JavaDemo.main(JavaDemo.java:73)

我有什么遗漏的吗？它直接连接到端口 9042。我该如何连接？

【问题讨论】：

关于主机 /192.168.1.219:9042 上的身份验证错误：主机 /192.168.1.219:9042 需要身份验证，但在集群配置中找不到身份验证器有哪些不清楚的地方？

标签： java cassandra apache-spark datastax

【解决方案1】：

您的 cassandra 集群上似乎配置了身份验证。由于您没有提供凭据，因此不允许您连接。您可以使用spark.cassandra.auth.username 和spark.cassandra.auth.password 属性传递身份验证凭据，如here 所述。

所以你可以这样做：

conf.set("spark.cassandra.auth.username", "cassandra");            
conf.set("spark.cassandra.auth.password", "cassandra");

在您的代码中完成这项工作。

如果您启用了authentication 并且您尚未创建/更改任何用户，您可以使用“cassandra”作为用户名和密码。在生产环境中，您应该创建一个单独的帐户并改用该帐户，并更改 cassandra 用户密码，因为它可以访问所有内容。

【讨论】：