【问题标题】:Spark - pickling error when accessing RDD in loopSpark - 在循环中访问 RDD 时出现酸洗错误
【发布时间】:2017-10-19 03:46:54
【问题描述】:

我正在 Spark 中实现 k-means 算法。当我运行下面的代码时,我得到一个酸洗错误(如下所示)。如果我修改它并将所有内容放在循环之外,它将正确计算质心。

sc = SparkContext(appName="Document Similarity")
lines = sc.wholeTextFiles(sys.argv[1])

articles = lines.flatMap(lambda x: re.split(r' ',x[1]))

shingles = articles.flatMap(shingle_pairs.get_pairs)

sig_vecs = shingles.groupBy(lambda x: x[1]) \
                   .map(lambda x: sig_vector.create_vector(x, a, b, n, p))

centroids = k_means.init_centroids(sig_size, k)

for i in range(max_it):
    # assign documents to closest cluster
    docs = sig_vecs.map(lambda x: k_means.classify_docs(x, centroids))

    # get count by key to use in mean calculation for new clusters
    doc_count = docs.countByKey()

    # recompute cluster centroids
    reduced_docs = docs.reduceByKey(k_means.reducer)
    centroids = reduced_docs.map(lambda x: k_means.mapper(x, doc_count))

错误如下:

pickle.PicklingError: Could not serialize object: Exception:
It appears that you are attempting to broadcast an RDD or reference an 
RDD from an action or transformation. RDD transformations and actions 
can only be invoked by the driver, not inside of other transformations; 
for example, rdd1.map(lambda x: rdd2.values.count() * x) is invalid 
because the values transformation and count action cannot be performed 
inside of the rdd1.map transformation. For more information, see SPARK-5063.

【问题讨论】:

    标签: python apache-spark pyspark


    【解决方案1】:

    正如SPARK-5063“Spark 不支持嵌套 RDD”中所述。您正在尝试访问map 中的centroids (RDD) sig_vecs (RDD):

    docs = sig_vecs.map(lambda x: k_means.classify_docs(x, centroids))
    

    将 centroids 转换为本地集合 (collect?) 并调整 classify_docs 应该可以解决问题。

    【讨论】:

      猜你喜欢
      • 2021-06-24
      • 2017-12-02
      • 1970-01-01
      • 2014-09-15
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      相关资源
      最近更新 更多