Spark上下文广播变量抛出java.io.NotSerializableException，即使它的Serializable

Question

我在 Ubuntu 14.04 上运行 Spark 2.1 我正在尝试在 spark 中广播查找变量。该变量的类型为 scala.collection.immutable.Map[String, MyObject]

MyObject 有以下字段

1. 字符串类型的“名称”
2. 字符串类型的“地址”
3. com.google.common.collect.{TreeRangeSet}
类型的“rangeSet”

线程“主”java.io.NotSerializableException 中的异常：com.google.common.collect.TreeRangeSet

Serialization stack:
    - object not serializable (class: com.google.common.collect.TreeRangeSet, value: {[/101.32.168.0‥/101.32.181.255][/4626:7800:4048:0:0:0:0:0‥/4626:7800:4048:ffff:ffff:ffff:ffff:ffff]})
    - field (class: com.test.MyObject, name: rangeSet, type: class com.google.common.collect.TreeRangeSet)
    - object (class com.test.MyObject, MyObject(Jack,Test,{[/101.32.168.0‥/101.32.181.255][/192.16.10.224‥/192.16.10.255][/4626:7800:4048:0:0:0:0:0‥/4626:7800:4048:ffff:ffff:ffff:ffff:ffff]}))
    - writeObject data (class: scala.collection.immutable.HashMap$SerializationProxy)
    - object (class scala.collection.immutable.HashMap$SerializationProxy, scala.collection.immutable.HashMap$SerializationProxy@708f7386)
    - writeReplace data (class: scala.collection.immutable.HashMap$SerializationProxy)

MyObject.scala

import com.google.common.collect.{TreeRangeSet}
@SerialVersionUID(123L)
case class MyObject(name:String, address:String,rangeSet:TreeRangeSet[CustomInetAddress]) {
}

CustomInetAddress.java

public class CustomInetAddress implements Comparable<CustomInetAddress>, Serializable {

    private InetAddress inetAddress;

    public CustomInetAddress(String ip) throws UnknownHostException {
        this.inetAddress = InetAddress.getByName(ip);
    }

    public CustomInetAddress(InetAddress address) throws UnknownHostException {
        this.inetAddress = address;
    }

    @Override
    public int compareTo(final CustomInetAddress address){
        byte[] ba1 = this.inetAddress.getAddress();
        byte[] ba2 = address.inetAddress.getAddress();

        if(ba1.length < ba2.length) return -1;
        if(ba1.length > ba2.length) return 1;

        for(int i = 0; i < ba1.length; i++) {
            int b1 = unsignedByteToInt(ba1[i]);
            int b2 = unsignedByteToInt(ba2[i]);
            if(b1 == b2)
                continue;
            if(b1 < b2)
                return -1;
            else
                return 1;
        }
        return 0;
    }

    @Override
    public String toString(){
        return this.inetAddress.toString();
    }

    private int unsignedByteToInt(byte b) {
        return (int) b & 0xFF;
    }
}

TreeRangeSet[CustomInetAddress] 是对象的实际类型。 CustomInetAddress 有一个 InetAddress 类型的字段。它们都是可序列化的。我不确定为什么会抛出异常。

Answer 1

消息很明确：com.google.common.collect.TreeRangeSet 没有实现 Serializable。