java.io.InvalidClassException 解决方案

原创
2017/06/01 14:59
阅读数 2.9K

一、出现场景

Spark读取Elasticsearch的时候,抛出的异常

二、错误日志

org.elasticsearch.hadoop.serialization.EsHadoopSerializationException: cannot deserialize object
    at org.elasticsearch.hadoop.util.IOUtils.deserializeFromBase64(IOUtils.java:84)
    at org.elasticsearch.hadoop.rest.RestService.createReader(RestService.java:303)
    at org.elasticsearch.hadoop.mr.EsInputFormat$ShardRecordReader.init(EsInputFormat.java:206)
    at org.elasticsearch.hadoop.mr.EsInputFormat$WritableShardRecordReader.init(EsInputFormat.java:386)
    at org.elasticsearch.hadoop.mr.EsInputFormat$ShardRecordReader.initialize(EsInputFormat.java:188)
    at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:133)
    at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104)
    at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
    at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:69)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:242)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)
    at org.apache.spark.scheduler.Task.run(Task.scala:70)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.InvalidClassException: org.elasticsearch.hadoop.serialization.dto.mapping.Field; local class incompatible: stream classdesc serialVersionUID = 4438505825805813822, local class serialVersionUID = -3451876473461899198
    at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
    at org.elasticsearch.hadoop.util.IOUtils.deserializeFromBase64(IOUtils.java:79)
    ... 19 more

三、解决方案

org.elasticsearch.hadoop.serialization.dto.mapping.Field 这个类在elasticsearch-hadoop-2.3.3.jar中。

应用程序lib下面有elasticsearch-hadoop-2.3.3.jar,它会被分发到spark集群的worker节点上去。而Spark的Worker节点在执行的时候也会加载/home/hadoop/software/spark-1.4.1/lib下的Jar,不巧的是/home/hadoop/software/spark-1.4.1/lib下面有个Jar叫做,elasticsearch-hadoop-hive-2.1.2.jar,反编译后发现,也有org.elasticsearch.hadoop.serialization.dto.mapping.Field同样的类。删除掉elasticsearch-hadoop-hive-2.1.2.jar,重启Spark集群。问题解决。

展开阅读全文
打赏
0
0 收藏
分享
加载中
更多评论
打赏
0 评论
0 收藏
0
分享
返回顶部
顶部