scala apache-spark jackson sbt incompatibletypeerror

scala - Spark2.1.0 Jackson versiones incompatibles 2.7.6



apache-spark sbt (3)

Para tu información En mi caso, estoy usando spark y kafka-streams en la aplicación, mientras que kafka-streams usa com.fasterxml.jackson.core 2.8.5. Adición de exclude como abajo solucionado el problema

(Gradle)

compile (group: "org.apache.kafka", name: "kafka-streams", version: "0.11.0.0"){ exclude group:"com.fasterxml.jackson.core" }

Estoy tratando de ejecutar un simple ejemplo de chispa en intellij, pero me sale el error así:

Exception in thread "main" java.lang.ExceptionInInitializerError at org.apache.spark.SparkContext.withScope(SparkContext.scala:701) at org.apache.spark.SparkContext.textFile(SparkContext.scala:819) at spark.test$.main(test.scala:19) at spark.test.main(test.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.7.6 at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64) at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19) at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:730) at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82) at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)

Intenté actualizar mi dependencia de Jackson, pero parece que no funciona, hago esto:

libraryDependencies += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7" libraryDependencies += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7"

pero siguen apareciendo los mismos mensajes de error, ¿puede alguien ayudarme a corregir el error?

Aquí está el código de ejemplo de chispa:

object test { def main(args: Array[String]): Unit = { if (args.length < 1) { System.err.println("Usage: <file>") System.exit(1) } val conf = new SparkConf() val sc = new SparkContext("local","wordcount",conf) val line = sc.textFile(args(0)) line.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_+_).collect().foreach(println) sc.stop() } }

Y aquí está mi built.sbt:

name := "testSpark2" version := "1.0" scalaVersion := "2.11.8" libraryDependencies += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7" libraryDependencies += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7" libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-mllib_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-repl_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-streaming-flume_2.10" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-network-shuffle_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-hive_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-streaming-flume-assembly_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-mesos_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-graphx_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-catalyst_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-launcher_2.11" % "2.1.0"



Spark 2.1.0 contiene com.fasterxml.jackson.core como dependencia transitiva. Por lo tanto, no es necesario incluirlos en la libraryDependencies .

Pero si desea agregar una com.fasterxml.jackson.core diferente de com.fasterxml.jackson.core dependencias com.fasterxml.jackson.core . Me gusta esto:

name := "testSpark2" version := "1.0" scalaVersion := "2.11.8" dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7" dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7" dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.8.7" libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-mllib_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-repl_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-streaming-flume_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-network-shuffle_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-hive_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-streaming-flume-assembly_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-mesos_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-graphx_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-catalyst_2.11" % "2.1.0" libraryDependencies += "org.apache.spark" % "spark-launcher_2.11" % "2.1.0"

Por lo tanto, cambie su build.sbt como el de arriba y funcionará como se esperaba.

¡Espero que ayude!