念念不忘
必有回响🎉

Hue整合Oozie和Spark2

在Hue中使用Oozie调度Spark2任务时总是调用不起来,默认Hue也只支持Spark,那么要支持Spark2还需要进行如下操作。

1.将Spark2中的jar包依赖上传到/user/oozie/share/lib/中:

hadoop fs -mkdir -p /user/oozie/share/lib/lib_20170328134250/spark2/

lib_20170328134250这个名字是原来配置CDH5那一套配置的名字,你可以重新换个路径,只要在/user/oozie/share下面即可。
上传Spakr2的jar包

hadoop fs -put /opt/cloudera/parcels/SPARK2/lib/spark2/jars/* /user/oozie/share/lib/lib_20170328134250/spark2

给新增的jar赋权限

hadoop fs -chown -R oozie:oozie /user/oozie/share/lib/lib_20170328134250/spark2
hadoop fs -chmod 755 /user/oozie/share/lib/lib_20170328134250/spark2

2.更新oozie的shareliblist。
添加新增的spark2到shareliblist

oozie admin -oozie http://10.10.1.100:11000/oozie -sharelibupdate

返回结果

oozie admin -oozie http://10.10.1.100:11000/oozie -sharelibupdate
[ShareLib update status]
        sharelibDirOld = hdfs://1010.10.1.100:8020/user/oozie/share/lib/lib_20170328134250
        host = http://10.10.1.100:11000/oozie
        sharelibDirNew = hdfs://10.10.1.100:8020/user/oozie/share/lib/lib_20170328134250
        status = Successful

查看更新后的Spark2

oozie admin -oozie http://10.10.161.101:11000/oozie -shareliblist spark2
[Available ShareLib]
spark2
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/JavaEWAH-0.3.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/RoaringBitmap-0.5.11.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/ST4-4.0.4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/antlr-2.7.7.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/antlr-runtime-3.4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/antlr4-runtime-4.5.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/aopalliance-repackaged-2.4.0-b34.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/apache-log4j-extras-1.2.17.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/arpack_combined_all-0.1.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/avro-1.7.6-cdh5.13.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/avro-ipc-1.7.6-cdh5.13.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/avro-mapred-1.7.6-cdh5.13.3-hadoop2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/bonecp-0.7.1.RELEASE.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/breeze-macros_2.11-0.13.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/breeze_2.11-0.13.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/calcite-avatica-1.0.0-incubating.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/calcite-core-1.0.0-incubating.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/calcite-linq4j-1.0.0-incubating.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/chill-java-0.8.0.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/chill_2.11-0.8.0.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/commons-cli-1.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/commons-codec-1.10.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/commons-collections-3.2.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/commons-compiler-3.0.8.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/commons-compress-1.4.1.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/commons-crypto-1.0.0.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/commons-dbcp-1.4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/commons-httpclient-3.1.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/commons-io-2.4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/commons-lang-2.6.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/commons-lang3-3.5.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/commons-logging-1.1.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/commons-math3-3.4.1.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/commons-net-2.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/commons-pool-1.5.4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/compress-lzf-1.0.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/core-1.1.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/datanucleus-api-jdo-3.2.6.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/datanucleus-core-3.2.10.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/datanucleus-rdbms-3.2.9.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/derby-10.11.1.1.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/eigenbase-properties-1.1.4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/fastutil-6.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/flume-ng-configuration-1.6.0-cdh5.13.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/flume-ng-core-1.6.0-cdh5.13.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/flume-ng-sdk-1.6.0-cdh5.13.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/gson-2.2.4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/hive-metastore-1.1.0-cdh5.13.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/hive-serde-1.1.0-cdh5.13.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/hive-shims-0.23-1.1.0-cdh5.13.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/hive-shims-1.1.0-cdh5.13.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/hive-shims-common-1.1.0-cdh5.13.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/hive-shims-scheduler-1.1.0-cdh5.13.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/hk2-api-2.4.0-b34.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/hk2-locator-2.4.0-b34.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/hk2-utils-2.4.0-b34.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/httpcore-4.4.4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/ivy-2.4.0.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jackson-annotations-2.6.5.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jackson-core-2.6.5.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jackson-databind-2.6.5-cloudera.1.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jackson-jaxrs-1.8.8.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jackson-module-paranamer-2.6.5.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jackson-module-scala_2.11-2.6.5.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jackson-xc-1.8.8.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/janino-3.0.8.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/javassist-3.18.1-GA.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/javax.annotation-api-1.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/javax.inject-2.4.0-b34.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/javax.servlet-api-3.1.0.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/javax.ws.rs-api-2.0.1.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/javolution-5.5.1.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jcl-over-slf4j-1.7.5.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jdo-api-3.0.1.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jersey-client-2.22.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jersey-common-2.22.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jersey-container-servlet-2.22.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jersey-container-servlet-core-2.22.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jersey-guava-2.22.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jersey-media-jaxb-2.22.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jersey-server-2.22.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jetty-6.1.26.cloudera.4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jetty-util-6.1.26.cloudera.4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jline-2.12.1.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/joda-time-2.9.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jodd-core-3.5.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/json-20090211.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/json4s-ast_2.11-3.2.11.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/json4s-core_2.11-3.2.11.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/json4s-jackson_2.11-3.2.11.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jta-1.1.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jtransforms-2.4.0.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/jul-to-slf4j-1.7.5.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/kryo-shaded-3.0.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/libfb303-0.9.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/libthrift-0.9.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/logredactor-1.0.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/lz4-1.3.0.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/machinist_2.11-0.6.1.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/macro-compat_2.11-1.1.1.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/metrics-core-3.1.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/metrics-graphite-3.1.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/metrics-json-3.1.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/metrics-jvm-3.1.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/mina-core-2.0.0-M5.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/minlog-1.3.0.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/mockito-all-1.8.5.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/netty-3.9.9.Final.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/netty-all-4.0.43.Final.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/objenesis-2.1.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/oozie-sharelib-spark-4.1.0-cdh5.8.0.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/oozie-sharelib-spark.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/opencsv-2.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/oro-2.0.8.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/osgi-resource-locator-1.0.1.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/paranamer-2.6.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/parquet-avro-1.5.0-cdh5.13.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/parquet-column-1.5.0-cdh5.13.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/parquet-common-1.5.0-cdh5.13.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/parquet-encoding-1.5.0-cdh5.13.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/parquet-format-2.1.0-cdh5.13.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/parquet-hadoop-1.5.0-cdh5.13.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/parquet-hadoop-bundle-1.5.0-cdh5.13.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/parquet-jackson-1.5.0-cdh5.13.3.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/pmml-model-1.2.15.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/pmml-schema-1.2.15.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/py4j-0.10.7.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/pyrolite-4.13.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/scala-compiler-2.11.8.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/scala-library-2.11.8.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/scala-parser-combinators_2.11-1.0.4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/scala-reflect-2.11.8.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/scala-xml_2.11-1.0.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/scalap-2.11.8.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/shapeless_2.11-2.3.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/snappy-0.2.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/spark-catalyst_2.11-2.2.0.cloudera4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/spark-core_2.11-2.2.0.cloudera4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/spark-graphx_2.11-2.2.0.cloudera4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/spark-hive-exec_2.11-2.2.0.cloudera4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/spark-hive_2.11-2.2.0.cloudera4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/spark-launcher_2.11-2.2.0.cloudera4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/spark-mllib-local_2.11-2.2.0.cloudera4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/spark-mllib_2.11-2.2.0.cloudera4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/spark-network-common_2.11-2.2.0.cloudera4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/spark-network-shuffle_2.11-2.2.0.cloudera4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/spark-repl_2.11-2.2.0.cloudera4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/spark-sketch_2.11-2.2.0.cloudera4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/spark-sql_2.11-2.2.0.cloudera4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/spark-streaming-flume-sink_2.11-2.2.0.cloudera4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/spark-streaming-flume_2.11-2.2.0.cloudera4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/spark-streaming_2.11-2.2.0.cloudera4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/spark-tags_2.11-2.2.0.cloudera4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/spark-unsafe_2.11-2.2.0.cloudera4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/spark-yarn_2.11-2.2.0.cloudera4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/spire-macros_2.11-0.13.0.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/spire_2.11-0.13.0.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/stax-api-1.0.1.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/stream-2.7.0.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/stringtemplate-3.2.1.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/univocity-parsers-2.2.1.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/validation-api-1.1.0.Final.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/xbean-asm5-shaded-4.4.jar
        hdfs://master:8020/user/oozie/share/lib/lib_20170328134250/spark2/xz-1.0.jar

3.配置Hue

点击右上角的齿轮,打开配置页面:

在Options List 中填入如下内容:

--driver-memory 2g --executor-memory 8g --executor-cores 2 --driver-cores 1 --num-executors 3

 
在PROPERTIES中填入如下内容:

oozie.action.sharelib.for.spark  spark2

此时便可以提交基于Local模式的Spark Job,但是在提交yarn模式以及cluster的任务时,会出现如下错误:

java.lang.IllegalStateException: Library directory '/yarn/nm/usercache/bus/appcache/application_1532510362310_2772/container_1532510362310_2772_01_000002/./assembly/target/scala-2.11/jars' does not exist; make sure Spark is built.
	at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
	at org.apache.spark.launcher.CommandBuilderUtils.findJarsDir(CommandBuilderUtils.java:347)
	at org.apache.spark.launcher.YarnCommandBuilderUtils$.findJarsDir(YarnCommandBuilderUtils.scala:38)
	at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:567)
	at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:857)
	at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:172)
	at org.apache.spark.deploy.yarn.Client.run(Client.scala:1134)
	at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1193)
	at org.apache.spark.deploy.yarn.Client.main(Client.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
	at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:256)
	at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:207)
	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:49)
	at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:52)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

以及

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Application application_1532510362310_2781 finished with failed status
org.apache.spark.SparkException: Application application_1532510362310_2781 finished with failed status
	at org.apache.spark.deploy.yarn.Client.run(Client.scala:1147)
	at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1193)
	at org.apache.spark.deploy.yarn.Client.main(Client.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
	at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:256)
	at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:207)
	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:49)
	at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:52)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

等有空了研究下这个问题如何解决。
 

赞(0) 打赏
未经允许不得转载:Charles's Blog » Hue整合Oozie和Spark2

评论 抢沙发

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏