[Spark] 提交Spark任务报错 return code 3

环境:百台2U,数据量较大

报错信息:
2017-05-28 00:06:09,997 | ERROR | HiveServer2-Background-Pool: Thread-2033273 | Error running hive query: | org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:248)
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:339)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:169)
at org.apache.hive.service.cli.operation.SQLOperation.access$200(SQLOperation.java:75)
at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:245)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726)
at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:258)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

原因及解决:

从Server日志看,出现了频繁Full GC,多次Full GC内存还是不够,自己kill了。


建议调整执行器的内存,调整方法如下
如果命令是在客户端提交的话,提交前在beeline客户端执行下面命令:
set spark.executor.memory=6G;(内存大小根据实际情况设置),建议不要超过20GB。
如果是程序中提交的,可以在程序中修改上面参数值,或者修改服务端的配置(需要重启生效)

分类上一篇:     分类下一篇:无,已是最新文章

2 Comments

  1. 斯蒂芬斯蒂芬

    你好你好好你好

    • admin (Author)

      你好

Leave a Reply