Spark Multi Tenancy系列 - 4 异常处理：初始化HiveSessionState失败

异常说明

SparkSession初始化一个lazy的HiveClient用于Hive MetaStore连接，一旦使用其内部就会实例化一个org.apache.spark.sql.hive.HiveSessionState对象，执行SessionState.start方法时会根据HiveConf里的配置执行一系列操作，包括创建一些临时的工作目录等。这些目录如果没有正确的权限会导致这个过程失败，致使连接不上Server.

客户端异常

hzyaoqin@hadoop980:~/spark-packages/spark-2.1.0-bin-ne-1.3.0-SNAPSHOT$ bin/beeline -u 'jdbc:hive2://hadoop980.hz.163.org:10001/default;principal=hive/hadoop980.hz.163.org@TEST.MAMMUT.NETEASE.COM;hive.server2.proxy.user=hzyaoqin;spark.sql.warehouse.dir=/user/hzyaoqin/warehouse'Connecting to jdbc:hive2://hadoop980.hz.163.org:10001/default;principal=hive/hadoop980.hz.163.org@TEST.MAMMUT.NETEASE.COM;hive.server2.proxy.user=hzyaoqin;spark.sql.warehouse.dir=/user/hzyaoqin/warehouse17/09/06 17:17:14 INFO jdbc.Utils: Supplied authorities: hadoop980.hz.163.org:1000117/09/06 17:17:14 INFO jdbc.Utils: Resolved authority: hadoop980.hz.163.org:1000117/09/06 17:17:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable17/09/06 17:17:15 INFO jdbc.HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://hadoop980.hz.163.org:10001/default;principal=hive/hadoop980.hz.163.org@TEST.MAMMUT.NETEASE.COM;hive.server2.proxy.user=hzyaoqin;spark.sql.warehouse.dir=/user/hzyaoqin/warehouseCan't overwrite cause with java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':Beeline version 1.2.1.spark2 by Apache Hive0: jdbc:hive2://hadoop980.hz.163.org:10001/de (closed)>

其中异常主体为，

Can't overwrite cause with java.lang.IllegalArgumentException:
      Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':

， 实际意为初始化HiveSessionState时发生了不可名状的事情；进一步去看服务端日志；
_{注：HiveSession级别的日志没有做到客户端这边，只有Operation级别的日志可以看哦}

服务端异常

Error opening session:
org.apache.spark.SparkException: Failed Init SparkSession for user[hzyaoqin]java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':
        at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl.createSparkSession(SparkHiveSessionImpl.scala:135)
        at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl.getOrCreateSparkSession(SparkHiveSessionImpl.scala:109)
        at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl.open(SparkHiveSessionImpl.scala:186)
        at org.apache.spark.sql.hive.thriftserver.multitenancy.ThriftServerSessionManager.openSession(ThriftServerSessionManager.scala:225)
        at org.apache.spark.sql.hive.thriftserver.multitenancy.ThriftServerCLIService.openSessionWithImpersonation(ThriftServerCLIService.scala:91)
        at org.apache.spark.sql.hive.thriftserver.multitenancy.ThriftClientCLIService.getSessionHandle(ThriftClientCLIService.scala:252)
        at org.apache.spark.sql.hive.thriftserver.multitenancy.ThriftClientCLIService.OpenSession(ThriftClientCLIService.scala:266)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1253)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1238)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)Caused by: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':
        at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:1120)
        at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:111)
        at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:110)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$11.apply(SparkSession.scala:888)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$11.apply(SparkSession.scala:888)
        at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
        at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
        at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
        at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
        at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:888)
        at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl$$anon$1.run(SparkHiveSessionImpl.scala:124)
        at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl$$anon$1.run(SparkHiveSessionImpl.scala:118)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
        at org.apache.spark.sql.hive.thriftserver.multitenancy.SparkHiveSessionImpl.createSparkSession(SparkHiveSessionImpl.scala:118)
        ... 15 moreCaused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:1117)
        ... 31 moreCaused by: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveExternalCatalog':
        at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:183)
        at org.apache.spark.sql.internal.SharedState.<init>(SharedState.scala:90)
        at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:102)
        at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:102)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.sql.SparkSession.sharedState$lzycompute(SparkSession.scala:102)
        at org.apache.spark.sql.SparkSession.sharedState(SparkSession.scala:101)
        at org.apache.spark.sql.internal.SessionState.<init>(SessionState.scala:157)
        at org.apache.spark.sql.hive.HiveSessionState.<init>(HiveSessionState.scala:32)
        ... 36 moreCaused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:180)
        ... 44 moreCaused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:264)
        at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:367)
        at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:271)
        at org.apache.spark.sql.hive.HiveExternalCatalog.<init>(HiveExternalCatalog.scala:65)
        ... 49 moreCaused by: java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: Permission denied: user=hzyaoqin, access=EXECUTE, inode="/tmp/hive/public/hzyaoqin/9869b00f-0629-4290-ad0f-71f3c9ed0566":hdfs:hadoop:drwxr-x---
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:259)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:205)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1722)
        at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:108)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3863)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1012)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:843)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)

非常明显,hzyaoqin用户没有用户hdfs所属目录的执行权限；
Permission denied: user=hzyaoqin, access=EXECUTE, inode="/tmp/hive/public/hzyaoqin/9869b00f-0629-4290-ad0f-71f3c9ed0566":hdfs:hadoop:drwxr-x---
这个目录为hive.exec.strachdir（缺省为/tmp/hive/public）下创建username/sessionId的临时目录，我们多租户下执行都是用hzyaoqin这个执行账号去真正执行的，这个目录如果是hdfs用户的，必然不是有本次session创建的，应该是以前的遗留目录

解决方案

• 1、删除/tmp/hive/public/hzyaoqin（如果这目录没啥用），重新连接

• 2、重新配置hive.exec.strachdir到另一个有权限的目录，可以在连接串中动态的session级别指定或classpath中的hive-site.xml进行server级别的全局设置，前者重新连接，后者重启server。

作者：风景不美
链接：https://www.jianshu.com/p/5ae027ba7c9c