HiveServer2 HA模型部署测试

配置zookeeper，启动多个HiveServer

配置HA Zookeeper信息如下：

    <property>
      <name>hive.server2.transport.mode</name>
      <value>binary</value>
    </property>
    
    <property>
      <name>hive.server2.zookeeper.namespace</name>
      <value>hiveserver2-lsm</value>
    </property>

    <property>
      <name>hive.vectorized.execution.enabled</name>
      <value>true</value>
    </property>

    <property>
      <name>hive.zookeeper.quorum</name>
      <value>hadoop710.lt.163.org:2181,hadoop711.lt.163.org:2181,hadoop712.lt.163.org:2181</value>
    </property>

首先，在host1上启动MetaStore和HiveServer进程；
然后，在另外一台机器上启动HiveServer进程，组成简单的HA集群；

2017-06-27T10:31:18,263  INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/home/hzlishuming/env/apache-hive-2.1.1-bin2017-06-27T10:31:18,264  INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop710.lt.163.org:2181,hadoop711.lt.163.org:2181,hadoop712.lt.163.org:2181 sessionTimeout=1200000 watcher=org.apache.curator.ConnectionState@3f9b7fe12017-06-27T10:31:18,264  INFO [Thread-11] zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop710.lt.163.org:2181,hadoop711.lt.163.org:2181,hadoop712.lt.163.org:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@47b178302017-06-27T10:31:18,295  INFO [main-SendThread(hadoop711.lt.163.org:2181)] zookeeper.Login: successfully logged in.2017-06-27T10:31:18,296  INFO [Thread-12] zookeeper.Login: TGT refresh thread started.2017-06-27T10:31:18,297  INFO [Thread-11-SendThread(hadoop710.lt.163.org:2181)] zookeeper.Login: successfully logged in.2017-06-27T10:31:18,298  INFO [Thread-13] zookeeper.Login: TGT refresh thread started.2017-06-27T10:31:18,301  INFO [main-SendThread(hadoop711.lt.163.org:2181)] client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.2017-06-27T10:31:18,301  INFO [Thread-11-SendThread(hadoop710.lt.163.org:2181)] client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.2017-06-27T10:31:18,304  INFO [Thread-11-SendThread(hadoop710.lt.163.org:2181)] zookeeper.ClientCnxn: Opening socket connection to server hadoop710.lt.163.org/10.120.219.54:2181. Will attempt to SASL-authenticate using Login Context section 'HiveZooKeeperClient'2017-06-27T10:31:18,304  INFO [main-SendThread(hadoop711.lt.163.org:2181)] zookeeper.ClientCnxn: Opening socket connection to server hadoop711.lt.163.org/10.120.219.55:2181. Will attempt to SASL-authenticate using Login Context section 'HiveZooKeeperClient'2017-06-27T10:31:18,304  INFO [Thread-11-SendThread(hadoop710.lt.163.org:2181)] zookeeper.ClientCnxn: Socket connection established to hadoop710.lt.163.org/10.120.219.54:2181, initiating session2017-06-27T10:31:18,304  INFO [main-SendThread(hadoop711.lt.163.org:2181)] zookeeper.ClientCnxn: Socket connection established to hadoop711.lt.163.org/10.120.219.55:2181, initiating session

Beeline连接

./bin/beeline !connect jdbc:hive2://hadoop710.lt.163.org:2181,hadoop711.lt.163.org:2181,hadoop712.lt.163.org:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-lsm hadoop ""

beeline> !connect jdbc:hive2://hadoop710.lt.163.org:2181,hadoop711.lt.163.org:2181,hadoop712.lt.163.org:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-lsm hadoop ""SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hzlishuming/env/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/ndp/0.1.0/yarn_client/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to jdbc:hive2://hadoop710.lt.163.org:2181,hadoop711.lt.163.org:2181,hadoop712.lt.163.org:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-lsm17/06/27 14:04:09 [main]: INFO jdbc.HiveConnection: Connected to hadoop692.lt.163.org:10000Connected to: Apache Hive (version 2.1.1)
Driver: Hive JDBC (version 2.1.1)17/06/27 14:04:09 [main]: WARN jdbc.HiveConnection: Request to set autoCommit to false; Hive does not support autoCommit=false.
Transaction isolation: TRANSACTION_REPEATABLE_READ0: jdbc:hive2://hadoop710.lt.163.org:2181,had> show databases;+--------------------+--+
|   database_name    |
+--------------------+--+
| default            |
| hive_examples      |
| tpcds_data_test_2  |
+--------------------+--+3 rows selected (2.017 seconds)

Zookeeper状态

目前有两台机器组成的集群，查看Zookeeper中的状态信息：

[zk: hadoop712.lt.163.org(CONNECTED) 1] ls /hiveserver2-lsm

[serverUri=hadoop691.lt.163.org:10000;version=2.1.1;sequence=0000000003, serverUri=hadoop692.lt.163.org:10000;version=2.1.1;sequence=0000000002]

[zk: hadoop712.lt.163.org(CONNECTED) 2] get /hiveserver2-lsm/serverUri=hadoop691.lt.163.org:10000;version=2.1.1;sequence=0000000003hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hadoop691.lt.163.org;hive.server2.thrift.port=10000;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_HOST@TEST.AMBARI.NETEASE.COM
cZxid = 0x70001d53dctime = Tue Jun 27 11:02:34 CST 2017mZxid = 0x70001d53dmtime = Tue Jun 27 11:02:34 CST 2017pZxid = 0x70001d53dcversion = 0dataVersion = 0aclVersion = 0ephemeralOwner = 0x25cc3695ce40f40dataLength = 296numChildren = 0

高可用性模拟

当前Session正连接与hadoop692.lt.163.org机器，现将该机器上的hiveserver进程kill掉，观察当前Session以及后续创建的Session会有如何影响？

当前Session处理会出现异常，如下：

0: jdbc:hive2://hadoop710.lt.163.org:2181,had> show databases;Unexpected end of file when reading from HS2 server. The root cause might be too many concurrent connections. Please ask the administrator to check the number of active connections, and adjust hive.server2.thrift.max.worker.threads if applicable.Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)

再次创建Session，连接另外一台Host成功，符合预期：

hzlishuming@hadoop691:~/env/hive$ ./bin/beeline
Beeline version 2.1.1 by Apache Hive
beeline> !connect jdbc:hive2://hadoop710.lt.163.org:2181,hadoop711.lt.163.org:2181,hadoop712.lt.163.org:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-lsm hadoop ""17/06/28 10:06:30 [main]: INFO jdbc.HiveConnection: Connected to hadoop691.lt.163.org:10000Connected to: Apache Hive (version 2.1.1)Driver: Hive JDBC (version 2.1.1)17/06/28 10:06:30 [main]: WARN jdbc.HiveConnection: Request to set autoCommit to false; Hive does not support autoCommit=false.
Transaction isolation: TRANSACTION_REPEATABLE_READ0: jdbc:hive2://hadoop710.lt.163.org:2181,had> show databases;

总结

• HiveServer2 HA模型配置、实现较为简单，是基于负载均衡的方式实现路由高可用的，Zookeeper层保存各节点的Host+Port信息，在创建Session的时候随机从中选择一台Host进行连接；

• 当有机器挂掉时，会在Zookeeper中删除该节点，实现高可用性能；

作者：分裂四人组
链接：https://www.jianshu.com/p/4e1345a659b8