Iceberg编译部署

编译

下载源码

git clone https://github.com/apache/iceberg.git

切换到0.12分支

gco 0.12.x 

安装gradle

由于iceberg采用gradle来管理项目, 在编译之前需要安装gradle
检查jdk版本, gradle需要jdk8以及以上版本

java -version
java version "1.8.0_261"
Java(TM) SE Runtime Environment (build 1.8.0_261-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.261-b12, mixed mode)

安装gradle

brew install gradle

修改环境变量

export GRADLE=/Users/home_name/opt/gradle-7.0

验证安装结果

gradle -v

------------------------------------------------------------
Gradle 7.0
------------------------------------------------------------

Build time:   2021-04-09 22:27:31 UTC
Revision:     d5661e3f0e07a8caff705f1badf79fb5df8022c4

Kotlin:       1.4.31
Groovy:       3.0.7
Ant:          Apache Ant(TM) version 1.10.9 compiled on September 27 2020
JVM:          1.8.0_261 (Oracle Corporation 25.261-b12)
OS:           Mac OS X 10.14.6 x86_64

编译

./gradlew build -x test -x integrationTest

部署

将jar包(iceberg-flink-runtime-0.12.0.jar)发送到远程服务器flink测试集群lib目录下, 重启集群

测试运行

创建Hadoop Catalog

 CREATE CATALOG hadoop_catalog WITH (
  'type'='iceberg',
  'catalog-type'='hadoop',
  'warehouse'='hdfs://xxx-hdfs/flink/tmp/iceberg_test',
  'property-version'='1'
)
;

创建测试库

CREATE DATABASE hadoop_catalog.iceberg_db;

创建format为v2的表

CREATE TABLE hadoop_catalog.iceberg_db.sample_test (
    id BIGINT COMMENT 'unique id',
    data STRING,
    PRIMARY KEY(id) NOT ENFORCED
)
WITH (
    'format-version'= '2',
    'write.upsert.enable'='true'
)
;

Insert 数据

INSERT INTO hadoop_catalog.iceberg_db.sample_test  VALUES (10, 'test10_U'), (11, 'test11'), (12, 'test12');

查看insert结果

select * from hadoop_catalog.iceberg_db.sample_test;