Flink Standalone HA搭建指南

基本环境

集群主机 IP
host147 10.45.151.147
host148 10.45.151.148
host149 10.45.151.149

配置域名映射

vim /etc/hosts

10.45.151.147 host147
10.45.151.148 host148
10.45.151.149 host149

下载Flink安装包,将其放到/data目录下,并解压

mkdir /data && cd /data
wget http://192.168.1.3:11180/downloads/flink-1.13.5-bin-scala_2.12.tgz
tar zxvf flink-1.13.5-bin-scala_2.12.tgz
cd flink-1.13.5-bin

使用flink内置zk启动zk集群

选定上述三个节点作为安装zk集群的节点。

在conf/flink-conf.yaml中设置java信息,新增如下配置

java.home: xxx

新建目录

mkdir /data/flink-1.13.5/zookeeper

修改配置文件conf/zoo.cfg

# The number of milliseconds of each tick
tickTime=2000

# The number of ticks that the initial  synchronization phase can take
initLimit=10

# The number of ticks that can pass between  sending a request and getting an acknowledgement
syncLimit=5

# The directory where the snapshot is stored.
dataDir=/data/flink-1.13.5/zookeeper

# The port at which the clients will connect
clientPort=2181

# ZooKeeper quorum peers
server.1=host147:2888:3888
server.2=host148:2888:3888
server.3=host149:2888:3888

进入host147主机的/data/flink-1.13.5/zookeeper目录,新建文件myid,并填入1

进入host148主机的/data/flink-1.13.5/zookeeper目录,新建文件myid,并填入2

进入host149主机的/data/flink-1.13.5/zookeeper目录,新建文件myid,并填入3

启动zookeeper集群

在host147中执行:/data/flink-1.13.5/bin/zookeeper.sh start 1

在host148中执行:/data/flink-1.13.5/bin/zookeeper.sh start 2

在host149中执行:/data/flink-1.13.5/bin/zookeeper.sh start 3

至此zookeeper高可用集群搭建完成

启动Flink高可用集群

由于不想用hdfs,本文以minio对象存储作为checkpoints和高可用存储路径。可参考文章:分布式minio搭建指南了解如何搭建minio集群。

修改Flink配置conf/flink-conf.yaml

(下述配置每台主机都一样)

配置minio访问信息

s3.endpoint: http://minio:9000
s3.path.style.access: true
s3.access.key: xxxxxxxxxxxxx
s3.secret.key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

配置高可用信息

high-availability: zookeeper
high-availability.storageDir: s3://flink/ha
high-availability.zookeeper.quorum: host147:2181,host148:2181,host149:2181
high-availability.zookeeper.path.root: /flink
high-availability.cluster-id: /cluster_one

设置checkpoint信息

state.checkpoints.dir: s3://flink/chk
state.savepoints.dir: s3://flink/chk
state.backend.incremental: true
state.backend: rocksdb
execution.checkpointing.interval: 15000ms

在host147节点conf/flink-conf.yaml配置文件中,修改

jobmanager.rpc.address:host147

在host148节点conf/flink-conf.yaml配置文件中,修改

jobmanager.rpc.address:host148

将opt/flink-s3-fs-hadoop-1.13.5.jar复制到lib/目录中

host147节点启动jobmanager

bin/jobmanager.sh start

host148节点启动jobmanager

bin/jobmanager.sh start

host147、host148、host149节点启动taskmanager

bin/taskmanager.sh start
0 0 投票数
文章评分

本文为从大数据到人工智能博主「xiaozhch5」的原创文章,遵循CC 4.0 BY-SA版权协议,转载请附上原文出处链接及本声明。

原文链接:https://lrting.top/backend/4008/

(0)
上一篇 2022-03-04 23:54
下一篇 2022-03-05 00:27

相关推荐

订阅评论
提醒
guest
0 评论
内联反馈
查看所有评论
0
希望看到您的想法,请您发表评论x