Rocketmq controller高可用部署

941 字
5 分钟
Rocketmq controller高可用部署
2024-06-09
📑数据库
Redis
浏览量 加载中...

[!info] 参考 # 自动主从切换快速开始 根据次文档改成 docker-compose 部署，并且使用三个NameServer 节点，可是部署到生产环境，根据需要添加broker

Ip角色
192.168.10.67NameServer 1，broker
192.168.10.68NameServer 2，broker
192.168.10.69NameServer 3

1 配置文件#

目录结构
tree
.
├── hosts.ini
└── rocketmq
    ├── broker-1
    │   ├── broker
    │   │   ├── broker.conf # broker配置文件
    │   │   └── plain_acl.yml # 权限控制
    │   ├── docker-compose.yml
    │   └── namesrv
    │       └── namesrv.conf
    ├── broker-2
    │   ├── broker
    │   │   ├── broker.conf # 与broker-1 配置一样
    │   │   └── plain_acl.yml
    │   ├── docker-compose.yml
    │   └── namesrv
    │       └── namesrv.conf # NameServer配置
    └── broker-3
        ├── docker-compose.yml
        ├── namesrv
        │   └── namesrv.conf
        └── users.properties # dashboard，账户配置

2 配置#

2.1 NameServer#

2.1.1 n0#

Terminal window
cat rocketmq/broker-1/namesrv/namesrv.conf
listenPort = 9876
enableControllerInNamesrv = true
controllerDLegerGroup = group1
controllerDLegerPeers = n0-192.168.10.67:9877;n1-192.168.10.68:9877;n2-192.168.10.69:9877
controllerDLegerSelfId = n0
controllerStorePath = /home/rocketmq/store/controller
enableElectUncleanMaster = false
notifyBrokerRoleChanged  =  true

2.1.2 n1#

Terminal window
cat rocketmq/broker-2/namesrv/namesrv.conf
listenPort = 9876
enableControllerInNamesrv = true
enableController = true
controllerDLegerGroup = group1
controllerDLegerPeers = n0-192.168.10.67:9877;n1-192.168.10.68:9877;n2-192.168.10.69:9877
controllerDLegerSelfId = n1
controllerStorePath = /home/rocketmq/store/controller
enableElectUncleanMaster = false
notifyBrokerRoleChanged  =  true

2.1.3 n2#

Terminal window
cat rocketmq/broker-3/namesrv/namesrv.conf
listenPort = 9876
enableController = true
enableControllerInNamesrv = true
controllerDLegerGroup = group1
controllerDLegerPeers = n0-192.168.10.67:9877;n1-192.168.10.68:9877;n2-192.168.10.69:9877
controllerDLegerSelfId = n2
controllerStorePath = /home/rocketmq/store/controller
enableElectUncleanMaster = false
notifyBrokerRoleChanged  =  true

2.2 Broker#

2.2.1 a#

两个节点配置一样

Terminal window
cat rocketmq/broker-1/broker/broker.conf
cat rocketmq/broker-2/broker/broker.conf
brokerClusterName=DefaultCluster
brokerName=broker-a
brokerId=-1
brokerRole=SLAVE
listenPort=10911
allAckInSyncStateSet=true
deleteWhen=04
fileReservedTime=48
flushDiskType=ASYNC_FLUSH
enableControllerMode = true
controllerAddr = 192.168.10.67:9877;192.168.10.68:9877;192.168.10.69:9877
namesrvAddr = 192.168.10.67:9876;192.168.10.68:9876;192.168.10.69:9876
storePathRootDir=/home/rocketmq/store
storePathCommitLog=/home/rocketmq/store/commitlog

2.2.2 b#

这里只是展示下配置多个broker 怎么写，此文档演示不会部署

Terminal window
brokerClusterName=DefaultCluster
brokerName=broker-b # 只需要修改这里
brokerId=-1
brokerRole=SLAVE
listenPort=10911
allAckInSyncStateSet=true
deleteWhen=04
fileReservedTime=48
flushDiskType=ASYNC_FLUSH
enableControllerMode = true
controllerAddr = 192.168.10.67:9877;192.168.10.68:9877;192.168.10.69:9877
namesrvAddr = 192.168.10.67:9876;192.168.10.68:9876;192.168.10.69:9876
storePathRootDir=/home/rocketmq/store
storePathCommitLog=/home/rocketmq/store/commitlog

2.3 docker-compose 配置#

2.3.1 第一个节点#

Terminal window
services:
  namesrv-1:
    image: apache/rocketmq:5.3.0
    container_name: rmqnamesrv-1
    network_mode: host
    environment:
      - MAX_POSSIBLE_HEAP=100000000
      - JAVA_OPT_EXT=-Xms512M -Xmx512M -Xmn128m -Drocketmq.namesrv.kvConfigPath=/home/rocketmq/store/kvConfig.json
    volumes:
      - ./namesrv/namesrv.conf:/home/rocketmq/rocketmq-5.3.0/conf/namesrv.conf
      - ./namesrv/store:/home/rocketmq/store
      - /etc/localtime:/etc/localtime
    expose:
      - 9877
      - 9877
    command: sh mqnamesrv -c /home/rocketmq/rocketmq-5.3.0/conf/namesrv.conf
  broker-a:
    image: apache/rocketmq:5.3.0
    container_name: rmqbroker-a
    network_mode: host
    environment:
      - JAVA_OPT_EXT=-Xms2048M -Xmx2048M -Xmn512m
      - JAVA_OPTS=-Duser.timezone=Asia/Shanghai
    expose:
      - 10909
      - 10911
      - 10912
    volumes:
      - ./broker/broker.conf:/home/rocketmq/rocketmq-5.3.0/conf/broker.conf
      - ./broker/plain_acl.yml:/home/rocketmq/rocketmq-5.3.0/conf/acl/plain_acl.yml
      - ./broker/logs:/home/rocketmq/logs
      - ./broker/store:/opt/rmqstore/store
      - /etc/localtime:/etc/localtime:ro
    command: sh mqbroker -c /home/rocketmq/rocketmq-5.3.0/conf/broker.conf
    restart: unless-stopped

2.3.2 第二个节点#

Terminal window
cat rocketmq/broker-2/docker-compose.yml
services:
  namesrv-2:
    image: apache/rocketmq:5.3.0
    container_name: rmqnamesrv-2
    network_mode: host
    environment:
      - MAX_POSSIBLE_HEAP=100000000
      - JAVA_OPT_EXT=-Xms512M -Xmx512M -Xmn128m -Drocketmq.namesrv.kvConfigPath=/home/rocketmq/store/kvConfig.json
    volumes:
      - ./namesrv/namesrv.conf:/home/rocketmq/rocketmq-5.3.0/conf/namesrv.conf
      - ./namesrv/store:/home/rocketmq/store
      - /etc/localtime:/etc/localtime
    expose:
      - 9877
    command: sh mqnamesrv -c /home/rocketmq/rocketmq-5.3.0/conf/namesrv.conf
  broker-a:
    image: apache/rocketmq:5.3.0
    container_name: rmqbroker-a
    network_mode: host
    environment:
      - JAVA_OPT_EXT=-Xms2048M -Xmx2048M -Xmn512m
      - JAVA_OPTS=-Duser.timezone=Asia/Shanghai
    expose:
      - 10909
      - 10911
      - 10912
    volumes:
      - ./broker/broker.conf:/home/rocketmq/rocketmq-5.3.0/conf/broker.conf
      - ./broker/plain_acl.yml:/home/rocketmq/rocketmq-5.3.0/conf/acl/plain_acl.yml
      - ./broker/logs:/home/rocketmq/logs
      - ./broker/store:/home/rocketmq/store
      - /etc/localtime:/etc/localtime:ro
    command: sh mqbroker -c  /home/rocketmq/rocketmq-5.3.0/conf/broker.conf

2.3.3 第三个节点#

Terminal window
cat rocketmq/broker-3/docker-compose.yml
services:
  namesrv-3:
    image: swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/apache/rocketmq:5.3.0
    container_name: rmqnamesrv-3
    network_mode: host
    environment:
      - MAX_POSSIBLE_HEAP=100000000
      - JAVA_OPT_EXT=-Xms512M -Xmx512M -Xmn128m -Drocketmq.namesrv.kvConfigPath=/home/rocketmq/store/kvConfig.json
    volumes:
      - ./namesrv/namesrv.conf:/home/rocketmq/rocketmq-5.3.0/conf/namesrv.conf
      - ./namesrv/store:/home/rocketmq/store
      - /etc/localtime:/etc/localtime
    expose:
      - 9877
    command: sh mqnamesrv -c /home/rocketmq/rocketmq-5.3.0/conf/namesrv.conf
  rocketmq-dashboard:
    image: swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/apacherocketmq/rocketmq-dashboard:latest
    container_name: rocketmq-dashboard
    environment:
      - JAVA_OPTS=-Drocketmq.namesrv.addr=192.168.10.67:9876;192.168.10.68:9876;192.168.10.69:9876 -Duser.timezone='Asia/Shanghai'
      - JAVA_OPT_EXT=-Xms512M -Xmx512M -Xmn128m
      - rocketmq.config.loginRequired=true
      - rocketmq.config.accessKey=vldRocketMQ
      - rocketmq.config.secretKey=ZKdyFW3AeknPDtz6cjDA
      - rocketmq.config.dataPath=/rocketmq-console-data
    volumes:
      - /etc/localtime:/etc/localtime
      - ./users.properties:/rocketmq-console-data/users.properties
    ports:
      - 8080:8080

3 启动服务#

[!attention] 这里使用 ansible 部署，也可以手动上次包

3.1 Hosts 文件#

已经配置免密，没有的可以取消注释修改

Terminal window
cat hosts.ini
[rocketmq]
192.168.10.67
192.168.10.68
192.168.10.69


[all:children]
rocketmq
[all:vars]
ansible_become=true
ansible_become_method=sudo
; ansible_ssh_user='csdn'
; ansible_ssh_pass="DSJzx@1234"
; ansible_become_pass="DSJzx@1234"
; ansible_ssh_port=22

3.2 批量启动服务#

Terminal window
# 修改ip
find rocketmq -type f -exec sed -i 's/192.168.10.67/192.168.10.67/g' {} +
find rocketmq -type f -exec sed -i 's/192.168.10.68/192.168.10.68/g' {} +
find rocketmq -type f -exec sed -i 's/192.168.10.69/192.168.10.69/g' {} +
手动替换plain_acl.yml网段192.168.10.*
修改jvm配置


# 创建目录并设置权限
ansible rocketmq -i hosts.ini -m file -a "path=/data/server/rocketmq/ state=directory owner=3000 group=3000 mode=0755" --become
# 拷贝文件
ansible 192.168.10.67 -i hosts.ini -m copy -a "src=rocketmq/broker-1/ dest=/data/server/rocketmq/" --become
ansible 192.168.10.68 -i hosts.ini -m copy -a "src=rocketmq/broker-2/ dest=/data/server/rocketmq/" --become
ansible 192.168.10.69 -i hosts.ini -m copy -a "src=rocketmq/broker-3/ dest=/data/server/rocketmq/" --become


# 启动docker-compose服务
ansible rocketmq -i hosts.ini -m shell -a "cd /data/server/rocketmq/ && docker-compose up -d" --become
ansible rocketmq -i hosts.ini -m shell -a "cd /data/server/rocketmq/ && docker-compose down" --become
ansible rocketmq -i hosts.ini -m file -a "path=/data/server/rocketmq/ state=directory owner=3000 group=3000 recurse=yes" --become  # 没有这部会导致没有权限启动不起来
ansible rocketmq -i hosts.ini -m shell -a "cd /data/server/rocketmq/ && docker-compose up -d" --become

4 查看服务#

Terminal window
# 查看 Controller 集群
docker exec -it rmqnamesrv-1 ./mqadmin getControllerMetaData -a 192.168.10.68:9877
#ControllerGroup  group1
#ControllerLeaderId  n0
#ControllerLeaderAddress  192.168.10.67:9877
#Peer:  n0:192.168.10.67:9877
#Peer:  n1:192.168.10.68:9877
#Peer:  n2:192.168.10.69:9877


# 查看 broker 状态
docker exec -it rmqnamesrv-1 ./mqadmin getSyncStateSet -a 192.168.10.67:9877 -b broker-a
#brokerName  broker-a
#MasterBrokerId  2
#MasterAddr  192.168.10.68:10911
#MasterEpoch  2
#SyncStateSetEpoch  4
#SyncStateSetNums  2
InSyncReplica:  ReplicaIdentity{brokerName='broker-a', brokerId=2, brokerAddress='192.168.10.68:10911', alive=true}
InSyncReplica:  ReplicaIdentity{brokerName='broker-a', brokerId=3, brokerAddress='192.168.10.67:10911', alive=true}
NotInSyncReplica:  ReplicaIdentity{brokerName='broker-a', brokerId=1, brokerAddress='192.168.10.67:10911', alive=false}

5 遇到的问题#

5.1 broker 节点同步不了，改成单网卡#

节点多网卡，broker 使用不能通信的网卡，如

Terminal window
# 日志报错
AutoSwitchHAClient connect to master 10.0.32.192:10912 failed
# 手动在容器里查看，正确的ip应该是192.168.10.67，但访问的是10.0.32.192
hostname -I
192.168.10.67 172.17.0.1 10.0.32.192 172.18.0.1

文章分享

如果这篇文章对你有帮助，欢迎分享给更多人！

Rocketmq controller高可用部署
https://ccops.cc/posts/caea4z38/
作者
蹦蹦站长
发布于
2024-06-09
许可协议
CC BY-NC-SA 4.0
atop 使用
MySQL 8.0 主从
相关文章 智能推荐
1
Redis 常用命令
📑数据库 使用 Redis 中经常使用到的命令
2
Redis 哨兵模式笔记
📑数据库 Redis 哨兵模式部署与高可用测试
3
Redis 备份与恢复
📑数据库 Redis 备份与恢复,保证数据完整
4
Redis 配置详解
📑数据库 Redis 非常全面的配置详解
5
Redis 集群模式笔记
📑数据库 Redis 集群模式部署与高可用测试
随机文章 随机推荐

评论区

分享你的想法，与大家交流讨论

Profile Image of the Author
蹦蹦站长
一个养兔子的运维
公告
感谢你的来访！希望在这里能找到对你有用的内容！
音乐
封面

音乐

暂未播放

0:00 0:00
暂无歌词
分类
标签
更多
站点统计
文章
81
分类
6
标签
45
总字数
88,456
运行时长
0 天
最后活动
0 天前
站点信息
构建平台
Local
博客版本
Firefly v6.12.1
文章许可
CC BY-NC-SA 4.0
© 2026 蹦蹦站长 版权所有 / RSS / Sitemap
由 Astro & Firefly 提供支持

文章目录