Prometheus监控RocketMQ - 云原生基站

1 安装 agent#

指标说明

名字	暴露信息
rocketmq_producer_tps	每秒每个主题生成的消息数
rocketmq_producer_message_size	主题每秒生成的消息的大小（字节）
rocketmq_producer_offset	主题生成消息的进度
rocketmq_consumer_tps	消费者群体每秒消耗的消息数
rocketmq_consumer_message_size	消费者群体每秒消耗的消息大小（字节）
rocketmq_consumer_offset	消费群体消费信息的进展
rocketmq_group_get_latency	消费者延迟对一个队列的某个主题
rocketmq_group_get_latency_by_storetime	消费群体的消费延迟时间
rocketmq_message_accumulation	消费者抵消滞后程度
rocketmq_client_consume_fail_msg_count	消耗的消息数量在一小时内失败
rocketmq_client_consume_fail_msg_tps	消耗的消息数量每秒失败
rocketmq_client_consume_ok_msg_tps	每秒消耗成功的消息数
rocketmq_client_consume_rt	消耗每条消息的平均时间
rocketmq_client_consumer_pull_rt	拉每个消息的平均时间
rocketmq_client_consumer_pull_tps	客户端每秒提取的消息数

1.1 下载插件#

Apache RocketMQ Prometheus Exporter

1.2 修改配置#

vim src/main/resources/application.yml

1
rocketmq:
2
  config:
3
    webTelemetryPath: /metrics
4
    rocketmqVersion: 4_8_0
5
    namesrvAddr: 127.0.0.1:9876  #地址
6
    enableCollect: true
7
    enableACL: false # if >=4.4.0
8
    accessKey:  # if >=4.4.0
9
    secretKey:  # if >=4.4.0
10
#检测频率
11
task:
12
  count: 5 # num of scheduled-tasks
13
  collectTopicOffset:
14
    cron: 30 0/1 * * * ?
15
  collectConsumerOffset:
16
    cron: 30 0/1 * * * ?
17
  collectBrokerStatsTopic:
18
    cron: 30 0/1 * * * ?
19
  collectBrokerStats:
20
    cron: 30 0/1 * * * ?
21
  collectBrokerRuntimeStats:
22
    cron: 30 0/1 * * * ?

1.3 打包#

mvn clean install

1.4 启动#

nohup /opt/java8/bin/java -jar rocketmq-exporter-0.0.2-SNAPSHOT.jar &

2 收集数据与监控#

kubectl edit configmap prometheus-server -n ops

2.1 配置 prometheus 收集任务#

1
scrape_configs:
2
- job_name: rocketmq
3
  static_configs:
4
  - targets:
5
    - 172.16.3.17:5557 #rocketmq-exporte地址

2.2 配置监控策略#

1
  rules: |-
2
    groups:
3
    - name: rocketmq
4
      rules:
5
      - alert: RocketMQ Exporter is Down
6
        expr: up{job="rocketmq"} == 0
7
        for: 20s
8
        labels:
9
          severity: '灾难'
10
        annotations:
11
          summary: RocketMQ {{ $labels.instance }} is down
12
      - alert: RocketMQ 存在消息积压
13
        expr: (sum(irate(rocketmq_producer_offset[1m])) by (topic)  - on(topic) group_right sum(irate(rocketmq_consumer_offset[1m])) by (group,topic)) > 5
14
        for: 5m
15
        labels:
16
          severity: '警告'
17
        annotations:
18
          summary: RocketMQ (group={{ $labels.group }} topic={{ $labels.topic }})积压数 = {{ .Value }}
19
      - alert: GroupGetLatencyByStoretime 消费组的消费延时时间过高
20
        expr: rocketmq_group_get_latency_by_storetime/1000  > 5 and rate(rocketmq_group_get_latency_by_storetime[5m]) >0
21
        for: 3m
22
        labels:
23
          severity: 警告
24
        annotations:
25
          description: 'consumer {{$labels.group}} on {{$labels.broker}}, {{$labels.topic}} consume time lag behind message store time
26
            and (behind value is {{$value}}).'
27
          summary: 消费组的消费延时时间过高
28
      - alert: RocketMQClusterProduceHigh 集群TPS > 20
29
        expr: sum(rocketmq_producer_tps) by (cluster) >= 20
30
        for: 3m
31
        labels:
32
          severity: 警告
33
        annotations:
34
          description: '{{$labels.cluster}} Sending tps too high. now TPS = {{ .Value }}'
35
          summary: cluster send tps too high

2.3 配置 grafana#

导入模板 10477

3 配置告警#

3.1 下载插件#

prometheus-webhook-dingtalk

3.2 钉钉插件配置#

cat config.yml

1
templates:
2
 \- /opt/prometheus-webhook-dingtalk/template.tmpl  #模板位置
3
targets:
4
 webhook1:
5
  url: https://oapi.dingtalk.com/robot/send?access_token=ac5f4916af10804b1aeffe9f5f45574a9af8e7cdd8436bcf1dc2448a85116fba  #钉钉url
6
  secret: SEC6f3e3e736f33a8f8692e3f4f9e1c0828ac41fc514c99c5215fd21659bxxxx   #钉钉加密字段
7
  mention:
8
   mobiles: ['1810133xxxx', '1871712xxxx']

cat template.tmpl

1
{{ define "ding.link.title" }}{{ template "legacy.title" . }}{{ end }}
2
{{ define "ding.link.content" }}
3
{{ if gt (len .Alerts.Firing) 0 -}}
4
告警列表:
5
{{ template "__text_alert_list" .Alerts.Firing }}
6
{{- end }}
7
{{ if gt (len .Alerts.Resolved) 0 -}}
8
恢复列表:
9
{{ template "__text_resolve_list" .Alerts.Resolved }}
10
{{- end }}
11
{{- end }}

3.3 启动 prometheus-webhook-dingtalk#

/opt/prometheus-webhook-dingtalk/prometheus-webhook-dingtalk --log.level=info > dingding.log 2>&1 &

3.3.1 查看钉钉插件接口#

3.4 prometheus 告警配置#

kubectl edit configmap prometheus-alertmanager -n ops

1
  alertmanager.yml: |-
2
    global:
3
      resolve_timeout: 5m
4
    route:
5
      receiver: webhook
6
      group_wait: 30s
7
      group_interval: 1m
8
      repeat_interval: 4h
9
      group_by: [alertname]
10
      routes:
11
      - receiver: webhook
12
        group_wait: 10s
13
    receivers:
14
    - name: webhook
15
      webhook_configs:
16
      - url: http://172.16.3.1x:8060/dingtalk/webhook1/send
17
        send_resolved: true