title: 3.3.Relabeling机制
order: 23

icon: lightbulb

Relabeling机制

前面讲了Prometheus的几种服务发现机制。通过服务发现的方式，可以在不重启Prometheus服务的情况下动态的发现需要监控的Target实例信息。

基于consul服务发现。如何加lable标签？Prometheus能够按照某些规则（比如标签）从服务发现注册中心返回的Target实例中有选择性的采集某些Exporter实例的监控数据。

接下来，我们将学习如何通过Prometheus强大的Relabel机制来实现以上这些具体的目标。

1、Relabeling机制

什么是Relabeling机制？

在采集样本数据之前，对Target实例的标签(Metadata)进行重写的机制在Prometheus被称为Relabeling。Relabeling最基本的应用场景就是基于Target实例中包含的metadata标签，动态的添加或者覆盖标签。

在采集任务设置中通过relabel_configs来添加自定义的Relabeling过程。如下图：

注：细心的同学可能发现了，在之前的课程中黑盒、域名的监控也配置过relabel_configs

因为要对Metadata标签进行重写，所以我们先来看下Metadata标签

默认Metadata标签

在Prometheus所有的Target实例中，都包含一些默认的Metadata标签信息。可以通过Prometheus UI的Targets页面中查看这些实例的Metadata标签的内容：

默认情况下，当Prometheus加载Target实例完成后，这些Target时候都会包含一些默认的标签：

__address__：当前Target实例的访问地址<host>:<port>，如果在重新标记期间未设置标签，则默认将__address__标签值赋值给instance。
__scheme__：采集目标地址的协议，HTTP或者HTTPS
__metrics_path__：采集目标服务访问地址的访问路径
__param_<name>：采集任务目标服务的中包含的请求参数
job：设置为job_name相应的抓取配置的值。
__scrape_interval__：scrape抓取数据时间间隔（秒）
__scrape_timeout__： scrape超时时间（秒）

一般来说，Target以__作为前置的标签是在系统内部使用的，因此这些标签不会被写入到样本数据中。不过这里有一些例外，例如，我们会发现所有通过Prometheus采集的样本数据中都会包含一个名为instance的标签，该标签的内容对应到Target实例的__address__。这里实际上是发生了一次标签的重写处理。

2、Consul动态发现的metadata标签

通过Consul动态发现的服务实例还会包含以下Metadata标签信息：

__meta_consul_address：consul地址
__meta_consul_dc：consul中服务所在的数据中心
__meta_consul_metadata：服务的metadata
__meta_consul_health: 健康检查
__meta_consul_node：服务所在consul节点的信息
__meta_consul_service_address：服务访问地址
__meta_consul_service_id：服务ID
__meta_consul_service_port：服务端口
__meta_consul_service：服务名称
__meta_consul_tags：服务包含的标签信息

如下图：

3、relabel_configs配置

source_labels：源标签，没有经过relabel处理之前的标签名字。

target_labels：通过relabel处理之后的标签名字。

separator：源标签的值的连接分隔符。默认是”;”

module：取源标签值散列的模数

regex：正则表达式，匹配源标签的值。默认是(.*)

replacement：通过分组替换后标签（target_label）对应的值。默认是$1

action：根据正则表达式匹配执行的动作。默认是replace

replace：替换标签值，根据regex正则匹配到原标签值，使用replacement来引用表达式匹配的分组
keep：满足regex正则条件的实例进行采集，把source_labels中没有匹配到regex正则内容的target实例丢掉，即只采集匹配成功的实例
drop：满足regex正则条件的实例不采集，把source_labels中没有匹配到regex正则内容的target实例丢掉，即只采集没有匹配成功的实例
hashmod：使用hashmod计算source_labels的hash值并进行对比，基于自定义的模数取模，以实现对目标进行分类、重新赋值等功能
labelmap：匹配regex所有标签名称，然后复制匹配标签的值进行分组，通过replacement分组引用($1,$2,…)替代
labeldrop：匹配regex所有标签名称，对匹配到的实例标签进行删除
labelkeep：匹配regex所有标签名称，对匹配到的实例标签进行保留

relabel_configs的配置我们已经知道了是对源数据重写的配置，接下来我们就来看具体案例

替换标签值

可以通过 replace 这个操作来完成，如果没有指定 action 字段，则默认就是 replace。

注：源数据需要加[]以便区分。目标数据不需要加。

案例1

    relabel_configs:
      - source_labels: [__meta_consul_service_address]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 192.168.11.62:9115

```
源数据：__meta_consul_service_address="192.168.11.62"
会变成：target="192.168.11.62",instance="192.168.11.62",endpoint="192.168.11.62:9115/prod"

案例2

从Node Exporter实例采集上来的样本数据如下所示：

node_load1{instance="test服务器", job="node_exporter"} 0.04

我们希望能有一个额外的标签dc，可以表示该样本所属的数据中心：

node_load1{instance="test服务器", job="node_exporter",dc="dc1"} 0.04

relabel_config配置：

    relabel_configs:
    - source_labels:  ["__meta_consul_dc"]
      target_label: "dc"

```
源数据：__meta_consul_dc="dc1"
会变成：dc="dc1"

保留或丢弃对象

通过keep 或 drop 这两个动作可以来完成

案例1

    relabel_configs:    
      - source_labels: [__meta_consul_tags]
        regex: .*exporter.*
        action: keep


源数据为：__meta_consul_tags=",exporter,"  会保留。如果action: drop就相反
源数据为：__meta_consul_tags=",domain,"    会丢弃，如果action: drop就相反

标签集映射

把源标签的值映射到一组新的标签中去，使用 labelmap 这个动作完成

案例1：

    relabel_configs:      
      - regex: __meta_consul_service_metadata_(.+)
        action: labelmap

源数据为：__meta_consul_service_metadata_job="test" 
会变成: job="test"

保留或删除标签

使用 labelkeep 和 labeldrop 这两个操作

案例1

    relabel_configs:
      - regex: job
        action: labeldrop


数据：probe_success{instance="192.168.11.61:9090", job="consul_blackbox_tcp"}   
变成：probe_success{instance="192.168.11.61:9090"}  

action: labelkeep 刚好相反

4、consul案例

Consul服务发现prometheus配置

 - job_name: 'consul_exporter'
    consul_sd_configs:
      - server: '192.168.11.61:8500'
        services: []
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*exporter.*
        action: keep
      - regex: __meta_consul_service_metadata_(.+)
        action: labelmap
      - source_labels:  ["__meta_consul_dc"]
        target_label: "dc"
#  Spring Boot 2.x 应用数据采集配置
  - job_name: 'consul_springboot_demo'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 5s
    consul_sd_configs:
      - server: '192.168.11.61:8500'
        services: []
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*springboot.*
        action: keep
      - regex: __meta_consul_service_metadata_(.+)
        action: labelmap

#http配置
  - job_name: "consul-blackbox_http"
    metrics_path: /probe
    params:
      module: [http_2xx]
    consul_sd_configs:
      - server: '192.168.11.61:8500'
        services: []
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*blackbox_http.*
        action: keep
      - regex: job
        action: labeldrop
      - regex: __meta_consul_service_metadata_(.+)
        action: labelmap
      - source_labels: [__meta_consul_service_address]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 192.168.11.62:9115
#tcp检查配置
  - job_name: "consul_blackbox_tcp"
    metrics_path: /probe
    params:
      module: [tcp_connect]
    consul_sd_configs:
      - server: '192.168.11.61:8500'
        services: []
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*blackbox_tcp.*
        action: keep
      - regex: __meta_consul_service_metadata_(.+)
        action: labelmap
      - source_labels: [__meta_consul_service_address]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 192.168.11.62:9115


#icmp检查配置
  - job_name: "consul_blackbox_icmp"
    metrics_path: /probe
    params:
      module: [icmp]
    consul_sd_configs:
      - server: '192.168.11.61:8500'
        services: []
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*blackbox_icmp.*
        action: keep
      - regex: __meta_consul_service_metadata_(.+)
        action: labelmap
      - source_labels: [__meta_consul_service_address]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 192.168.11.62:9115

#域名检测
  - job_name: consul_domain_exporter
    scrape_interval: 10s
    metrics_path: /probe
    consul_sd_configs:
      - server: '192.168.11.61:8500'
        services: []
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*domain.*
        action: keep
      - regex: __meta_consul_service_metadata_(.+)
        action: labelmap
      - source_labels: [__meta_consul_service_address]
        target_label: __param_target
      - target_label: __address__
        replacement: 192.168.11.62:9222
                                                                                                   135,25

curl调用consul api命令

上节课通过curl调用consul api添加服务的命令

#node_exporter
curl -X PUT -d '{"id": "node1","name": "node_exporter","address": "node_exporter","port": 9100,"tags": ["exporter"],"meta": {"job": "node_exporter","instance": "Prometheus服务
器"},"checks": [{"http": "http://192.168.11.61:9100/metrics", "interval": "5s"}]}'  http://localhost:8500/v1/agent/service/register


#nginx
curl -X PUT -d '{"id": "nginx1","name": "nginx_exporter","address": "192.168.11.62","port": 9113,"tags": ["exporter"],"meta": {"job": "nginx_exporter","instance": "test服务器","env":"test"},"checks": [{"http": "http://192.168.11.62:9113/metrics", "interval": "5s"}]}'  http://localhost:8500/v1/agent/service/register

#rabbitmq
curl -X PUT -d '{"id": "rabbitmq1","name": "rabbitmq_exporter","address": "192.168.11.62","port": 9419,"tags": ["exporter"],"meta": {"job": "rabbitmq_exporter","instance": "test服务器","env":"test"},"checks": [{"http": "http://192.168.11.62:9419/metrics", "interval": "5s"}]}'  http://localhost:8500/v1/agent/service/register

#redis
curl -X PUT -d '{"id": "redis1","name": "redis_exporter","address": "192.168.11.62","port": 9121,"tags": ["exporter"],"meta": {"job": "redis_exporter","instance": "test服务器","env":"test"},"checks": [{"http": "http://192.168.11.62:9121/metrics", "interval": "5s"}]}'  http://localhost:8500/v1/agent/service/register

#mongodb
curl -X PUT -d '{"id": "mongodb1","name": "mongodb_exporter","address": "192.168.11.62","port": 9216,"tags": ["exporter"],"meta": {"job": "mongodb_exporter","instance": "test服务器","env":"test"},"checks": [{"http": "http://192.168.11.62:9216/metrics", "interval": "5s"}]}'  http://localhost:8500/v1/agent/service/register

#mysql
curl -X PUT -d '{"id": "mysql1","name": "mysqld_exporter","address": "192.168.11.62","port": 9104,"tags": ["exporter"],"meta": {"job": "mysqld_exporter","instance": "test服务器","env":"test"},"checks": [{"http": "http://192.168.11.62:9104/metrics", "interval": "5s"}]}'  http://localhost:8500/v1/agent/service/register

#cadvisor
curl -X PUT -d '{"id": "cadvisor1","name": "cadvisor","address": "cadvisor","port": 8080,"tags": ["exporter"],"meta": {"job": "cadvisor","instance": "Prometheus服务器","env":"test"},"checks": [{"http": "http://192.168.11.61:8080/metrics", "interval": "5s"}]}'  http://localhost:8500/v1/agent/service/register
curl -X PUT -d '{"id": "cadvisor2","name": "cadvisor","address": "192.168.11.62","port": 8080,"tags": ["exporter"],"meta": {"job": "cadvisor","instance": "test服务器","env":"test"},"checks": [{"http": "http://192.168.11.62:8080/metrics", "interval": "5s"}]}'  http://localhost:8500/v1/agent/service/register

#springboot
curl -X PUT -d '{"id": "springboot1","name": "springboot","address": "192.168.11.62","port": 8081,"tags": ["springboot"],"meta": {"job": "springboot","instance": "test服务器","env":"test"},"checks": [{"http": "http://192.168.11.62:8081/actuator/prometheus", "interval": "5s"}]}'  http://localhost:8500/v1/agent/service/register


#process_exporter
curl -X PUT -d '{"id": "process1","name": "process_exporter","address": "192.168.11.62","port": 9256,"tags": ["exporter"],"meta": {"job": "process_exporter","instance": "test服务器","env":"test"},"checks": [{"http": "http://192.168.11.62:9256/metrics", "interval": "5s"}]}'  http://localhost:8500/v1/agent/service/register

#http
curl -X PUT -d '{"id": "http1","name": "blackbox_http","address": "https://www.jd.com","tags": ["blackbox_http"],"checks": [{"http": "http://192.168.11.62:9115", "interval": "5s"}]}'  http://localhost:8500/v1/agent/service/register

#tcp
curl -X PUT -d '{"id": "tcp1","name": "blackbox_tcp","address": "192.168.11.61:9090","tags": ["blackbox_tcp"],"checks": [{"http": "http://192.168.11.62:9115", "interval": "5s"}]}'  http://localhost:8500/v1/agent/service/register

#icmp
curl -X PUT -d '{"id": "icmp1","name": "blackbox_icmp","address": "192.168.11.62","tags": ["blackbox_icmp"],"checks": [{"http": "http://192.168.11.62:9115", "interval": "5s"}]}'  http://localhost:8500/v1/agent/service/register


#domin
curl -X PUT -d '{"id": "domain1","name": "domain_exporter","address": "baidu.com","tags": ["domain"],"checks": [{"http": "http://192.168.11.62:9222", "interval": "5s"}]}'  http://localhost:8500/v1/agent/service/register

目录CONTENT

3.3.Relabeling机制