Prometheus Pushgateway和blackbox_exporter黑盒

 2018年3月31日 16:54   Nick王   运维  自动化    0 评论   2340 浏览 

blackbox_exporter版本: 0.12

AA

Pushgateway

什么时候使用Pushgateway

Pushgateway是一个中介服务,它允许在你收集不到的目标节点上,使用主动推送的方式推送metric

仅推荐在一些特定的场景来使用Pushgateway。如果使用Pushgateway来代替Prometheus的pull会有以下几个缺陷:

  • 当使用单个的Pushgateway来监控多个实例的时候,这个Pushgateway会有单点故障的隐患,并且还有性能问题。

  • 丧失了Prometheus的监控检查的功能。

  • Pushgateway不会丢弃已经推送过的序列,并且永远会暴露给Prometheus。除非通过Pushgateway的API手动维护这些时间序列。

推送Metric

有些时候你需要监控抓取不到的目标节点。Prometheus Pushgateway允许你推送时间序列到Prometheus可以抓取的节点上。

默认情况下Pushgateway不会持久化数据到硬盘上,需要配置参数-persistence.file string来进行Metric数据的持久化。

首先,Pushgateway要可以被Prometheus收集,并且在收集配置中应该设置honor_labels: true

另外正常情况下,客户端是通过被动的,Prometheus会主动来收集数据。为了支持Pushgateway,客户端必须支持主动将Metric推送到Pushgateway。

推送Metric非常的简单,使用HTTP API就可以完成,所以没有提供任何命令行工具。

项目地址: https://github.com/prometheus/pushgateway


Prometheus使用blackbox进行网络监控

blackbox_exporter允许通过HTTP、HTTPS、DNS、TCP和ICMP对Endpoint进行黑盒探测。

同样的,黑盒可以通过配置文件和命令行参数进行配置。也可以在运行时状态下重新加载配置文件。

官方配置文件参考: https://github.com/prometheus/blackbox_exporter/blob/master/example.yml

配置文件如下:

# cat blackbox.yml
modules:
  http_2xx_kubeapi:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2"]
      valid_status_codes: []
      method: GET
      headers:
        Authorization: Bearer xxxx
      tls_config:
        ca_file: "/root/prometheus/prometheus-bearer-ca.crt"
      preferred_ip_protocol: "ip4"
  tcp_connect:
    prober: tcp
    timeout: 5s
  icmp:
    prober: icmp
    timeout: 5s
    icmp:
      preferred_ip_protocol: "ip4"
      source_ip_address: "10.6.28.37"

启动blackbox:

# blackbox_exporter --config.file="blackbox.yml" --web.listen-address=":9115" --log.level=debug > blackbox.log 2>&1 &

注意: 测试的时候模块名称要和配置文件中的一一对应。

测试http:

# curl "http://10.6.28.37:9115/probe?target=https://10.6.28.167:6443/api&module=http_2xx_kubeapi&debug=true"
Logs for the probe:
ts=2018-03-27T04:06:01.05768491Z caller=main.go:116 module=http_2xx_kubeapi target=https://10.6.28.167:6443/api level=info msg="Beginning probe" probe=http timeout_seconds=4.5
ts=2018-03-27T04:06:01.057948104Z caller=utils.go:42 module=http_2xx_kubeapi target=https://10.6.28.167:6443/api level=info msg="Resolving target address" preferred_ip_protocol=ip4
ts=2018-03-27T04:06:01.058045851Z caller=utils.go:65 module=http_2xx_kubeapi target=https://10.6.28.167:6443/api level=info msg="Resolved target address" ip=10.6.28.167
ts=2018-03-27T04:06:01.058353555Z caller=http.go:282 module=http_2xx_kubeapi target=https://10.6.28.167:6443/api level=info msg="Making HTTP request" url=https://10.6.28.167:6443/api host=10.6.28.167:6443
ts=2018-03-27T04:06:01.066115076Z caller=http.go:300 module=http_2xx_kubeapi target=https://10.6.28.167:6443/api level=info msg="Received HTTP response" status_code=200
ts=2018-03-27T04:06:01.066168013Z caller=http.go:354 module=http_2xx_kubeapi target=https://10.6.28.167:6443/api level=info msg="Response timings for roundtrip" roundtrip=0 start=2018-03-27T12:06:01.058446474+08:00 dnsDone=2018-03-27T12:06:01.058446474+08:00 connectDone=2018-03-27T12:06:01.059387579+08:00 gotConn=2018-03-27T12:06:01.065263325+08:00 responseStart=2018-03-27T12:06:01.066073577+08:00 end=2018-03-27T12:06:01.06611014+08:00
ts=2018-03-27T04:06:01.066319555Z caller=main.go:127 module=http_2xx_kubeapi target=https://10.6.28.167:6443/api level=info msg="Probe succeeded" duration_seconds=0.008515612



Metrics that would have been returned:
# HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds
# TYPE probe_dns_lookup_time_seconds gauge
probe_dns_lookup_time_seconds 2.9931e-05
# HELP probe_duration_seconds Returns how long the probe took to complete in seconds
# TYPE probe_duration_seconds gauge
probe_duration_seconds 0.008515612
……省略……

测试ICMP:

# curl "http://10.6.28.37:9115/probe?target=10.6.28.167&module=icmp&debug=true"
Logs for the probe:
ts=2018-03-27T04:11:08.968039173Z caller=main.go:116 module=icmp target=10.6.28.167 level=info msg="Beginning probe" probe=icmp timeout_seconds=4.5
ts=2018-03-27T04:11:08.968143194Z caller=utils.go:42 module=icmp target=10.6.28.167 level=info msg="Resolving target address" preferred_ip_protocol=ip4
ts=2018-03-27T04:11:08.968171044Z caller=utils.go:65 module=icmp target=10.6.28.167 level=info msg="Resolved target address" ip=10.6.28.167
ts=2018-03-27T04:11:08.968191071Z caller=icmp.go:68 module=icmp target=10.6.28.167 level=info msg="Using source address" srcIP=10.6.28.37
ts=2018-03-27T04:11:08.968213088Z caller=icmp.go:71 module=icmp target=10.6.28.167 level=info msg="Creating socket"
ts=2018-03-27T04:11:08.968272842Z caller=icmp.go:117 module=icmp target=10.6.28.167 level=info msg="Creating ICMP packet" seq=2 id=20441
ts=2018-03-27T04:11:08.968316642Z caller=icmp.go:129 module=icmp target=10.6.28.167 level=info msg="Writing out packet"
ts=2018-03-27T04:11:08.968442971Z caller=icmp.go:157 module=icmp target=10.6.28.167 level=info msg="Waiting for reply packets"
……省略……

测试TCP:

# curl "http://10.6.28.37:9115/probe?target=10.6.28.167:6443&module=tcp_connect&debug=true"

Prometheus配置

- job_name: 'blackbox'
  metrics_path: /probe
  params:
    module: [http_2xx_kubeapi]  # 和blackbox中配置的模块名要对应blackbox配置文件中定义的
  static_configs:
    - targets:
      - https://10.6.28.167:6443/api/   # 检测目标
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: 127.0.0.1:9115  # blackbox 的地址

在Kubernetes中部署blackbox

# cat monitor-blackbox.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: prometheus-blackbox
  name: prometheus-blackbox
  namespace: monitor
data:
  blackbox.yml: |-
    modules:
      http_2xx:
        prober: http
        timeout: 10s
        http:
          valid_http_versions: ["HTTP/1.1", "HTTP/2"]
          valid_status_codes: []
          method: GET
          preferred_ip_protocol: "ip4"
      tcp_connect:
        prober: tcp
        timeout: 10s
      icmp:
        prober: icmp
        timeout: 10s
        icmp:
          preferred_ip_protocol: "ip4"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-blackbox
  namespace: monitor
spec:
  selector:
    matchLabels:
      app: prometheus-blackbox
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus-blackbox
    spec:
      restartPolicy: Always
      containers:
      - name: prometheus-blackbox
        image: prom/blackbox-exporter:v0.12.0
        imagePullPolicy: IfNotPresent
        ports:
        - name: blackbox-port
          containerPort: 9115
        readinessProbe:
          tcpSocket:
            port: 9115
          initialDelaySeconds: 5
          timeoutSeconds: 5
        resources:
          requests:
            memory: 50Mi
            cpu: 100m
          limits:
            memory: 60Mi
            cpu: 200m
        volumeMounts:
        - name: config
          mountPath: /etc/blackbox_exporter
        args:
        - --config.file=/etc/blackbox_exporter/blackbox.yml
        - --log.level=debug
        - --web.listen-address=:9115
      volumes:
      - name: config
        configMap:
          name: prometheus-blackbox
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: prometheus-blackbox
  name: prometheus-blackbox
  namespace: monitor
  annotations:
    prometheus.io/scrape: 'true'
spec:
  type: NodePort
  selector:
    app: prometheus-blackbox
  ports:
  - name: blackbox
    port: 9115
    targetPort: 9115
    nodePort: 30915
    protocol: TCP

官方推荐的使用Kubernetes自动发现监控Kubernetes SVC的Prometheus配置:

- job_name: 'kubernetes-service-blackbox'
  metrics_path: /probe
  params:
    module: [tcp_connect]
  kubernetes_sd_configs:
  - role: service
    api_server: https://10.6.28.167:6443
    tls_config:
      ca_file: /root/prometheus/prometheus-bearer-ca.crt
    bearer_token_file: /root/prometheus/prometheus-bearer-token
  relabel_configs:
  - source_labels: [__address__]
    target_label: __param_target
  - target_label: __address__
    replacement: 10.6.28.167:30915
  - source_labels: [__param_target]
    target_label: instance
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    target_label: kubernetes_name
  - source_labels: [__meta_kubernetes_service_port_protocol]
    target_label: kubernetes_protocol




如无特殊说明,文章均为本站原创,转载请注明出处