Skip to content
Snippets Groups Projects

custom-metrics-generator

Pluggable cron-like prometheus textform metrics generator for node-exporter textfile collector.

Use-case

Each node may need to export custom runtime metrics like:

  • S.M.A.R.T. disk statistics
  • puppet state
  • container restarts
  • node role

Those metrics are generated by jobs (scripts) and text-file metrics representation is then fed to Prometheus node-exporter which is able to periodically read text-file metrics using its textfile collector.

This approach allows add export custom metrics to node-exporter. Although every custom metrics may have its dedicated prometheus exporter, Prometheus node-exporter textfile collector approach helps to reduce and maintain reasonable number of prometheus targets.

How to run custom-metrics-generator

docker run -it --privileged \
  -v /var/run/docker.sock:/var/run/docker.sock
  -v /etc/node-exporter/node-role.prom:/etc/node-exporter/node-role.prom:ro
  -v /opt/puppetlabs/puppet/cache/state/last_run_summary.yaml:/var/lib/puppet/state/last_run_summary.yaml:ro \
  -v /var/run/node-exporter:/opt/custom-metrics-generator/out-metrics-dir \
  -e 'CMG_EXEC_MODULES=smartmon dockermon puppetmon noderolemon' \
  registry.gitlab.ics.muni.cz:443/cloud/custom-metrics-generator:latest

Debugging

  • list containers and exec to custom-metrics-generator
    • docker ps
  • get custom-metrics-generator logs
    • docker logs custom-metrics-generator
  • exec to custom-metrics-generator
    • docker exec -it custom-metrics-generator bash
  • list and inspect crontab rules
    • crontab -l
  • list and inspect exported metrics
    • ls -la /opt/custom-metrics-generator/out-metrics-dir
  • list runner execution env. dump (for instance puppetmon), seek env. variable STAGE_NAME value to identify what happened
    • more /tmp/cmg-puppetmon-trace.log
  • try to run the job manually in trace mode (for instance puppetmon)
    • bash -x /opt/custom-metrics-generator/src/metric-generator-exec.sh puppetmon /opt/custom-metrics-generator/out-metrics-dir/puppetmon.prom /opt/custom-metrics-generator/out-metrics-dir/custom-metrics-generator_puppetmon.prom