custom-metrics-generator
Pluggable cron-like prometheus textform metrics generator for node-exporter textfile collector.
Use-case
Each node may need to export custom runtime metrics like:
- S.M.A.R.T. disk statistics
- puppet state
- container restarts
- node role
Those metrics are generated by jobs (scripts) and text-file metrics representation is then fed to Prometheus node-exporter which is able to periodically read text-file metrics using its textfile collector.
This approach allows add export custom metrics to node-exporter. Although every custom metrics may have its dedicated prometheus exporter, Prometheus node-exporter textfile collector approach helps to reduce and maintain reasonable number of prometheus targets.
How to run custom-metrics-generator
docker run -it --privileged \
-v /var/run/docker.sock:/var/run/docker.sock
-v /etc/node-exporter/node-role.prom:/etc/node-exporter/node-role.prom:ro
-v /opt/puppetlabs/puppet/cache/state/last_run_summary.yaml:/var/lib/puppet/state/last_run_summary.yaml:ro \
-v /var/run/node-exporter:/opt/custom-metrics-generator/out-metrics-dir \
-e 'CMG_EXEC_MODULES=smartmon dockermon puppetmon noderolemon' \
registry.gitlab.ics.muni.cz:443/cloud/custom-metrics-generator:latest
Debugging
- list containers and exec to custom-metrics-generator
docker ps
- get custom-metrics-generator logs
docker logs custom-metrics-generator
- exec to custom-metrics-generator
docker exec -it custom-metrics-generator bash
- list and inspect crontab rules
crontab -l
- list and inspect exported metrics
ls -la /opt/custom-metrics-generator/out-metrics-dir
- list runner execution env. dump (for instance puppetmon), seek env. variable STAGE_NAME value to identify what happened
more /tmp/cmg-puppetmon-trace.log
- try to run the job manually in trace mode (for instance puppetmon)
bash -x /opt/custom-metrics-generator/src/metric-generator-exec.sh puppetmon /opt/custom-metrics-generator/out-metrics-dir/puppetmon.prom /opt/custom-metrics-generator/out-metrics-dir/custom-metrics-generator_puppetmon.prom