Taking a whack at custom Prometheus alerting
After many years of being on-call under my belt, I never thought I’d say I have a favorite alerting method. But that changed after watching one of Justin Garrison’s videos which had an excellent depiction of how Linux’s Out-of-Memory Killer works. I was no stranger to the OOM Killer visiting my Kubernetes clusters, so this gave me a dumb idea for a (perhaps) fun alerting mechanism: the OOM Bonker. ...