Evaluating monitoring solutions; Prometheus, Thanos, Mimir, Victoria Metrics

Quick summary comparing monitoring options

Hierarchical federation

One (federated) prometheus scraping (child) prometheus for metrics. Child prometheus has low retention whereas the federated prometheus stores long term data on disk

Pros

• Simple setup
• Easy to maintain
• Quick to identify and remediate issues as small architecture

Cons

• Federated prometheus becomes a bottleneck
• Complicated to implement high availability
• Vertical scaling to alleviate resource issues
• No option to store long term metrics in object storage for retrieval

Resource consumption

The following dashboards showcase resource consumption based on 1million time series ingested into child prometheus at a rate of 5.5k samples per second

Federated prometheus

Global prometheus that scrapes child prometheus for metric aggregation. The child prometheus has 1 million time series

Overview