I hope everyone enjoyed the quarterly Best Of issue last week. This week brings us an interesting mix of Prometheus and OTel topics, network monitoring, and a big announcement from Grafana. Enjoy!

This issue is sponsored by:

Chronosphere logo

See how companies like DoorDash are “no longer flying blind” with increased visibility and reliability from Chronosphere’s end-to-end solution. Chronosphere is the only observability platform that puts you back in control by taming rampant data growth and cloud-native complexity, delivering increased business confidence. Learn more here.

Articles & News on monitoring.love

Observability & Monitoring Community Slack

It’s been amazing to see the community continue to grow. We’d love to have you join us and share what you’ve been working on.

From The Community

Distributed Tracing with Hypertrace

How (and why) Razorpay engineers switched from Jaeger to Hypertrace for their distributed tracing needs.

Announcing Grafana Mimir, the most scalable open source TSDB in the world

Pretty big announcement from Grafana. Interesting to read that they plan to support more than just Prometheus metrics.

How we scaled our new Prometheus TSDB Grafana Mimir to 1 billion active series

Great read on some of the challenges they faced scaling Mimir to its current capacity. Personally, I’m less interested in their write path improvements and more curious how it compares to similar (complex) queries in something like ClickHouse or Victoria Metrics.

How to drop and delete metrics in Prometheus

As someone who’s had to do this in production, I wish this article existed three years ago. I really appreciate that the author covers both the how-to as well as any potential gotchas.

Increasing Postmark’s capacity: A parable of pipes

This is more about scaling than monitoring, but I always enjoy a good story with network graphs and bandwidth challenges.

Log-based Alerting in GCP

TIL that you couldn’t previously alert on GCP logs directly (rather, you’d have to transform them into log-based metrics first). Still, nice to see this feature finally land, and this article covers the feature set and use cases.

How To Troubleshoot Network Connections On Your Kubernetes Workloads

Mizu looks like a pretty handy tool for visualizing network traffic and service activity on your Kubernetes clusters.

Whistle-stop tour of AWS DevOps — Part 3: Monitoring & Alerting

One part of a larger series on AWS products, this post is particularly interesting for it’s broad coverage of the monitoring and observability services in AWS.

OpenTelemetry in Kubernetes: Deploying your Collector and Metrics Backend

A straightforward example for setting up the OpenTelemetry Collector with Prometheus on your Kubernetes cluster.

Get Started with OpenTelemetry Python: A Practical Guide

Speaking of OpenTelemetry, here’s a handy guide for updating your Python apps to begin sending spans and collecting traces.

Azure Monitoring is no replacement for prometheus grafana

It’s natural to want to consolidate tooling, especially when it comes to certain cloud-provider services. This author cautions that you might be getting less (or more) than you bargained for.

Resiliency and Chaos Engineering — Part 5

Frankly I’m only including this one because I found it such a fascinating contrast (to the previous article) in the same week. 😂



“_A simple-yet-powerful API traffic viewer for Kubernetes enabling you to view all API communication between microservices to help your debug and troubleshoot regressions… [t]hink TCPDump and Wireshark re-invented for Kubernetes.


Monitorama PDX 2022 - June 27-29 (Portland, OR)

Monitorama is returning to Portland, OR this summer. It looks like a return to form for one of our favorite events (ok, we might be biased). Hope to see you there!

Job Opportunities

DevOps Engineer at Amount Small Business (Remote)

Sr. DevOps Engineer at Barracuda (Remote)

Senior Site Reliability Engineer at Replicated (Remote)

Senior Platform Engineer at Replicated (Remote)

Customer Reliability Engineer at Replicated (Remote)

Negotiating your AWS contract? Let us help. At The Duckbill Group, we’re on your side and we see dozens of these a year–more than most AWS account managers! We’ve helped negotiate everything from $3mm contracts to $650mm contracts and a whole slew in between. Check out our AWS contract negotiation services. (SPONSORED)

See you next week!

– Jason (@obfuscurity) Monitoring Weekly Editor