Issue 163
I hope everyone enjoyed the quarterly Best Of issue last week. This week brings us an interesting mix of Prometheus and OTel topics, network monitoring, and a big announcement from Grafana. Enjoy!
This issue is sponsored by:
See how companies like DoorDash are “no longer flying blind” with increased visibility and reliability from Chronosphere’s end-to-end solution. Chronosphere is the only observability platform that puts you back in control by taming rampant data growth and cloud-native complexity, delivering increased business confidence. Learn more here.
Articles & News on monitoring.love
Observability & Monitoring Community Slack
It’s been amazing to see the community continue to grow. We’d love to have you join us and share what you’ve been working on.
From The Community
Distributed Tracing with Hypertrace
How (and why) Razorpay engineers switched from Jaeger to Hypertrace for their distributed tracing needs.
Announcing Grafana Mimir, the most scalable open source TSDB in the world
Pretty big announcement from Grafana. Interesting to read that they plan to support more than just Prometheus metrics.
How we scaled our new Prometheus TSDB Grafana Mimir to 1 billion active series
Great read on some of the challenges they faced scaling Mimir to its current capacity. Personally, I’m less interested in their write path improvements and more curious how it compares to similar (complex) queries in something like ClickHouse or Victoria Metrics.
How to drop and delete metrics in Prometheus
As someone who’s had to do this in production, I wish this article existed three years ago. I really appreciate that the author covers both the how-to as well as any potential gotchas.
Increasing Postmark’s capacity: A parable of pipes
This is more about scaling than monitoring, but I always enjoy a good story with network graphs and bandwidth challenges.
TIL that you couldn’t previously alert on GCP logs directly (rather, you’d have to transform them into log-based metrics first). Still, nice to see this feature finally land, and this article covers the feature set and use cases.
How To Troubleshoot Network Connections On Your Kubernetes Workloads
Mizu looks like a pretty handy tool for visualizing network traffic and service activity on your Kubernetes clusters.
Whistle-stop tour of AWS DevOps — Part 3: Monitoring & Alerting
One part of a larger series on AWS products, this post is particularly interesting for it’s broad coverage of the monitoring and observability services in AWS.
OpenTelemetry in Kubernetes: Deploying your Collector and Metrics Backend
A straightforward example for setting up the OpenTelemetry Collector with Prometheus on your Kubernetes cluster.
Get Started with OpenTelemetry Python: A Practical Guide
Speaking of OpenTelemetry, here’s a handy guide for updating your Python apps to begin sending spans and collecting traces.
Azure Monitoring is no replacement for prometheus grafana
It’s natural to want to consolidate tooling, especially when it comes to certain cloud-provider services. This author cautions that you might be getting less (or more) than you bargained for.
Resiliency and Chaos Engineering — Part 5
Frankly I’m only including this one because I found it such a fascinating contrast (to the previous article) in the same week. 😂
Tools
“_A simple-yet-powerful API traffic viewer for Kubernetes enabling you to view all API communication between microservices to help your debug and troubleshoot regressions… [t]hink TCPDump and Wireshark re-invented for Kubernetes.
Events
Monitorama PDX 2022 - June 27-29 (Portland, OR)
Monitorama is returning to Portland, OR this summer. It looks like a return to form for one of our favorite events (ok, we might be biased). Hope to see you there!
Job Opportunities
DevOps Engineer at Amount Small Business (Remote)
Sr. DevOps Engineer at Barracuda (Remote)
Senior Site Reliability Engineer at Replicated (Remote)
Senior Platform Engineer at Replicated (Remote)
Customer Reliability Engineer at Replicated (Remote)
Negotiating your AWS contract? Let us help. At The Duckbill Group, we’re on your side and we see dozens of these a year–more than most AWS account managers! We’ve helped negotiate everything from $3mm contracts to $650mm contracts and a whole slew in between. Check out our AWS contract negotiation services. (SPONSORED)
See you next week!
– Jason (@obfuscurity) Monitoring Weekly Editor