Feature Flags, System Monitoring, and Developer Experience — Cheat Sheet¶
[!TIP] This cheat sheet is a reference for after you've learned the material — not a shortcut to avoid learning it. If you're looking something up here before truly understanding it, go back to the relevant module first. A cheat sheet only helps people who already know what they're looking for.
Quick Navigation¶
- PromQL Quick Reference
- OpenTelemetry SDK Setup Patterns
- Prometheus Metric Types
- SLO / Error Budget Formulas
- Feature Flag Patterns
- Sentry SDK Quick Setup
- PostHog SDK Quick Setup
- Decision Guide: Which Tool?
- Gotchas and Pitfalls
- DORA Metrics Reference
PromQL Quick Reference¶
The most commonly used PromQL operators and functions.
Instant Vector Selectors¶
# Select all time series for a metric
http_requests_total
# Filter by label (equality)
http_requests_total{status="200", job="api"}
# Filter by label (regex match)
http_requests_total{status=~"2.."}
# Filter by label (not equal)
http_requests_total{status!="500"}
# Filter by label (regex not match)
http_requests_total{status!~"5.."}
Rate and Increase Functions¶
# Per-second rate over a 5-minute window (use for counters)
rate(http_requests_total[5m])
# Total increase over a 1-hour window
increase(http_requests_total[1h])
# Derivative for gauges (not counters)
deriv(memory_usage_bytes[5m])
Aggregation Operators¶
# Sum across all label dimensions
sum(rate(http_requests_total[5m]))
# Sum grouped by a label
sum by (status) (rate(http_requests_total[5m]))
# Sum grouped by all labels except some
sum without (instance) (rate(http_requests_total[5m]))
# Average
avg(memory_usage_bytes)
# Maximum
max(memory_usage_bytes)
# Count the number of time series
count(http_requests_total)
Histogram Functions¶
# Calculate p99 latency from a histogram
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
# Calculate p50 latency
histogram_quantile(0.50, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
# Average latency from histogram
rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m])
SLO-Specific Queries¶
# Error rate (fraction of requests returning 5xx)
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
# Availability SLI (fraction of successful requests)
sum(rate(http_requests_total{status!~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
# Burn rate over 1 hour (how fast error budget is being consumed)
# Assumes 99.9% SLO (error rate budget = 0.001)
(
sum(rate(http_requests_total{status=~"5.."}[1h]))
/
sum(rate(http_requests_total[1h]))
) / 0.001
OpenTelemetry SDK Setup Patterns¶
Python: Basic Tracing Setup¶
# pip install opentelemetry-sdk opentelemetry-exporter-otlp
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
# Configure the tracer provider with service metadata
resource = Resource.create({"service.name": "my-service", "service.version": "1.0.0"})
provider = TracerProvider(resource=resource)
# Configure OTLP exporter (sends to OTel Collector or Jaeger)
exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
provider.add_span_processor(BatchSpanProcessor(exporter))
# Set the global tracer provider
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)
# Create spans in your code
with tracer.start_as_current_span("operation-name") as span:
span.set_attribute("user.id", "123") # Add custom attributes
span.set_attribute("http.method", "GET")
# ... do work here
Python: Auto-Instrumentation for Flask¶
# pip install opentelemetry-instrumentation-flask
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from flask import Flask
app = Flask(__name__)
FlaskInstrumentor().instrument_app(app) # Automatically traces all Flask routes
JavaScript/Node: Basic Tracing Setup¶
// npm install @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-http
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const sdk = new NodeSDK({
resource: new Resource({
[SemanticResourceAttributes.SERVICE_NAME]: 'my-node-service',
}),
traceExporter: new OTLPTraceExporter({
url: 'http://localhost:4318/v1/traces',
}),
});
sdk.start(); // Must call before importing other modules
Prometheus Metric Types¶
| Type | Use Case | Example |
|---|---|---|
| Counter | Monotonically increasing value (never decreases) | http_requests_total, errors_total |
| Gauge | Value that can go up and down | memory_usage_bytes, active_connections |
| Histogram | Distribution of observations; calculates percentiles | http_request_duration_seconds |
| Summary | Pre-calculated quantiles client-side | rpc_duration_seconds (use histogram instead in most cases) |
Python: Defining and Using Each Type¶
from prometheus_client import Counter, Gauge, Histogram, start_http_server
# Counter — use .inc() or .inc(amount)
REQUESTS = Counter('http_requests_total', 'Total HTTP requests',
['method', 'endpoint', 'status'])
REQUESTS.labels(method='GET', endpoint='/api/users', status='200').inc()
# Gauge — use .set(), .inc(), .dec()
ACTIVE_CONNECTIONS = Gauge('active_connections', 'Number of active connections')
ACTIVE_CONNECTIONS.set(42)
ACTIVE_CONNECTIONS.inc()
ACTIVE_CONNECTIONS.dec()
# Histogram — use .observe(value)
REQUEST_LATENCY = Histogram('http_request_duration_seconds',
'HTTP request latency',
['endpoint'],
buckets=[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0])
REQUEST_LATENCY.labels(endpoint='/api/users').observe(0.042)
# Start the metrics server (exposes /metrics endpoint)
start_http_server(8000)
SLO / Error Budget Formulas¶
| Formula | Description |
|---|---|
Error Budget = 1 - SLO Target |
Monthly error budget as a fraction |
Error Budget (minutes) = (1 - SLO%) × 30 days × 24h × 60min |
Monthly error budget in minutes |
Burn Rate = Actual Error Rate / Error Budget Rate |
How fast the budget is being consumed (1.0 = exactly on pace) |
Time to Exhaustion = Error Budget Remaining / Burn Rate |
When the budget will run out at current pace |
Error Budget Examples¶
| SLO Target | Monthly Error Budget (minutes) | Weekly Error Budget (minutes) |
|---|---|---|
| 99.9% | 43.2 min | 10.1 min |
| 99.5% | 216 min (3.6 hr) | 50.4 min |
| 99.0% | 432 min (7.2 hr) | 100.8 min |
| 99.95% | 21.6 min | 5.0 min |
Feature Flag Patterns¶
PostHog Python: Basic Flag Check¶
import posthog
posthog.api_key = 'your-api-key'
posthog.host = 'https://app.posthog.com' # or your self-hosted URL
# Boolean flag
if posthog.feature_enabled('new-checkout-flow', 'user-123'):
show_new_checkout()
else:
show_old_checkout()
# Multivariate flag (returns string variant or None)
variant = posthog.get_feature_flag('checkout-experiment', 'user-123')
if variant == 'variant-a':
show_variant_a()
elif variant == 'variant-b':
show_variant_b()
else:
show_control()
Homegrown Kill Switch (Simple Pattern)¶
import os
import json
from functools import wraps
# Read flags from environment variable or file
def get_flags():
flags_json = os.getenv('FEATURE_FLAGS', '{}')
return json.loads(flags_json)
def feature_flag(flag_name, default=False):
"""Decorator that wraps a function with a feature flag check."""
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
if get_flags().get(flag_name, default):
return func(*args, **kwargs)
# Return None or a fallback when flag is off
return wrapper
return decorator
# Usage: FEATURE_FLAGS='{"new_algo": true}' python app.py
@feature_flag('new_algo')
def run_new_algorithm(data):
return process_with_new_algo(data)
Sentry SDK Quick Setup¶
# pip install sentry-sdk
import sentry_sdk
from sentry_sdk.integrations.flask import FlaskIntegration
sentry_sdk.init(
dsn="https://your-dsn@o123456.ingest.sentry.io/7890",
integrations=[FlaskIntegration()], # Auto-capture Flask errors
traces_sample_rate=0.1, # 10% of transactions for performance
environment="production",
release="my-app@1.2.3", # Link errors to releases
)
# Manual error capture
try:
risky_operation()
except ValueError as e:
sentry_sdk.capture_exception(e)
# Add context to errors
with sentry_sdk.push_scope() as scope:
scope.set_tag("payment.provider", "stripe")
scope.set_user({"id": "user-123", "email": "user@example.com"})
sentry_sdk.capture_message("Payment failed", level="error")
PostHog SDK Quick Setup¶
# pip install posthog
import posthog
posthog.api_key = 'phc_your_api_key'
posthog.host = 'https://app.posthog.com'
# Track an event
posthog.capture(
distinct_id='user-123',
event='checkout_completed',
properties={
'order_value': 49.99,
'items_count': 3,
'payment_method': 'card',
}
)
# Identify a user (set person properties)
posthog.identify(
distinct_id='user-123',
properties={
'email': 'user@example.com',
'plan': 'pro',
'created_at': '2025-01-15',
}
)
# Feature flag check with automatic event tracking
if posthog.feature_enabled('new-dashboard', 'user-123'):
render_new_dashboard()
Decision Guide: Which Tool?¶
What are you trying to answer?
│
├── "Is my service up and how is it performing?"
│ └── Prometheus + Grafana (metrics + SLO dashboard)
│
├── "Why did this error happen and who was affected?"
│ └── Sentry (error tracking + context)
│
├── "Where in my distributed system is the slowness?"
│ └── OpenTelemetry + Jaeger/Tempo (distributed traces)
│
├── "Are users completing the checkout flow?"
│ └── PostHog (product analytics + funnels)
│
├── "Is this new feature working as expected in production?"
│ └── PostHog feature flags + PostHog analytics, or LaunchDarkly + DataDog
│
├── "Full APM for a team that wants one platform?"
│ └── DataDog (metrics + traces + logs + SLOs in one place)
│
└── "How fast is my team shipping and how stable is production?"
└── DORA metrics dashboard (deploy frequency + lead time + MTTR + change failure rate)
Gotchas and Pitfalls¶
[!WARNING] High-cardinality labels crash Prometheus Never use
user_id,request_id, or any unbounded identifier as a Prometheus label.Wrong:
http_requests_total{user_id="12345"}— creates millions of time series Right:http_requests_total{endpoint="/api/users", method="GET", status="200"}— bounded set of values[!WARNING] Counters always go up — never use them for values that can decrease If a counter resets (e.g., process restart), Prometheus detects the reset automatically and handles it correctly via
rate()— but usingincrease()on a gauge-like value will give wrong results.Wrong: Using a Counter for
active_connections(which goes down when connections close) Right: Use a Gauge for values that decrease; Counter only for cumulative totals[!WARNING] Feature flags without cleanup become permanent complexity Every feature flag that is never removed adds a branch to your code that must be tested, understood, and maintained. Set a target removal date when creating every flag.
Symptom: A codebase with 50+ flags, half of which are always
truein production and no one knows what they control.
Other things to watch out for:
- Sampling distributed traces — tracing every request at high load is expensive; use head-based sampling for consistent traces or tail-based sampling for selective capture of errors
- Alert noise kills on-call morale — every alert should be actionable; if you can't write a runbook for it, it shouldn't fire
- SLOs too strict or too loose — 99.99% is probably too strict for most services; 95% is probably too loose; start with 99.5% and adjust based on error budget consumption
- Sentry DSN in client-side code — PostHog API keys should be treated carefully; Sentry DSNs for frontend SDKs are designed to be public, but backend DSNs are secret
DORA Metrics Reference¶
| Metric | Definition | Elite Target | Poor Target |
|---|---|---|---|
| Deployment Frequency | How often code is deployed to production | On-demand (multiple/day) | Less than once/month |
| Lead Time for Changes | Time from commit to production | Less than 1 hour | More than 6 months |
| MTTR | Time to restore service after an incident | Less than 1 hour | More than 1 week |
| Change Failure Rate | % of deployments causing a production failure | 0–15% | 46–60% |
Source: DORA State of DevOps Report — dora.dev
Module Cross-References¶
| If you need to recall... | See module |
|---|---|
| PromQL basics and metric types | [[modules/02_metrics-and-prometheus]] |
| Grafana dashboard design | [[modules/03_dashboards-and-grafana]] |
| Sentry SDK setup | [[modules/04_error-tracking-with-sentry]] |
| OpenTelemetry SDK patterns | [[modules/05_distributed-tracing]] |
| PostHog flag SDK | [[modules/06_posthog-product-analytics]] |
| DataDog APM configuration | [[modules/07_datadog-apm]] |
| Flag types and targeting | [[modules/08_feature-flags-deep-dive]] |
| Canary and blue-green | [[modules/09_progressive-delivery]] |
| Runbook and post-mortem format | [[modules/10_incident-response]] |
| DORA + SPACE in detail | [[modules/11_developer-experience]] |
Last updated: 2026-06-09