Exercises — Module 01: Introduction to Observability¶

Work through exercises in order — they're designed to build on each other. Attempt each problem genuinely before looking at the solution. Seeing the solution first might feel like progress, but it isn't.

Instructions¶

1. Attempt first. Spend at least the estimated time on each problem before checking hints or solutions.
2. Write your work. Don't just read mentally — actually write or type your attempt.
3. Check your answer against the acceptance criteria, not just the solution code.
4. Score yourself honestly in the Scoring Log at the bottom.
5. If you're stuck after a genuine effort, use the hints one at a time — not all at once.

Difficulty Legend¶

Exercise 1: Classify the Signal [🟢 Easy] [1 pt]¶

Context¶

Given a list of statements about what data a team is collecting, classify each as primarily a metric, log, or trace.

Task¶

Classify each of the following as "metric", "log", or "trace" and briefly explain why (1 sentence each):

1. http_requests_total{status="200"} = 15432 collected every 15 seconds
2. 2026-06-09T14:05:22Z ERROR user 456 checkout failed: invalid payment method
3. A record showing that user request A went through services: API Gateway (120ms) → Cart Service (45ms) → Payment Service (850ms) with a shared request ID
4. memory_usage_bytes = 524288000 at a specific timestamp
5. {"event": "user_signup", "user_id": "123", "plan": "pro", "timestamp": "..."}

Requirements¶

• All 5 items classified correctly
• Each classification includes a 1-sentence justification
• At least one example of each signal type appears in your answers

Hints¶

Expected Output / Acceptance Criteria¶

1. Metric — it's a numeric measurement (counter) collected over time
2. Log — it describes a specific event at a specific timestamp with context
3. Trace — it links operations across multiple services with timing information
4. Metric — it's a numeric measurement (gauge) at a point in time
5. Log — it's a structured event record for a single occurrence

Solution¶

Exercise 2: Calculate Error Budgets [🟢 Easy] [1 pt]¶

Context¶

Your team is defining SLOs for three services. You need to calculate the monthly error budget for each.

Task¶

Calculate the monthly error budget in minutes for each of the following SLO targets:

1. Payment service: 99.9% availability SLO, 30-day window
2. Email notification service: 99.5% availability SLO, 30-day window
3. Internal analytics dashboard: 99.0% availability SLO, 30-day window

Additionally, answer: If the payment service has already had 40 minutes of downtime this month, what percentage of its error budget has been consumed?

Requirements¶

• Correct calculation for all 3 services
• Budget consumption percentage for the follow-up question
• Show your calculation steps

Hints¶

Expected Output / Acceptance Criteria¶

1. Payment service (99.9%):  43.2 minutes/month
2. Email service (99.5%):   216.0 minutes/month
3. Analytics (99.0%):       432.0 minutes/month

Budget consumed: 40 / 43.2 = 92.6% — nearly exhausted

Solution¶

def error_budget_minutes(slo_percent: float, window_days: int = 30) -> float:
    window_minutes = window_days * 24 * 60  # 43,200 for 30 days
    return (1 - slo_percent / 100) * window_minutes

# Payment service: 99.9%
payment_budget = error_budget_minutes(99.9)
print(f"Payment service: {payment_budget:.1f} minutes")   # 43.2

# Email service: 99.5%
email_budget = error_budget_minutes(99.5)
print(f"Email service: {email_budget:.1f} minutes")        # 216.0

# Analytics: 99.0%
analytics_budget = error_budget_minutes(99.0)
print(f"Analytics: {analytics_budget:.1f} minutes")        # 432.0

# Budget consumption
consumed = 40 / payment_budget * 100
print(f"Budget consumed: {consumed:.1f}%")                  # 92.6%

Exercise 3: Identify the Right Pillar [🟡 Medium] [2 pts]¶

Context¶

A teammate is debugging a production incident. For each scenario, identify which observability pillar (metrics, logs, traces) would be most useful, explain why, and describe what specific signal you would look for.

Task¶

For each incident scenario below, answer: - Which pillar is most useful for this stage of the investigation? - What specific query/search/filter would you use? - What information gap remains that requires another pillar?

Scenario A: An alert fires saying "API error rate is 3.5%, above the 1% SLO threshold."

Scenario B: After seeing the alert, you know errors are happening but need to understand what type of errors they are and what's causing them.

Scenario C: You've found the error message is "database connection timeout" in the logs. You need to understand whether the timeout is in your service's database call or in a downstream service's database call.

Requirements¶

• All 3 scenarios answered with correct pillar identification
• Each answer includes a concrete, specific query or search approach
• Information gap identified for each scenario

Hints¶

Expected Output / Acceptance Criteria¶

Your answers should demonstrate understanding of when each pillar is the appropriate tool and what it can/cannot tell you.

Solution¶

Exercise 4: Design a Feature Flag [🟡 Medium] [2 pts]¶

Context¶

You are building a feature flag system for a new checkout experience. You need to decide how to implement the flag, what types of targeting you need, and how to handle the rollout safely.

Task¶

Design a feature flag for rolling out a new payment form to production users. Your design must answer:

1. Flag type: Should this be a boolean flag or a percentage-based rollout? Why?
2. Targeting rules: Who should get the flag first? Design at least 3 targeting rules in priority order (e.g., "employees first, then beta users, then 5% of general users").
3. Rollout plan: Write out the rollout stages as a numbered list with the criteria for advancing to the next stage.
4. Kill switch: What condition would cause you to immediately set the flag to 0% (no users)?
5. Cleanup: When is it safe to remove the flag entirely and delete the old code path?

Requirements¶

• All 5 questions answered
• Targeting rules are in a sensible priority order
• Rollout plan includes observable success criteria for each stage
• Kill switch condition is specific (not just "if there are bugs")
• Cleanup criteria are specific and verifiable

Hints¶

Expected Output / Acceptance Criteria¶

A thoughtful design document (bullet points acceptable) that demonstrates understanding of safe progressive rollout principles.

Solution¶

Exercise 5: Instrument a Minimal Flask App [🔴 Hard] [3 pts]¶

Context¶

You have a minimal Flask API. Your task is to add Prometheus metrics instrumentation that would allow you to detect a checkout SLO breach. This is a multi-step practical exercise requiring you to write runnable code.

Task¶

Given this stub Flask application, add instrumentation to make it observable:

# starter: app.py
from flask import Flask, jsonify, request
import random
import time

app = Flask(__name__)

@app.route('/api/checkout', methods=['POST'])
def checkout():
    # Simulates variable latency
    time.sleep(random.uniform(0.01, 0.5))
    # Simulates ~5% error rate
    if random.random() < 0.05:
        return jsonify({"error": "payment_failed"}), 500
    return jsonify({"order_id": random.randint(1000, 9999)}), 200

@app.route('/api/products', methods=['GET'])
def products():
    time.sleep(random.uniform(0.005, 0.05))
    return jsonify({"products": ["a", "b", "c"]}), 200

if __name__ == '__main__':
    app.run(port=5000)

Add: 1. A Counter for total HTTP requests with method, endpoint, and status labels 2. A Histogram for request latency with endpoint label and appropriate buckets 3. A /metrics endpoint for Prometheus scraping 4. A decorator or middleware that automatically instruments all routes (not per-route manual calls) 5. Write a PromQL query that would fire an alert when the checkout error rate exceeds 1%

Requirements¶

• Counter defined with correct metric name, help text, and labels
• Histogram defined with appropriate buckets for web request latency
• All routes automatically instrumented (no manual inc() calls in each route)
• /metrics endpoint returns Prometheus-format output
• PromQL alert query correctly calculates error rate for /api/checkout

Hints¶

# Structure hint:
@app.before_request
def start_timer():
    request._start_time = time.time()  # Store start time

@app.after_request
def record_metrics(response):
    latency = time.time() - request._start_time
    # Use response.status_code to get the status
    # Use request.method, request.path for labels
    return response

Expected Output / Acceptance Criteria¶

• Running curl localhost:5000/metrics returns Prometheus-format output with both metrics
• After sending 100 requests to /api/checkout, the metrics show ~5 errors
• The PromQL alert query, when evaluated, returns > 0.01 when error rate is above 1%

Solution¶

# app.py — instrumented Flask application

from flask import Flask, jsonify, request
from prometheus_client import (
    Counter, Histogram, generate_latest, CONTENT_TYPE_LATEST
)
import random
import time

app = Flask(__name__)

# --- Prometheus Metric Definitions ---

# Counter: tracks total number of HTTP requests
# Labels: method (GET/POST), endpoint (/api/checkout), status (200/500)
REQUEST_COUNT = Counter(
    'http_requests_total',
    'Total number of HTTP requests processed',
    ['method', 'endpoint', 'status']
)

# Histogram: tracks the distribution of request latency
# Buckets: chosen to be meaningful for web request latency (in seconds)
# Rule of thumb: include your SLO target latency as a bucket boundary
REQUEST_LATENCY = Histogram(
    'http_request_duration_seconds',
    'HTTP request duration in seconds',
    ['endpoint'],
    buckets=[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0]
)

# --- Automatic Instrumentation via Flask Hooks ---

@app.before_request
def start_timer():
    """Record the request start time for latency calculation."""
    request._start_time = time.time()

@app.after_request
def record_metrics(response):
    """Instrument every request with latency and count metrics."""
    if request.path == '/metrics':
        # Don't instrument the metrics endpoint itself
        return response

    latency = time.time() - request._start_time

    # Record latency for this endpoint
    REQUEST_LATENCY.labels(endpoint=request.path).observe(latency)

    # Increment request counter with method, endpoint, and status code
    REQUEST_COUNT.labels(
        method=request.method,
        endpoint=request.path,
        status=str(response.status_code)
    ).inc()

    return response

# --- Application Routes ---

@app.route('/api/checkout', methods=['POST'])
def checkout():
    time.sleep(random.uniform(0.01, 0.5))
    if random.random() < 0.05:
        return jsonify({"error": "payment_failed"}), 500
    return jsonify({"order_id": random.randint(1000, 9999)}), 200

@app.route('/api/products', methods=['GET'])
def products():
    time.sleep(random.uniform(0.005, 0.05))
    return jsonify({"products": ["a", "b", "c"]}), 200

# --- Metrics Endpoint ---

@app.route('/metrics')
def metrics():
    """Expose Prometheus metrics for scraping."""
    return generate_latest(), 200, {'Content-Type': CONTENT_TYPE_LATEST}

if __name__ == '__main__':
    app.run(port=5000)

# This expression returns the error rate for /api/checkout
# It fires (returns > 0.01) when error rate exceeds 1%
(
  sum(rate(http_requests_total{endpoint="/api/checkout", status=~"5.."}[5m]))
  /
  sum(rate(http_requests_total{endpoint="/api/checkout"}[5m]))
) > 0.01

Scoring Log¶

Record your performance honestly. Include the date and whether you used hints.

Passing threshold: 6/9 (67%). Aim for 8/9 (89%) before taking the test.