ARCHIVED from builddistributedsystem.com on 2026-04-28 — URL: https://builddistributedsystem.com/tracks/tracer/tasks/task-23-1-5-distributed-tracing
TASK

Implementation

Individual span operations give you the building blocks. End-to-end tracing connects them: every service creates and propagates spans, logs are correlated with trace IDs, and you get complete visibility across the entire call chain without manual wiring.

Implement a node that supports full distributed tracing infrastructure:

// Auto-instrument three services, then handle a request
{ "type": "instrument_service", "msg_id": 1,
  "services": ["web","api","db"], "auto": true }
{ "type": "http_request", "path": "/api/users/123" }
-> { "type": "trace_complete",
    "trace_id": "<uuid>",
    "services": ["web","api","db"], "span_count": 3 }

// Attach trace_id to logs for correlation
{ "type": "log", "msg_id": 3,
  "message": "Processing request", "trace_id": "abc123" }
{ "type": "search_logs", "msg_id": 4, "trace_id": "abc123" }
-> { "type": "search_results", "in_reply_to": 4,
    "logs": [{"service":"service","message":"Processing request",
               "trace_id":"abc123"}] }

Sample Test Cases

Instrument multiple servicesTimeout: 5000ms
Input
{"src":"instrumentor","dest":"services","body":{"type":"instrument_service","msg_id":1,"services":["web","api","db"],"auto":true}}
{"src":"user","dest":"web","body":{"type":"http_request","path":"/api/users/123"}}
Expected Output
{"type": "trace_complete", "trace_id": ".*", "services": ["web", "api", "db"], "span_count": 3}
Manual instrumentationTimeout: 5000ms
Input
{
  "src": "developer",
  "dest": "code",
  "body": {
    "type": "manual_instrument",
    "msg_id": 1,
    "code": "async function process() { const span = tracer.startSpan('process'); try { await work(); span.end(); } catch(e) { span.recordException(e); throw; } }"
  }
}
Expected Output
{"type": "instrumentation_ok", "in_reply_to": 1, "spans_created": 1}

Hints

Hint 1
Auto-instrumentation: wrap each service with a tracing interceptor that creates spans automatically
Hint 2
Manual: tracer.startSpan(name) -> do work -> span.end() (or span.recordException(e) on error)
Hint 3
Log-trace correlation: attach trace_id to every log entry; index logs by trace_id
Hint 4
Service mesh sidecar: injects tracing at the network layer without application code changes
Hint 5
span_count = number of services instrumented in the call chain
OVERVIEW

Theoretical Hub

Concept overview coming soon

Key Concepts

auto-instrumentationmanual instrumentationlog-trace correlationservice mesh tracing
main.py
python
Implement End-to-End Distributed Tracing System - The Tracer | Build Distributed Systems