Tracing and operational observability
Logs describe events, metrics summarize measurements and traces show a request’s path across instrumented components. None alone proves customer success. Start with CloudWatch lessons and the workload’s service objectives.
Instrumentation decisions
Propagate trace context across HTTP and supported messaging boundaries; distinguish parent/child relationships from asynchronous links. Use supported OpenTelemetry/ADOT collectors and exporters. X-Ray is a tracing backend/integration, not a substitute for application instrumentation.
Sample intentionally: reducing trace volume lowers cost but can hide rare failures. Never attach credentials, sensitive document text or unbounded customer identifiers to telemetry attributes. Metrics with high-cardinality dimensions can grow costs sharply.
Cross-account observability provides controlled visibility through OAM-supported links. Log centralization/copying is a different data movement/retention design. Establish source/sink permissions and data residency explicitly.
Correlation exercise
Prerequisites: the sandbox Lambda or App Runner lab, supported tracing setup, a telemetry budget and non-sensitive input.
- Send one successful and one deliberately failing request with a correlation ID.
- Locate request logs, latency/error metrics and the trace; show the failing dependency/span rather than only a generic 500.
- Compare event timestamp with ingestion time when data arrives late.
- Create an alarm tied to a user-impacting signal with a runbook and owner; trigger it safely.
- Acknowledge the alert, follow the runbook, correct the sandbox fault and verify recovery.
- Restore settings and delete dedicated alarms/dashboards/logs/tracing resources after preserving required evidence.
Expected evidence: one correlated request story and an operator action that restores the defined outcome. A dashboard with charts but no investigation path is not an operational handoff.
Capacity and cost
Distinguish publishing interval, storage resolution, roll-up retention and supported modern OTel metric types. Check actual account quotas rather than copying conflicting subscription limits. Price metric identities/cardinality, API usage, ingested/retained/scanned logs, traces and dashboards separately.
For EKS, reuse ADOT setup and the observability track.