aifromzero.dev

Operating GenAI in Production: Observability, Monitoring, and Troubleshooting (AWS GenAI Developer Pro, Module 14)

Module 14 of 16 20 min read D4 · 12%D5 · 11% Lab code ↗

This is Module 14 of AWS GenAI Pro Mastery, a free 16-module course that takes you from your first Amazon Bedrock call to AWS-certified.

Monday, 9 a.m. The CloudCart customer-success manager (CSM) forwards you three customer complaints from the weekend. Relay cited a documentation page that was deleted two sprints ago. Relay took 40 seconds to answer a simple shipping question. And the Bedrock bill for Saturday and Sunday doubled — with no traffic spike to explain it. You open your tooling to investigate, and there is nothing to open. No invocation logs, no dashboard, no alarm fired. The only signal you ever had was the invoice, and it arrives weeks late. Relay is deployed, guarded, cost-metered, and evaluated — last module’s regression gate even blocks bad deploys — but between deploys it runs blind. That is the trap with a GenAI app: it does not fail with a stack trace. It degrades in silence — answers get vaguer, retrieval drifts, costs creep. By the end of this module you will diagnose all three of those Monday-morning symptoms in minutes, with a dashboard and a runbook you proved against three real faults injected into a live Relay.

In this module

You’ll learn how to:

  • Instrument Relay end to end — Model Invocation Logs, CloudWatch metrics for tokens, latency, and quality, with X-Ray the right tool for tracing across service boundaries (the same boundary tracing you met in Module 11).
  • Build an operational and business dashboard ($/ticket, p95, escalation rate, eval score) with alarms and anomaly detection for cost and token bursts.
  • Monitor what classic ML does not: agent tool calling, multi-agent coordination, and vector-store health.
  • Apply FM-specific diagnostics — golden datasets in production, output diffing, reasoning-path tracing.
  • Troubleshoot the four families of GenAI failure with a symptom → diagnosis → remedy runbook.

You’ll build: Relay’s ops layer — invocation logs, a CloudWatch dashboard ($/ticket, p95, escalation rate, eval grounding), four alarms, and a troubleshooting runbook you battle-test against three injected faults.

Exam domains covered: D4 — Operational Efficiency and Optimization — 12% (this module covers Task 4.3, monitoring; Module 12 covered cost and performance). D5 — Testing, Validation, and Troubleshooting — 11% (Task 5.2, troubleshooting; Module 13 covered evaluation).

Prerequisites: Modules 1–13, an AWS_PROFILE configured for us-east-1.

Where you are in the build

You are 14 modules into a 16-module build. Relay — CloudCart’s customer-support agent — is built, governed, costed, and evaluated, but nothing watches it run:

  • ✅ M1–M13 — Relay triages, answers from a Knowledge Base with citations, acts through tools and an agent, is guarded and PII-redacted, ships behind an API with CI/CD, knows what every ticket costs, and gates its own deploys on a golden set.
  • 👉 M14 — you are here. Invocation logs, a relay-ops dashboard, four alarms, and a runbook proven on three injected faults.
  • ⬜ M15–M16 — the capstone and the exam.

Relay before this module: deployed and evaluated, but blind in production. Relay after: observed, alarmed, and diagnosable — the last construction module before the capstone assembles everything.

Observability for GenAI: what’s different

You already monitor infrastructure. Module 11 gave you Lambda durations and API Gateway 5xx counts — that tells you whether the plumbing works, nothing about whether Relay gave a good answer. The skill the exam calls holistic observability — instrumenting a system across operational metrics, FM-interaction tracing, and business impact — means watching three layers at once, and the FM layers are the two classic infra misses.

Picture them as a stack. The bottom layer is infrastructure: Lambda, API Gateway, SQS — already covered. The middle layer is FM invocations: which model ran, how many tokens it burned, how long the call took, what the prompt and response actually were. The top layer is business: $/ticket, escalation rate, and the eval grounding score from Module 13. A 200 OK with a green Lambda duration can sit directly on top of a hallucinated refund policy — the infra layer is healthy and the business layer is on fire. You cannot see that without the middle and top layers.

The middle layer comes almost free. Model Invocation Logs are Bedrock’s built-in capture of the full request, response, and metadata for every call, delivered to CloudWatch Logs or Amazon S3. Per the AWS documentation, logging is disabled by default and carries no Bedrock surcharge — you pay only CloudWatch Logs storage (cents). Turn it on once with put_model_invocation_logging_configuration, point it at a log group, and every Converse call becomes searchable in Logs Insights by token count, latency, and model ID — for a GenAI app, the single richest diagnostic you own.

⚠️ Common misconception: “If CloudWatch shows no errors, the GenAI app is healthy.” The worst GenAI failures are 200 OK — ungrounded answers, retrieval drift, hallucinated policy. Error codes catch crashes; they are blind to wrong-but-fluent answers, so you need quality signals — a golden set run in production, a grounding score, output diffing. And the converse trap: invocation logging is not observability. Raw logs with no metrics, no alarms, and no runbook wake nobody at 3 a.m. Logs are the data; observability is the data plus the signals plus the procedure.

One more distinction the exam loves: Model Invocation Logs are not CloudTrail. Invocation logs carry the content of each Bedrock call — prompt, completion, token counts. CloudTrail (Module 10) records the management API calls — who invoked bedrock:Converse, from what role, when — not what was said. You need both, for different jobs: CloudTrail for forensic audit, invocation logs for troubleshooting.

For the modern tracing path, AWS shipped CloudWatch generative AI observability (GA at re

2025) — native LLM and agent tracing that the Strands Agents SDK exports to via built-in OpenTelemetry (OTel/ADOT). This is the 2026 way to trace agent steps and tool calls. Do not hand-roll the pre-GA pattern of metric filters scraping prompt text out of log lines — that approach predates the GA product and the exam treats it as the wrong answer. (It is distinct from Bedrock AgentCore Observability, the runtime-side view of a deployed agent; one positioning sentence is enough — they are different products.) For the cross-service hop — API → SQS → agent — the right tool is AWS X-Ray, the same boundary-crossing tracing you met in Module 11, stitching the request into one service map. Here is how the pieces connect.

flowchart TD
    subgraph Sources
        L[Lambda + API Gateway<br/>infra metrics M11]
        A[Strands agent<br/>tool-call traces]
        B[Bedrock Model<br/>Invocation Logs]
        E[run_evals.py<br/>golden-set canary M13]
    end
    L -.boundary tracing.-> X[AWS X-Ray service map<br/>API → SQS → agent · renvoi M11]
    A -->|OTel / ADOT| G[CloudWatch generative<br/>AI observability]
    B --> CWL[CloudWatch Logs<br/>+ Logs Insights]
    E -->|EvalGrounding| CWM[CloudWatch metrics<br/>Relay/Ops namespace]
    L -->|EMF| CWM
    G --> D[relay-ops dashboard<br/>8 widgets]
    CWL --> D
    CWM --> D
    D --> AL[4 alarms]
    AL -->|SNS email| H[on-call human<br/>→ runbook]

The Relay dashboard: KPIs that matter for an FM app

A dashboard is not decoration. Every widget should answer a question an operator actually asks at 9 a.m. Relay’s relay-ops dashboard has exactly eight widgets, each tied to one question:

  1. FM tokens in / out — the FM signal classic infra omits; a token climb with flat traffic is your first cost-leak tell.
  2. $/ticket — the Module 12 cost_cents, now a live business metric.
  3. Worker p95 latency — the “40-second answer” symptom, read off the worker Lambda’s Duration p95; end-to-end latency is queue wait + worker Duration, so use X-Ray (the API→SQS→agent map) when you need the full customer-perceived path.
  4. Errors / throttling — the signal feeding FM API-integration troubleshooting (skill 5.2.2).
  5. Escalation rate — the deflection KPI; a spike means Relay is handing off more.
  6. Guardrail block rate — the Module 9 safety signal.
  7. Eval grounding — the golden-set canary score; the quality line the grounding alarm watches.
  8. Agent tool latency — the tool-calling observability widget (skill 4.3.4); the worker emits ToolLatencyMs only once it threads tool-call timing through run_relay, so the panel ships ready and fills in as that signal lands.

The blueprint names the KPIs you are obligated to watch for an FM app: token usage, hallucination rate, and response quality. You rarely get a true hallucination count, so you use a proxy — the judge’s grounding score from the golden set. The custom metrics flow two ways. Inside the worker Lambda you emit EMF (Embedded Metric Format) — a JSON log line CloudWatch turns into metrics for free, no extra API call on the customer’s ticket; outside Lambda the eval harness pushes its aggregate grounding with PutMetricData. All metric names, the Relay/Ops namespace, and the single Service=Relay dimension live in relay/config.py — deliberately low-cardinality, because CloudWatch bills per metric.

For alarms, the exam wants you to know when a static threshold is wrong. A p95-latency or throttling alarm is fine with a fixed line; a cost alarm is not — a growing app trips a static dollar threshold on every busy day. The right tool is anomaly detection. Per the CloudWatch docs, an anomaly-detection alarm “compares the metric’s value to the expected value based on the anomaly detection model” rather than to a static threshold — it learns a band of normal values and alarms only outside it. That is how you catch a Bedrock bill that doubled without false-alarming on Black Friday.

ConcernStatic thresholdAnomaly detection
p95 latency > 10 s✅ — a hard SLA lineoverkill
Throttles > 0 / 5 min✅ — any throttle is badoverkill
Daily total Bedrock cost❌ — a growing app trips it every busy day✅ — learns the band (the alarm sums daily CostCents), catches the unexpected doubling
Token burst❌ — varies with traffic✅ — flags spend that outpaces volume

Beyond the eight widgets, two FM-specific monitoring jobs round out the picture. Tool calling (skill 4.3.4): track call patterns per tool and baseline normal usage, so an agent that suddenly calls one tool ten times per ticket stands out — that pattern is invisible in Lambda Duration but obvious in tool-call counts and X-Ray spans. And vector-store ops (skill 4.3.5): monitor S3 Vectors query latency and Knowledge Base sync freshness. The course runs on Amazon S3 Vectors, which manages the index for you, so index-optimization-at-scale stays theory — you watch latency and freshness, you do not re-shard.

GenAI troubleshooting: symptom → diagnosis → remedy

This is the heart of the module. When something breaks, work in a fixed order: symptom → signal → diagnosis → remedy → verify — skip a step and you fix the wrong thing. Below is the runbook table — six failure classes, each mapped to the exact signal to read and the right remedy — the lookup you reach for when an alarm fires.

SymptomSignal to readDiagnosisRemedySkill
Errors/truncation only on long inputslargest_prompts Logs Insights — one input_tokens row dwarfs the restContext-window overflow — prompt got too big (content, not infra)Dynamic chunking / truncation of the oversized field5.2.1
ThrottlingException in burststhrottling_errors query; Errors/throttling widget spikesHitting a model quota, not a code bugConfirm M3 backoff engages; request a quota increase; move eval/backfill to Flex5.2.2
Triage JSON suddenly invalid after a prompt changetriage_ok falls; nothing changed in infraPrompt regressionDiff Prompt Management versions + output diffing; revert the version5.2.3
Citations point at obsolete / wrong docsEvalGrounding drops; citations cluster on one sourceRetrieval drift — a bad doc edit or re-syncRestore the doc, re-sync the KB, confirm embeddings5.2.4
Fluent answers, vague or wrong, zero errorsGolden-set grounding in prod (the canary)Prompt confusion / ungrounded generationTemplate testing; tighten the system prompt; re-ground5.2.5
Cost doubled, flat ticket volumecost_per_ticket query — tokens/ticket up, count flatAgent tool loop — re-calling a tool in a cycleCap the agent’s tool-iteration budget (M7 stop conditions); fix the tool4.3.4

Three FM-specific diagnostic frameworks (skill 4.3.6) carry that table. First, golden datasets in production: re-run the Module 13 golden set against the production environment on a schedule and it becomes a quality canary — its grounding is the EvalGrounding metric, and a drop below the floor trips an alarm. Second, output diffing: compare the golden set’s answers before and after a change; if grounding fell but infra is unchanged, the diff points straight at the prompt or KB doc that moved. Third, reasoning-path tracing: the agent’s tool-call spans (via OTel traces) show why it looped, not just that it was slow.

flowchart LR
    S[Symptom] --> C{Check the<br/>dashboard}
    C --> N[Narrow with logs<br/>+ X-Ray traces]
    N --> H[Hypothesis]
    H --> F[Apply remedy]
    F --> V{Verify with<br/>golden set}
    V -->|grounding back<br/>≥ 0.8| OK[Resolved]
    V -->|still low| H

Notice the verify step is almost never a green HTTP code — it is a golden-set re-run. A 200 OK was the symptom, so it cannot be the proof of a fix.

The runbook: writing it before you need it

An alarm that fires at 3 a.m. and points nowhere is worse than no alarm — it is scheduled panic. So every relay-ops alarm links to exactly one runbook entry, and every entry follows the same anatomy: symptom, severity, the precise signal to read, diagnosis steps, remedy, verification, and escalation. The discipline that separates a real runbook from a wiki page nobody opens is precision of signal. “Check the logs” fails under pressure; “run largest_prompts.logsinsights against /relay/bedrock/model-invocations and read the top input_tokens row” works, because it names the query, the log group, and the field. That is why Relay’s runbook references files in observability/queries/, not prose.

Severities keep the response proportional: SEV1 is customer-facing wrong answers at scale or runaway cost; SEV2 is degraded quality or a partial outage; SEV3 is a single contained ticket. The SNS topic relay-ops-alarms emails the on-call — SEV1 also loops in the CSM lead. No PagerDuty in the lab; an SNS email is the on-call signal, and that is a deliberate scope line, not a shortcut.

Build it: equip Relay with an ops layer, then break it on purpose

The lab does two things: one idempotent script turns on Relay’s full observability layer, then three injected faults let you prove the runbook. setup_observability.py prints the dashboard URL, and three inject_fault → diagnose → restore → re-run evals cycles return Relay to baseline. This lab cost me $0.08 on June 2026 prices. The complete code lives in module-14/; the excerpts below are the load-bearing parts.

Turn on invocation logging

One call enables it. The role and log group are created first; Bedrock validates the delivery role inline, which on a clean account fails once on IAM propagation — so the enable retries with a short backoff (it succeeded on attempt 3 on my cold run).

def enable_model_invocation_logging(bedrock, *, role_arn, sleep=time.sleep):
    config_kwargs = {"loggingConfig": {
        "cloudWatchConfig": {
            "logGroupName": config.RELAY_INVOCATION_LOG_GROUP,   # /relay/bedrock/model-invocations
            "roleArn": role_arn,
        },
        "textDataDeliveryEnabled": True,
        "imageDataDeliveryEnabled": True,
        "embeddingDataDeliveryEnabled": True,
    }}
    for attempt in range(1, _LOGGING_VALIDATION_RETRIES + 1):
        try:
            bedrock.put_model_invocation_logging_configuration(**config_kwargs)
            break
        except ClientError as err:
            # IAM eventual consistency: the just-created role has not propagated to
            # Bedrock's inline validation yet. Retry that one case — never silently swallow.
            code = err.response["Error"]["Code"]
            message = err.response["Error"].get("Message", "")
            is_propagation = (code == "ValidationException"
                              and "validate permissions" in message
                              and attempt < _LOGGING_VALIDATION_RETRIES)
            if not is_propagation:
                raise
            sleep(_LOGGING_VALIDATION_BACKOFF_SECONDS * attempt)

The log group gets a 14-day retention — prompts and responses are sensitive (link Module 10’s PII work) and voluminous, so you keep them short. Now explore three invocations in Logs Insights:

fields @timestamp, modelId,
       input.inputTokenCount as input_tokens,
       output.outputTokenCount as output_tokens
| sort @timestamp desc
| limit 20

Run live against the real log group, one 20-ticket canary run (~3 model calls per ticket, ~60 invocations) returned rows with their canonical inference-profile model IDs and token counts — here is one representative invocation per tier:

us.amazon.nova-micro-v1:0            1303 in    19 out   (triage)
us.amazon.nova-2-lite-v1:0           1451 in   238 out   (KB answer)
us.anthropic.claude-haiku-4-5-...    1295 in   304 out   (judge)

The reranker (cohere.rerank-v3-5:0) does not show up here: it rides inside the KB answer’s RetrieveAndGenerate call on the bedrock-agent-runtime data plane, not as a standalone bedrock-runtime invocation, so model-invocation logging never captures it as a separate row.

Emit custom metrics and build the dashboard

The worker emits one EMF line per ticket by addition — the existing log line carries the metrics. The dashboard body is a pure function the smoke test asserts offline, so eight widgets are guaranteed before any AWS call.

def build_emf(metrics, *, extra=None):
    directives = {
        "Namespace": config.RELAY_METRIC_NAMESPACE,          # "Relay/Ops"
        "Dimensions": [[config.METRIC_DIMENSION_SERVICE]],   # one low-cardinality dim
        "Metrics": [{"Name": m["name"], "Unit": m["unit"]} for m in metrics],
    }
    body = {"_aws": {"CloudWatchMetrics": [directives]},
            config.METRIC_DIMENSION_SERVICE: config.METRIC_SERVICE_VALUE}  # Service=Relay
    for m in metrics:
        body[m["name"]] = m["value"]           # CostCents, InputTokens, Escalated, ...
    return body

The four alarms — and the one shared constant

The cost alarm is anomaly detection (a band), the other three are static thresholds. The grounding alarm reuses the same 0.8 constant as the Module 13 deploy gate and the Module 9 per-answer escalation — defined once in config.py as GROUNDING_THRESHOLD, with EVAL_GROUNDING_FLOOR and ALARM_GROUNDING_THRESHOLD aliasing it. Gate ↔ alarm ↔ escalation coherence, proven by a test that asserts all three are equal.

def cost_anomaly_alarm_spec(topic_arn):
    return {
        "AlarmName": config.ALARM_COST_ANOMALY,
        "Metrics": [
            {"Id": "m1", "MetricStat": {"Metric": {...CostCents...},
                                        "Period": 86400, "Stat": "Sum"}, "ReturnData": True},
            {"Id": "ad1", "Expression": f"ANOMALY_DETECTION_BAND(m1, {band})",
             "ReturnData": True},   # a learned band — NOT a scalar dollar line
        ],
        "ThresholdMetricId": "ad1",
        "ComparisonOperator": "GreaterThanUpperThreshold",
        "AlarmActions": [topic_arn],
    }

Live, setup_observability.py enabled invocation logging, created two 14-day log groups, the relay-ops dashboard (eight widgets), an SNS email topic, and the four alarms: relay-ops-cost-anomaly (band, no scalar), relay-ops-grounding (LessThanThreshold 0.8), relay-ops-p95-latency (> 10000 ms), relay-ops-throttling (> 0). Re-running is a pure no-op — no duplicates.

The evals contract — consumed unchanged

Verification leans entirely on Module 13’s frozen evals contract, reproduced here field-for-field, never modified:

uv run python evals/run_evals.py --out evals/results/run-<name>.json
→ { run_name, config,
    scores: [ {id, triage_ok, grounding, coverage, citations} ],
    aggregate, cost_cents }
Gate: fail if aggregate.grounding < 0.8 or a regression > 5 pts vs baseline.

Break it on purpose — three faults

The runbook is only proven if you break Relay and fix it. inject_fault.py injects one of three visible, reversible faults — --list shows them, --restore undoes them, the mechanism commented in full (no opaque sabotage):

$ uv run python observability/inject_fault.py --list
Injectable faults (one at a time; each maps to a docs/runbook.md entry):
  context-overflow   runbook: Truncated answers / context-window overflow
  kb-corruption      runbook: Vague answers / grounding drop (retrieval drift)
  prompt-regression  runbook: Triage JSON suddenly wrong (prompt regression)

Each cycle is inject → follow the runbook → restore → run_evals --gate. Live, all three returned to the baseline gate at grounding 0.963: (a) a 197 KB context-overflow ticket; (b) a contradictory KB doc that flipped the refund policy; (c) the prompt-regression, which I let fail first to prove the gate — the degraded prompt scored grounding 0.400 — 0.400 below the 0.8 floor and a 0.563 regression versus the 0.963 baseline, so the gate FAILED on both checks — then restored to 0.963. That degraded run is exactly what the grounding alarm is built to catch.

Those gate runs all sit comfortably above the floor. A separate scheduled canary run — the same 20-ticket golden set, run later in the day as the production quality check, not the post-restore gate (real Nova-Micro triage + Nova-2-Lite KB answer + Cohere reranker + Haiku-4.5 judge, cost_cents=2.94, $0.0294) — drew grounding 0.662 on that draw. Judge stochasticity put it below 0.8, which correctly tripped the relay-ops-grounding alarm and emitted EvalGrounding=0.662 to the Relay/Ops namespace. A real alarm on a real number is the proof the layer works; you re-run the canary to confirm a dip was judge noise, not a regression.

Try it yourself: (1) add a fourth fault — corrupt the Module 12 semantic cache with a wrong answer — and write its runbook entry; (2) write an output-diffing script that diffs the golden set’s answers between two model tiers and flags the tickets that change, the systematic check before any model switch.

In production

Three things bite at scale. Log retention and cost. Beyond the short retention and PII handling already in place, sample at high volume — you do not need every payload, and full prompts cost real money to store. Metric cardinality. CloudWatch bills per metric, so never key metrics by ticket_id — that mints a metric per ticket and the bill explodes. Relay carries one bounded Service=Relay dimension on purpose; per-ticket detail belongs in the logs, not in a metric dimension. On-call and severities. SNS email is fine for a lab; a real team wires SEV1 to a pager and writes the runbook entry before the incident, not during it. Two operational habits pay for themselves: schedule the golden-set canary (EventBridge) so quality is checked between deploys, not only at deploy; and make output diffing mandatory before any model switch — diff the golden set across the old and new model and read the changed tickets before you flip the tier.

Exam corner

What the exam tests here. Task 4.3 is monitoring and observability for FM apps — invocation logging, custom dashboards, anomaly detection, tool-call and vector-store monitoring, and FM-specific diagnostics. Task 5.2 is troubleshooting — context-window overflow, FM API-integration errors, prompt regressions, and retrieval drift. The recurring theme: GenAI failures are usually 200 OK, so quality signals beat error codes, and the right diagnosis depends on reading the right signal first.

Quiz.

  1. Relay’s answers turn suddenly vague, with zero API errors and normal Lambda durations. What do you check first?

    • A. Lambda CPU and memory utilization.
    • B. API Gateway access logs for 4xx/5xx.
    • C. The golden-set grounding score run as a production canary.
    • D. The SQS queue depth.

  2. An app fails only when customers paste very long documents; short tickets are fine. What is the diagnosis and remedy?

    • A. A Lambda timeout — raise the function timeout.
    • B. Context-window overflow — inspect prompt size in the invocation logs, then apply dynamic chunking / truncation.
    • C. A throttling problem — request a quota increase.
    • D. A cold start — enable provisioned concurrency.

  3. The weekend Bedrock bill tripled with flat ticket volume. Which mechanism catches this earliest?

    • A. The monthly invoice.
    • B. An errors-and-throttling alarm.
    • C. Anomaly detection on a daily cost metric plus token-burst monitoring.
    • D. A static $/day threshold alarm.

  4. A multi-step agent keeps looping on one tool, inflating cost and latency, while Lambda Duration looks normal-ish. What instrumentation reveals it?

    • A. Raw Lambda CloudWatch logs.
    • B. Tool-call pattern tracking and X-Ray / agent traces showing the repeated tool span.
    • C. The API Gateway 5xx count.
    • D. A bigger answer model.

  5. After months of doc updates, citations drift increasingly off-topic. What is the diagnosis and remedy?

    • A. Prompt regression — revert the system prompt.
    • B. A guardrail misconfiguration — loosen the filter.
    • C. Retrieval drift — run embedding/drift diagnostics, restore the correct docs, and re-sync the Knowledge Base.
    • D. Model deprecation — switch models.

Answers. 1 — C. Vague-but-error-free answers are a quality failure, and the only quality signal you have is the golden set re-run in production; the grounding number bends down while every infra metric (A, B, D) stays green. The other three measure the healthy layer. 2 — B. The failure correlates with input length, which points at the prompt outgrowing the context window; the invocation logs show the oversized input_tokens, and the fix is dynamic chunking or truncation, not infra knobs (A, C, D are infra distractors for a content problem). 3 — C. Anomaly detection learns the cost band and flags spend that outpaces volume, catching the doubling in a day; the invoice (A) is weeks late, an errors alarm (B) watches the wrong metric, and a static threshold (D) false-alarms on every busy day. 4 — B. A tool loop is invisible in Duration but obvious in tool-call counts and trace spans — the agent-observability signal; raw logs (A) and 5xx counts (C) miss it, and a bigger model (D) costs more for the same loop. 5 — C. Slow degradation of citation relevance after repeated doc churn is retrieval drift; you diagnose with embedding/drift diagnostics and remedy by restoring docs and re-syncing the KB — the prompt (A) and guardrail (B) did not change, and the model (D) is not the cause.

Traps to avoid.

  • Watching only error codes. Typical GenAI failures answer 200 OK. Without quality metrics — grounding, a golden set in production — your dashboards look green while customers get wrong answers.
  • Confusing Model Invocation Logs with CloudTrail. Invocation logs carry the prompt/response content (turn them on); CloudTrail logs the management API call. The exam tests this distinction directly.
  • Treating a prompt regression as an infra bug. If nothing changed in infra, do not chase Lambda metrics — diff the prompt versions and the outputs (output diffing). The cause is the prompt, not the plumbing.

Key takeaways

  • Observe the invocations and the business, not just the infra — three layers; the FM and business layers are the two classic infra misses.
  • GenAI failures are silent degradations, not stack traces — the worst ones are 200 OK, so quality signals beat error codes.
  • Work in order: symptom → signal → diagnosis → remedy → verify — and verify with a golden-set re-run, not a green HTTP code.
  • The golden set is a production tool, not just a test asset — re-run it as a quality canary; its grounding is the alarm-watched metric.
  • A dashboard without alarms and a runbook is a poster — and an alarm without a runbook entry is scheduled panic; one constant (0.8) ties the gate, the alarm, and the escalation.
  • Use anomaly detection for cost, static thresholds for SLAs — a learned band catches the doubling a fixed line would miss or false-alarm on.
  • Model Invocation Logs are free on the Bedrock side — turn them on day one; you pay only CloudWatch Logs storage, in cents.

What’s next

Relay is built, guarded, evaluated, and now observed — every invocation logged, a dashboard, alarms that wake the right people, and a runbook proven on three real faults. Module 15 is the capstone: assemble everything, harden it (idempotence, timeouts, quotas), run the full 20-ticket costed demo against the deployed API, review it against the Well-Architected Generative AI Lens, and ship v1.0.

Want the full AIP-C01 question bank and the next module in your inbox? Subscribe here — it’s free, like everything in this series.

Links: Module 14 lab · ← Module 13 · Course index · Module 15 →

References