The Token Economy: Cost and Performance Optimization for GenAI (AWS GenAI Developer Pro, Module 12)

This is Module 12 of AWS GenAI Pro Mastery, a free 16-module course that takes you from your first Amazon Bedrock call to AWS-certified.

It’s the last day of the month, and CloudCart’s Amazon Bedrock bill just doubled. Nobody can say why. Was it the flash-sale traffic spike? Agents looping three times on a stuck tool call? Or the triage step quietly paying for a smart model to answer “where is my order?” — a question a model a tenth the price would nail? You go looking for the per-ticket cost and find the truth: Relay has shipped to production (Module 11), it is safe and governed, and it has never once measured what a single ticket costs. The cost_cents field has sat on TicketRecord at 0.0 since Module 7. “It feels expensive” is not a diagnosis, and you cannot cut a cost you cannot see. By the end of this module, every ticket knows its real dollar cost, and four measured levers — prompt caching, a semantic cache, batch inference, and the Flex tier — cut it, with a before/after table to prove it.

In this module

You’ll learn how to:

• Measure the real cost of a ticket before optimizing — token estimation and tracking, and a real cost_cents per ticket as a product metric.
• Reduce tokens without hurting quality — prompt compression, context pruning, response limiting, and the router re-read as a cost decision.
• Cache intelligently — tell prompt caching, semantic caching, and deterministic request hashing apart, and implement all three.
• Batch latency-tolerant work — batch inference (−50%) for eval backfills, and the throughput/capacity tradeoff.
• Optimize latency — latency-optimized models, parallelization, pre-computation, retrieval tuning, the Flex tier (−50%), and a p95 measured before and after.
• Decide between levers on numbers — cost vs latency vs quality — and produce a defensible before/after table, exactly the shape Domain 4 questions ask for.

You’ll build: Relay’s cost instrumentation plus four optimization levers — a real cost_cents per ticket, prompt caching on Converse, a semantic cache on DynamoDB + Amazon Titan embeddings, batch inference for eval backfills, and a before/after table for cost and p95.

Exam domains covered: D4 — Operational Efficiency and Optimization for GenAI Applications — 12% (this module covers Tasks 4.1 and 4.2; Module 14 covers 4.3, monitoring).

Prerequisites: Modules 1–11, an AWS_PROFILE configured for us-east-1.

Where you are in the build

You are 12 modules into a 16-module build. Relay — CloudCart’s customer-support agent — is deployed, safe, and governed:

• ✅ M1–M11 — Relay triages, answers from a Knowledge Base with citations, acts through tools and an agent, is guarded and PII-redacted, and ships behind an API with CI/CD.
• 👉 M12 — you are here. Instrument the real cost_cents per ticket and cut it on four fronts.
• ⬜ M13–M16 — evaluation, observability, the capstone, and the exam.

Relay before this module: deployed and reachable, but its cost is a black box — cost_cents has been 0.0 since Module 7. Relay after this module: every ticket carries its measured dollar cost, prompt caching and a semantic cache are live on the interactive path, batch and Flex are wired for eval backfills, and a before/after table for cost and p95 is committed.

Measuring before optimizing: the cost of a ticket

The first rule of the token economy is the one teams skip: you cannot cut what you cannot see. Before any optimization, you instrument. Token efficiency — getting the same answer for fewer billed tokens — starts with token estimation and tracking: counting the input and output tokens of every model call and multiplying by the price of the tier that served it.

Amazon Bedrock hands you the raw numbers. Every Converse response carries a usage block — inputTokens, outputTokens, and (once caching is on) cacheReadInputTokens. You never guess a cost; you read it from the API and multiply by a price map. In Relay that map has lived in relay/config.py since Module 3, one entry per tier:

# relay/config.py — per-tier price, $ per 1K tokens (as of June 2026; re-verify)
PRICE_PER_1K = {
    "fast":  {"input": 0.000035, "output": 0.00014},   # Nova Micro
    "smart": {"input": 0.00030,  "output": 0.00250},   # Nova 2 Lite
}

def estimate_cost(tier, input_tokens, output_tokens):
    p = PRICE_PER_1K.get(tier, PRICE_PER_1K["fast"])
    return input_tokens / 1000 * p["input"] + output_tokens / 1000 * p["output"]

The subtle part is that a ticket is not one call. A single CloudCart ticket fans out into several model invocations: triage on the fast tier, a retrieval-grounded answer on the smart tier, and however many tool loops the agent runs. The real cost of a ticket is the sum over all of them. So the cost_cents you report is a per-ticket roll-up, not a per-call line lost in a log.

That decomposition is the mental model the exam tests, and it is the first diagram:

flowchart LR
    T["Incoming ticket"] --> A["triage — fast tier"]
    T --> B["search_kb retrieval"]
    T --> C["answer generation — smart tier"]
    T --> D["agent loops — tool calls"]
    A --> S(("sum"))
    B --> S
    C --> S
    D --> S
    S --> R["TicketRecord.cost_cents"]

One honest caveat on that diagram: cost_cents sums the FM converse() calls — triage, the answer generation, and the agent’s tool loops. The search_kb node draws the trigger for retrieval, but the vector Retrieve query itself is billed on a separate Bedrock line and is not metered into cost_cents here; only the model invocations it feeds are.

TicketRecord.cost_cents is a float that has existed since Module 7 — Module 12 finally populates it, by addition, without touching the frozen schema. (In Module 14 this number lands on a CloudWatch dashboard; here we only measure it.)

Token efficiency and tiered usage

Once you can see the cost, you attack the tokens. Four levers, in roughly the order you should reach for them.

Prompt compression is the cheapest win: tighten instructions, trim few-shot examples to the smallest set that still works, drop boilerplate the model does not need. Context pruning is the RAG-specific cousin — do not inject eight retrieved chunks when three answer the question. (You built retrieval and top-k in Modules 4–5; here you re-read them through a cost lens.) Response limiting caps generation with maxTokens sized to the task, so a one-line answer cannot run to a paragraph on the model’s whim.

Then comes the biggest lever of all, one you already built: the router. Tiered FM usage based on query complexity means routing each ticket to the cheapest model that can do the job — fast for triage and simple lookups, smart for grounded answers and agent reasoning. The Module 3 router was always a cost decision dressed as an architecture decision. Here is the opinion, with a number behind it: roughly 80% of CloudCart’s tickets are fast-tier work, so the fast tier carries about 80% of the savings. If you optimize one thing, optimize the routing.

Lever	Implementation effort	Typical cost saving	Quality risk
Prompt compression	Low — edit the prompt	10–30% on input	Low if you keep the load-bearing instructions
Context pruning	Low — cap top-k	20–40% on input	Medium — too few chunks drops recall
Response limiting	Low — set maxTokens	10–25% on output	Medium — truncates long but valid answers
Tiered routing	Reuses the M3 router	The largest single lever	Low — wrong tier under-answers, so gate by complexity
Smaller model per task	Medium — per-task choice	High where it fits	Medium — verify quality holds task by task

(Those percentages are illustrative, workload-dependent estimates, not AWS-published figures — measure your own.) One nuance the exam loves to punish: cheaper per token is not cheaper per task. A model with a lower per-token price can cost more per ticket if it needs retries, writes longer answers, or sends an agent into extra loops. Always compare the cost of the finished task, not the sticker price of a token.

⚠️ Common misconception: “Caching is one thing.”

It is three things, with opposite risk profiles, and the exam separates them on purpose.

Prompt caching caches a reused input prefix on the provider side — a long, identical system prompt, say — and bills the cached portion at roughly 10% (up to −90%). It caches input, so it can never serve a stale answer.

Semantic caching matches a new question that is close in meaning to one you already answered (by embedding similarity above a threshold) and serves the stored answer. The saving is huge — a near-duplicate costs ≈ $0 — but the risk is real: a false hit serves a stale or wrong answer to a customer.

Deterministic request hashing (a.k.a. result fingerprinting) keys on an exact normalized question: identical question → identical key → the identical stored answer. Zero risk, no embedding needed.

You configure prompt caching and semantic caching nothing alike, because their failure modes are nothing alike.

Caching: prompt, semantic, and deterministic

Relay uses all three, each where it fits.

Prompt caching rides Relay’s long, unchanging system prompt. The same instructions go to the model on every ticket — a perfect reused prefix. It threads through the existing converse() call by parameter; the layer inserts a Converse cache point on the system block, and the response reports how much input was served from cache:

res = llm.converse(messages, tier="auto", system=[{"text": SYSTEM_PROMPT}],
                   cache_prompt=True)          # cache point on the system prefix
res.usage["cacheReadInputTokens"]              # served from cache → bills at ~10%

This is the only cache lever wired onto the interactive path, precisely because caching input can never go stale.

The semantic cache lives in a new module, relay/cache.py, in front of the answer path for frequent questions. It tries the deterministic hash first — an exact-match GetItem, zero risk, no embedding — then falls back to a semantic lookup: embed the question with the pinned Titan V2 embedder (the same one the Knowledge Base uses, never swapped), and serve the stored answer only if cosine similarity clears a strict threshold.

flowchart LR
    Q["incoming question"] --> E["embed — Titan V2"]
    E --> L["DynamoDB similarity lookup"]
    L -->|"score ≥ threshold"| H["return cached answer — cost ≈ 0"]
    L -->|"miss"| C["converse() / answer()"]
    C --> ST["store {embedding, answer}"]
    ST --> Ret["return answer"]

The real danger of a semantic cache is the false hit — a question that only looks similar getting a stale answer. So it is never a blind cache: a strict similarity threshold (cosine ≥ 0.95) gates every semantic hit, a TTL ages entries out as passive invalidation (a CloudCart doc change flushes within a day — the Module 5 freshness story), and an explicit invalidate() drops an entry the moment you know the doc behind it changed.

Cache	What it caches	Typical hit rate	Main risk	Where in Relay
Prompt caching	A reused input prefix (system prompt)	High on steady traffic — the warm prefix is reused while calls keep arriving (the provider cache ages out after a short idle window)	None — it caches input, never an answer	Interactive path, on every ticket
Semantic caching	A stored answer for a near-duplicate question	Workload-dependent — high on FAQ traffic	False hit → stale/wrong answer	Frequent questions, threshold-gated
Deterministic hashing	A stored answer for an exact repeat	Lower — exact repeats only	None — exact key, exact answer	First check inside the semantic cache

The cache table is DynamoDB on-demand (relay-cache, ~$0 at idle). It scans a small, course-scale table and ranks in process; at production scale (millions of entries) you key the semantic lookup off a vector store, not a scan — the point here is the pattern and its correctness guards.

Throughput and latency: batch, Flex, and p95

The last two levers are for latency-tolerant work only, and the boundary is load-bearing.

Batch inference (CreateModelInvocationJob) runs asynchronously at −50%. Re-scoring a whole golden set offline tolerates no latency constraint, which makes it the textbook batch job — submit it, walk away, pay half. (The eval harness that backfills through this road is Module 13; Module 12 provides only the road.) The Flex service tier (re

2025) is the same idea for synchronous-shaped calls: −50% in exchange for higher, variable latency, selected per call by parameter. Both fit eval and backfill jobs.

Both also share one hard rule: never on Relay’s interactive traffic. A customer waiting on a chat reply cannot eat asynchronous or variable latency, and the 50% saving never justifies a slower answer. Relay’s interactive path stays on the Standard tier; the agent, worker, and API handlers never pass service_tier="flex" — a rule the smoke test asserts.

For the interactive path you optimize latency differently: latency-optimized models where the catalog offers them, parallel requests to fan out independent steps of a complex workflow, pre-computation of predictable queries, and retrieval tuning (index and query). You measure the result with API call profiling — specifically the p95 latency, the tail number that actually catches a slow path, before and after.

That before/after table — cost and p95 — is the graded deliverable. Here is the real run, cost_report.py replaying the reference tickets baseline vs optimized against the live models (one planted near-duplicate is the 1 semantic-cache hit at cost ≈ 0):

Metric	Baseline	Optimized	Delta
$/ticket	0.0313c	0.0055c	−82.4%
Total cost	0.3131c	0.0551c	−82.4%
p95 latency	18.07 ms	10.30 ms	−43.0%

Baseline forces every ticket onto a full-price smart-tier answer; optimized lets the router pick the tier, turns on prompt caching, and runs the semantic cache — the planted near-duplicate hits the cache and reports cost_cents ≈ 0, and the tail latency falls with it. That is the exact shape Domain 4 questions ask you to defend.

Build it

The lab makes Relay’s $/ticket measurable, then attacks it on four fronts. You inherit the full Module 11 Relay — the deployed API, agent, Knowledge Base, guardrail, PII/IAM, and the converse() layer — and add relay/cache.py (new) and cost_report.py (new), plus additive edits to relay/llm.py, relay/config.py, and the worker handler. The frozen relay/models.py does not change — cost_cents has existed since Module 7.

This lab cost me $0.02 on June 2026 prices (the Module 12 budget is < $2). Everything M12 adds is on-demand / ~$0 idle. Re-verify pricing the day you run it — re-check the prompt caching, batch, and service-tier sections of the Bedrock pricing page, as of June 2026.

1. Instrument cost_cents. A CostMeter context manager in relay/llm.py records every converse() call’s usage and totals it through the M3 price map. The worker wraps the agent run in it and writes the metered cost onto the persisted TicketRecord:

from relay import llm

with llm.CostMeter() as meter:
    run_relay(payload)            # any number of converse() calls, any tier
record.cost_cents = meter.cost_cents   # the M7 placeholder is finally real

converse()’s signature is untouched (converse(messages, *, tier="auto", stream=False, **params)) — it has been byte-identical since Module 3, and stays so. Prompt caching and Flex ride in through **params; we never add a second Bedrock client.

2. Prompt caching inserts a cache point on the reused system prefix, as shown above, and the discounted cost is computed from the usage block:

def estimate_cost_discounted(tier, input_tokens, output_tokens, *,
                             discount=0.0, cached_input_tokens=0):
    p = PRICE_PER_1K.get(tier, PRICE_PER_1K["fast"])
    cached = max(0, min(cached_input_tokens, input_tokens))
    uncached = input_tokens - cached
    raw = (uncached / 1000 * p["input"]
           + cached / 1000 * p["input"] * (1 - PROMPT_CACHE_INPUT_DISCOUNT)  # ~10%
           + output_tokens / 1000 * p["output"])
    return raw * (1 - discount)                            # Flex/batch −50% when set

(The real config.py adds int() casts and a 0.0–1.0 clamp on discount; the math is the same.) With both levers at their defaults this returns exactly the Module 3 cost, so an interactive call’s cost line never moves.

3. The semantic cache (relay/cache.py) wires in front of the answer path:

from relay import cache as cache_module

cache = cache_module.SemanticCache()          # DynamoDB relay-cache + Titan V2
hit = cache.lookup("where is my order 1042?")
if hit.hit:
    return hit.answer                         # cost ≈ 0, cache_hit=True
answer = kb.answer(question)                  # the MISS path: a real converse()
cache.store(question, answer)                 # so the next near-duplicate hits

setup.py creates the on-demand relay-cache table (HASH key question_hash, native TTL on expires_at) — verified ACTIVE, PAY_PER_REQUEST, with TTL enabled; the live run wrote nine real Titan-V2-embedded entries.

4. Batch inference demonstrates the eval-backfill road. setup.py builds a small JSONL backfill, uploads it under s3://relay-<account>/batch/input/, and --submit-batch submits the −50% job:

uv run python setup.py --submit-batch   # real CreateModelInvocationJob, −50%, then stopped

In the live run this submitted a real job to Bedrock (modelId us.amazon.nova-micro-v1:0, status Submitted), then stopped it at ~$0 compute. Flex is the same **params channel — llm.converse(messages, tier="fast", service_tier="flex") — eval/backfill only.

5. The before/after report. cost_report.py replays the reference tickets twice and prints the table above:

uv run python cost_report.py            # live: baseline vs optimized, $/ticket AND p95
uv run python cost_report.py --offline  # same shape, no AWS, no cost

Running it live printed the graded table — $/ticket 0.0313c → 0.0055c (−82.4%) and p95 18.07 ms → 10.30 ms (−43.0%), one real semantic-cache hit at cost ≈ 0. The cumulative offline suite (Modules 2–12) passes 293 tests; the live marker collects 11 tests across all modules, and the 6 that the deployed-API-free environment can reach ran (3 real Nova Micro calls with prompt caching on, plus an inherited Knowledge Base grounded-and-cited answer — the rest skip without a deployed API). Teardown drops the relay-cache table, the batch role, and the batch/ artifacts — nothing M12 added is left idle-billed.

Try it yourself: (1) Sweep the semantic-cache threshold — run cost_report.py --offline --threshold 0.85 then --threshold 0.98 and watch the hit-rate-vs-false-hit tradeoff. The product owner owns this number — a false hit is a wrong answer to a customer. (2) Submit the backfill with setup.py --submit-batch and compare the total cost of the 100-record job to running the same work synchronously (−50%).

In production

At scale, $/ticket becomes a KPI you track over time, not a one-off measurement — and the last prompt change that doubled the bill becomes visible the day it ships (you’ll alarm on cost anomalies in Module 14). The semantic cache earns its keep only if you watch its hit rate: a cache running at a 5% hit rate does not pay for the code that maintains it, and you should be ready to retire it.

The hard call is the similarity threshold, and it is not only the engineer’s. A false hit is a stale or wrong answer reaching a real customer, so the product owner should own where that bar sits — and you need an invalidation story for when CloudCart’s docs change, so the cache does not keep serving an answer that cited a page that moved. The TTL handles the slow case; an explicit invalidate() handles the known one. Neither is optional — a never-expiring cache is a quiet correctness bug.

Batch is a budget line of its own — backfills run on their own schedule and their own (Flex/batch) discount, well away from the customer-facing path. And provisioned throughput exists for the opposite problem: a guaranteed-throughput SLA at steady high volume, billed whether or not you use it (it’s a deployment choice you weighed in Module 11; here it is the cost foil to on-demand and Flex). One freshness note: re-verify the latency-optimized model catalog the day you build — name a model only if it is confirmed available, otherwise teach the concept.

Exam corner

Domain 4 is where the exam loves a numbers argument: cost vs latency vs quality, defended with a before/after table. Tasks 4.1 and 4.2 ask you to pick the right lever for a traffic shape and to know which caches and tiers carry which risks.

1. A team wants to cut the foundation-model bill of a RAG application that re-sends a large, identical system prompt and a block of reference docs on every request, with no quality loss. Which lever should they reach for first?

   • A. Switch every request to a smaller model.
   • B. Turn on prompt caching for the reused system-prompt-and-docs prefix.
   • C. Add a semantic cache over the user questions.
   • D. Move the workload to batch inference.

2. Two traffic patterns land on your desk. App A re-sends a 4,000-token system prompt unchanged on every call. App B gets thousands of customer questions that are near-duplicates of each other. Which cache fits which?

   • A. Prompt caching for A; semantic caching for B.
   • B. Semantic caching for A; prompt caching for B.
   • C. Prompt caching for both.
   • D. Deterministic request hashing for both.

3. A nightly job re-scores 10,000 stored responses against an updated rubric. There is no latency constraint — it just has to finish by morning. What is the most cost-effective way to run it?

   • A. Synchronous on-demand invocations, parallelized.
   • B. Batch inference with CreateModelInvocationJob (−50%).
   • C. Provisioned throughput for the duration of the job.
   • D. A latency-optimized model.

4. You run two workloads: a latency-tolerant nightly evaluation job, and a real-time customer chat endpoint. Which service tier suits each?

   • A. Flex for both — it’s −50%.
   • B. Flex for the nightly eval job; Standard (or Priority) for the chat endpoint.
   • C. Standard for the eval job; Flex for the chat endpoint.
   • D. Priority for both.

5. Your traffic is 80% trivial lookups and 20% genuinely complex tickets, on a tight budget. What routing minimizes cost without breaking the hard tickets?

   • A. Route everything to a single mid-tier model.
   • B. Route everything to the frontier model for safety.
   • C. Route the 80% trivial traffic to the fast tier and the 20% complex traffic to the smart tier.
   • D. Route everything to the fast tier and accept the misses.

Answers. 1 — B. A large reused input prefix is the textbook prompt-caching case: up to −90% on the cached input, and zero chance of a stale answer because it caches input, not output. A smaller model (A) or compression risks quality; a semantic cache (C) targets duplicate questions, not a reused prefix; batch (D) is for latency-tolerant jobs, not an interactive RAG app. 2 — A. Prompt caching fits a reused prefix (App A); semantic caching fits near-duplicate questions (App B). They have opposite risk profiles, so swapping them (B) is wrong, and one cache cannot cover both patterns (C, D). 3 — B. A large, latency-tolerant re-scoring job is exactly what batch inference is for: asynchronous, −50%. Parallelized on-demand (A) pays full price; provisioned throughput (C) is for steady high-volume SLAs and bills idle; a latency-optimized model (D) optimizes the wrong axis. 4 — B. Flex is −50% but latency-tolerant — perfect for the nightly eval job, never for a customer who is waiting. Standard or Priority serves the interactive path. 5 — C. Tiered routing by complexity — fast for the trivial 80%, smart for the complex 20% — minimizes cost while protecting the hard tickets. A single mid model (A) overpays on the easy ones and may under-serve the hard ones; all-frontier (B) is the most expensive; all-fast (D) breaks the complex 20%.

Traps to avoid.

• The three caches are not interchangeable. Prompt caching = a reused input prefix on the provider side (no staleness risk). Semantic caching = a similarity match that serves a stored answer (real false-hit risk). Deterministic hashing = an exact key. The exam mixes them on purpose.
• Flex and batch are not “for everything.” −50% is seductive, but the asynchronous/variable latency is fatal on the interactive path. They belong on eval and backfill jobs, never on a customer who is waiting.
• Cheaper per token is not cheaper per task. Retries, longer answers, and extra agent loops routinely flip the math. Compare the cost of the finished task, not the price of a token.

Key takeaways

• Measure before you optimize — $/ticket is your only ground truth. Token estimation and tracking populate the real cost_cents; “it feels expensive” is not a diagnosis.
• The fast-tier router pays roughly 80% of the savings. Tiered FM usage by complexity was a cost decision before it was an architecture decision; optimize it first.
• Prompt caching ≠ semantic caching ≠ deterministic hashing — three tools, three risk profiles. Prompt caching can never go stale; a semantic cache can serve a false hit.
• A semantic cache needs a threshold and an invalidation story, never blind faith — a strict similarity bar, a TTL, and explicit invalidation when the docs change.
• Flex and batch are −50% but latency-tolerant — eval and backfill jobs only, never the interactive path.
• Cheaper per token is not cheaper per task — retries, longer answers, and agent loops can flip the math.
• Prove an optimization with a before/after table for cost AND p95 — exactly the format Domain 4 questions ask for.

What’s next

Relay is now cheap and fast — but is it still good? We compressed prompts and downshifted tiers to save money; did answer quality survive? Module 13 builds the eval harness: a 20-ticket golden set, Bedrock RAG evaluations, an LLM-as-a-judge, and a regression gate that blocks any deploy that makes Relay worse. (The batch road you just built is the path that backfill runs on.)

Want the full AIP-C01 question bank and the next module in your inbox? Subscribe here — it’s free, like everything in this series.

Links: Module 12 lab · ← Module 11 · Course index · Module 13 →

References

• Prompt caching for cost-effective AI models — Amazon Bedrock — how cache points work on Converse and what the usage block reports.
• Run batch inference jobs — Amazon Bedrock — CreateModelInvocationJob, the JSONL input format, and the asynchronous −50% model.
• Amazon Bedrock pricing — on-demand, batch, prompt caching, and service-tier figures (dated; re-verify on generation day).
• Increase throughput with cross-Region inference / inference profiles — Amazon Bedrock — the inference profiles every Relay tier resolves to.
• Amazon DynamoDB on-demand capacity mode — the ~$0-idle billing model behind the semantic-cache table.
• AWS Certified Generative AI Developer - Professional (AIP-C01) exam guide — Tasks 4.1 and 4.2, the skills this module covers.