When I began building Lemtika, the model question sounded familiar: which language model should power the product?
It was the wrong question.
Lemtika turns a Google Maps listing into a researched, published website. It reads reviews, extracts evidence, clusters patterns, checks the findings, plans a page, and drafts only claims supported by that evidence. Those are not variations of one task. They are different jobs with different failure costs.
A model suited to homepage copy can be wasteful when extracting hundreds of individual observations. A fast extraction model may miss negation. A cheap synthesis model may produce weak candidates that trigger more validation, repair, and review downstream.
The useful question became: which model makes this step, and therefore the complete workflow, perform best?
Start with jobs, not model names
Before comparing providers, I decomposed the product into jobs whose outputs could be evaluated independently.
| Job | The expensive failure | What I measured |
|---|---|---|
| Observation extraction | Evidence silently disappears | Coverage, observation recall, quote accuracy |
| Observation validation | An extraction error enters every later count | False accepts, false rejects, independence from extraction |
| Aspect clustering | One theme is split or unrelated themes are merged | Cluster coherence and stability |
| Insight proposal | Small but useful signals are missed | Candidate recall and evidence relevance |
| Global curation | Thin or duplicate claims survive | Accepted findings, merge quality, invalid output rate |
| Adversarial review | A real quote is attached to the wrong conclusion | Unsupported-claim detection |
| Page planning and copy | Plausible prose outruns the evidence | Reference validity, guard violations, useful final sections |
This decomposition matters because average benchmark scores hide product-specific errors. A model can score well on general reasoning and still be poor at returning every atomic fact of use from a short, informal review. Another can write beautifully while adding a number that no source contains.
The model roster came after the list of failure modes.
Build a corpus that can embarrass the system
A model comparison is only as useful as the examples it is allowed to fail on.
I used both straightforward and difficult businesses. A business with dense vocabulary tests whether the pipeline can group varied phrasing under one theme. A fragmented business - where reviews cover many unrelated services, symptoms, constraints, and practical details - tests whether small signals disappear inside a large context.
The difficult corpus changed decisions that looked settled on the easy one.
One cheaper synthesis candidate reached near-parity on a simple business. On the more complex corpus it ran almost three times slower, produced a much larger pile of rejected or invalid candidates, and cost more by the end of the stage. In that test, the cheaper token rate did not produce a cheaper workflow.
An extraction and validation pairing moved in the opposite direction. It cost slightly more at the first call, but produced roughly one-third more usable observations, lost fewer reviews to throttling, and improved citation discipline. That denser ledger gave later stages more usable evidence.
I did not need a universal model ranking. I needed a repeatable way to expose where each candidate broke the product.
Count the work a response creates
The cost of an API call is not the cost of the decision.
For a multi-stage workflow, I use the following accounting frame. It is a list of work created, not a precise equation:
initial call
+ retries and schema repairs
+ fallback calls
+ downstream candidates created
+ validation and adversary work
+ human review caused by ambiguity
+ recovery work after partial failure
A verbose candidate generator looks productive because it returns more options. If most are generic praise wearing the shape of an insight, each one now has to be curated, cited, challenged, stored, and perhaps shown to a person. Cheap noise compounds.
The same effect appears in structured output. A model with a low input price but frequent schema failures consumes repair generations. A model that burns its output budget on hidden reasoning may return an empty or truncated object, creating a retry despite having already been billed.
For Lemtika I tracked attempts, schema errors, token exhaustion, latency, reported cost, and the final disposition of each logical call. The winning model was the one that reduced the cost of an accepted result, not the one with the cheapest successful HTTP response.
Separate recall from precision
The first version of the research workflow asked one model to synthesize each insight type from a large evidence slice. It was simple and produced credible results. It also missed isolated details that were present in the ledger but too small to win the model's attention.
I split synthesis into three roles:
- Propose locally for recall. Small scoped calls search for candidate findings and may return an empty list rather than pad the answer.
- Curate globally for precision. A global pass sees compact candidates with representative evidence, merges duplicates, enforces the requested insight type, and demotes findings that are true but thin.
- Verify independently. Code checks references and counts. An adversarial model checks whether the quoted evidence supports the conclusion.
That separation changed model selection. The proposal step needs attention to small evidence and disciplined abstention. The curator needs stronger global judgment. The adversary needs independence and precision more than a pleasant writing voice.
Trying to make one model equally good at all three would have hidden the trade-offs rather than removed them.
Independence is part of quality
When generation and validation share a model, they can repeat the same blind spot.
The first semantic check verifies whether an extracted quote matches the assigned aspect and sentiment. A later check asks whether a set of quotes supports an insight. These checks benefit from a different prompt role, a deliberately narrow context, and - where the measurements justify it - a different model family from the one that created the candidate.
This does not make model-based evaluation objective. It reduces correlated failure. Deterministic code still owns the questions it can answer exactly:
- Is the quote a substring of the source review?
- Do the cited observation IDs exist?
- What are the complete positive, neutral, and negative counts?
- Does a page reference an approved insight or photo?
- Did the copy introduce an unsupported number?
Models handle the semantic question left over: does this evidence really mean what the claim says it means?
The layers are intentionally unequal. Code is authoritative about structure and arithmetic. Models contribute bounded judgment. A person controls changes to the published website.
Make model choice a versioned policy
Once a roster works, hard-coding model IDs beside prompts creates a new problem. Models, prices, routing behavior, and provider support change faster than the product contract.
In Lemtika, prompts stay in code because they are structurally coupled to schemas and deterministic guards. Model policy is versioned configuration. Each step resolves an ordered candidate and fallback policy with bounded reasoning, output, concurrency, retry, and repair settings.
A new configuration is completed before activation. A run snapshots the version it starts with and never consults a mutable assignment halfway through. That gives model experiments two properties I care about:
- Reproducibility: I can identify the policy that produced a stored result.
- Safe rollback: a new assignment changes future runs without rewriting the history or behavior of one already in progress.
The model transport sits behind an internal interface, so pipeline code never imports provider SDKs directly. Lemtika currently uses one model-routing rail to keep credentials, cost reporting, and fallback behavior consistent, but the research and website stages own none of that integration detail.
This is the practical form of avoiding model lock-in. It is not supporting every vendor on day one. It is preventing vendor mechanics from becoming the shape of the product.
Observe attempts, not just successful calls
The most informative model behavior often happens before the final result.
A logical call may receive a provider error, retry the same model, reject a malformed object, repair the schema, move to a fallback, and finally succeed. Recording only the last response makes that run look healthy while hiding its cost and fragility.
I record each managed attempt with its model, upstream provider, outcome, duration, token usage, and provider-reported cost. The logical call records whether an earlier failure was recovered. Aggregate rows make spend visible by pipeline step, while diagnostic attempts explain why it moved.
Raw customer prompts and replies do not need to be copied into a third-party observability product to achieve this. Run identifiers connect the model ledger to stored research evidence and the immutable configuration snapshot. That is enough to replay a failure against the same source material without turning customer text into telemetry.
Unreported cost data is a system failure. If a provider has already billed a call but does not return a trustworthy cost, the attempt is recorded first and the workflow fails closed. Retrying would spend more money on an accounting defect that another generation cannot repair.
Design for the model roster to become obsolete
Any article that ends with a list of recommended model IDs starts aging as soon as it is published.
The durable asset is the evaluation method:
- Define the bounded job and its expensive failures.
- Assemble simple, complex, sparse, multilingual, and adversarial fixtures.
- Replay the same inputs across candidates.
- Measure accepted outputs and downstream work, not isolated responses.
- Check whether generation and validation fail independently enough.
- Version the winning policy and snapshot it onto each run.
- Watch production attempts for drift in quality, latency, retries, and cost.
The specific models will change. The observation ledger, evidence laws, validation boundaries, and replay harness should not have to.
Lemtika is live at lemtika.com. The case study covers the wider product: how one Google Maps link becomes structured research and a published website whose claims link back to real customer evidence.
The lesson I carry into other AI products is simple: model selection is not procurement. It is systems design. The cheapest model is the one that helps the complete product reach an acceptable, recoverable result with the least total work.
