Coastline MetricsOperator

Learn

How LLMs work (the parts that matter for visibility)

You don’t need to become an AI engineer to make good decisions. This is the minimum you need to understand why AI answers can differ, and what “verification” really means.

1) A model predicts text

An LLM is a probabilistic text generator. Small differences in prompts, system settings, and model versions can change outputs.

2) Retrieval changes the answer

Many products add retrieval (search/browse) so the model can quote current web sources. If your site/profile data is missing or inconsistent, it may not be retrieved or cited.

3) “Citations” are signals, not guarantees

Even when a UI shows citations, the model may summarize, paraphrase, or blend sources. You still need to verify what was used and what changed.

4) Verification requires comparability

To compare runs, hold constant: prompt pack, locale, provider/model, and methodology. Otherwise, differences may just be drift.

What we show in provenance

  • Provider and model
  • Prompt pack kind/version and hash
  • Methodology version
  • Date/time and latency
  • Any warnings about partial failures

That’s how we stay honest about what we measured—and what we didn’t.

Selected sources