Traditional APM tells you a request was slow or failed. LLM observability adds model-specific signals: which prompt and model produced a response, how many tokens it used, what it cost, and whether the output was accurate, relevant, and safe.
Core metrics are latency (including time to first token), token usage, cost per request, error and rate-limit rates, and quality scores from evaluations such as LLM-as-a-judge. For RAG apps, retrieval quality and vector database latency matter too. For agents, observability extends to tool calls and full trajectories (see agent observability).
OpenTelemetry GenAI semantic conventions standardize how LLM spans and metrics are recorded. OpenLIT is an open-source LLM observability platform that emits those spans natively, stores them in ClickHouse, and can export the same data to Grafana, Datadog, or any OTLP backend.