Observe Business Critical Services end to end with Elastic Observability

When a streaming play fails at 20:00 on a Saturday, the business question is rarely “which span in one microservice was slow?” It is: Is the Business Critical Service healthy end to end?

That service usually spans user experience, CDN or edge, origin applications, network path, and platform. Many of those steps are async. Often there is no single trace.id handed from browser to CDN to packaging worker. End-to-end observability, in the sense Elastic Content Share means it, is the practice of answering the service-health question across those layers anyway.

This article walks through the what, why, who, when, where, which, and how of that model, using TV streaming as the running example, then maps the same pattern to retail, banking, and telco. It also explains how CMDB relationships add ownership and declared dependency context, how AI can synthesize the many observability angles without replacing evidence, and which Content Share packs turn the model into something you can import and adapt.

What is end-to-end observability here?

What it is: a composite view of a Business Critical Service (BCS) — the customer-facing capability the business must keep healthy, such as tv_streaming, checkout, or mobile_banking. The BCS is not a single microservice and not a single distributed trace.

What it is not:

• Only a RUM funnel or synthetics journey (those are the experience layer)
• Only an APM service map (that is hop-local application evidence)
• Only host or Kubernetes saturation (platform is necessary but not sufficient)
• A clone of a vendor “journey” or “business flow” UI; Elastic already has the signals — Content Share packages the composed operating model

What “healthy” means: enough of the capability chain succeeds within error budget that customers can complete the outcome the business sells (start play with acceptable QoE, complete payment, fulfill an order, attach a subscriber session).

Why does this matter now?

Why single-trace thinking fails as the top object: distributed tracing is excellent inside one instrumentation domain. It is a weak top-level model when:

• the CDN only knows a session token, content ID, or cache key
• origin packaging ran minutes or hours earlier on a queue worker
• network quality sits outside application spans
• many CDN deliveries reuse one origin fetch (“package once, serve many”)

Why DORA-style oversight needs a different lens: executives and risk owners ask whether the critical service is up and within budget. That answer is a composite of experience, edge, origin, network, and platform SLIs — not the status of one waterfall.

Why other solutions can feel “easier”: Some platforms package multi-signal journey or business-flow stories as product UX. Elastic already provides RUM, Synthetics, APM/OpenTelemetry, logs, metrics, network signals, and SLOs. The gap for many teams is not missing pillars; it is missing a reusable BCS model, join conventions, and investigation path. That is what the End-to-End Observability hub and packs are for.

Who should care?

If your team only optimizes for one persona, you will rebuild siloed dashboards and re-learn the same incident every quarter. The BCS model is the shared vocabulary across those owners.

When should you use this approach?

When BCS / async E2E is the right frame:

• Customer outcomes depend on systems that do not share one instrumentation domain
• Edge, CDN, messaging, or batch packaging sits between user action and “backend truth”
• Incidents start as “service is bad” and then bounce between CX, SRE, network, and platform without a shared join key
• You need DORA-style service reporting that executives can read without opening APM

When hop-local tracing is enough (and BCS is overkill as the first move):

• A single well-instrumented app owns the whole request path and propagates trace.id end to end
• You are debugging one release regression inside one service graph
• You already know the failing layer and only need span-level root cause

Most real Business Critical Services eventually outgrow the second list. Use traces as drill-down; use BCS as the top of the investigation stack.

Where does the story live? (layers)

Use TV streaming as the concrete map. Remap the same rows to your industry later.

Each layer should publish: a stable labels.critical_service, at least one cross-layer join key, health signals, and optional hop-local trace.id.

How does CMDB dependency information become part of end-to-end observability?

A CMDB and an observability platform answer different questions. Elastic observes what systems are doing: which services communicate, where latency changed, which objective is burning, and which log pattern appeared. A CMDB records what the organization believes it operates: configuration-item identity, owner, business criticality, lifecycle state, maintenance windows, and declared upstream or downstream dependencies. Neither view is complete on its own.

The useful result is not a one-time CMDB export displayed beside a dashboard. It is a maintained identity and relationship layer that enriches findings. When Elastic detects abnormal latency in payment-api, the CMDB context should answer: which Business Critical Services use it, who owns it, whether it is customer-facing, what support group and runbook apply, and which declared dependencies may extend the blast radius.

Which CMDB data is worth bringing into Elastic?

Start with fields that change an operational decision, not every attribute ever collected by ITSM:

• Stable identity: CI ID, service ID, application ID, environment, and source-system key.
• Business context: business-service or BCS name, capability, criticality, customer impact, and regulatory classification.
• Accountability: owning team, support group, escalation route, runbook URL, and service owner.
• Lifecycle context: active/retired state, planned maintenance, version, location, and last successful CMDB synchronization.
• Relationships: depends_on, runs_on, hosted_by, connects_to, or organization-specific relation types, with source, target, and validity timestamps.

Keep provenance with every record: source CMDB, synchronization time, matching method, and confidence. Responders need to distinguish a verified CI match from a name-based inference, especially when the declared graph is stale.

How do telemetry entities match CMDB configuration items?

The hard part is identity reconciliation. A CMDB may call an application “Payments Platform – Production,” while telemetry reports service.name: payment-api, a Kubernetes deployment, several cloud instances, and short-lived pods. Do not join only on a display name. Define an identity contract that prefers durable keys and includes environment or namespace where necessary:

• service.name + service.environment to application/service CI
• host.id, cloud instance ID, or asset ID to compute CI
• Kubernetes cluster UID + namespace + workload name to workload CI
• explicit cmdb.ci_id or service.id emitted by instrumentation when the organization can govern it

Store the normalized CMDB snapshot in a dedicated Elasticsearch index and make the entity index suitable for ES|QL LOOKUP JOIN. That article is useful here because it shows how to add maintained reference data at query time instead of copying changing ownership data into every raw trace or metric. For stable, high-volume enrichment that must exist at ingest, an Elasticsearch enrich policy and enrich processor are another option. Query-time lookup is usually safer while the identity model is still evolving.

How should declared and observed dependencies work together?

APM traces and OpenTelemetry spans reveal observed communication. The CMDB supplies declared relationships, including dependencies that may not produce a trace, such as DNS, certificate, network, vendor, batch, or manually operated components. Treat them as complementary evidence:

• Observed and declared: high-confidence dependency; use it for blast-radius and ownership context.
• Observed but not declared: possible CMDB drift, a new release path, or an unmanaged dependency worth reviewing.
• Declared but not observed: dormant/failover path, missing instrumentation, stale relationship, or simply no calls in the selected time window.

Absence of an observed edge is not proof that the CMDB is wrong. Time range, sampling, async processing, and missing instrumentation all matter. Show both provenance and last-seen time instead of silently merging the two graphs.

Step by step: add CMDB context without breaking the telemetry flow

1. Ingest a read-only CMDB snapshot. Use the source API, Logstash, or an existing integration to synchronize configuration items and relationships into namespaced indices. Do not make incident dashboards call the CMDB live.
2. Normalize identities. Build a small entity lookup keyed by durable telemetry fields plus environment. Record unmatched and ambiguous entities instead of guessing.
3. Normalize dependency edges. Store source CI, target CI, relationship type, source system, validity, and last synchronization. Preserve direction.
4. Enrich findings first. Join CMDB context to SLO breaches, anomalies, alerts, and affected services. This adds value with less storage and staleness risk than enriching every raw event.
5. Compare graphs. Flag observed-but-undeclared and declared-but-unobserved relationships for review; do not automatically rewrite the CMDB from telemetry.
6. Expose operational context. Put owner, criticality, affected BCS, runbook, and dependency path beside the finding that triggered the investigation.
7. Monitor freshness. A visible cmdb.last_sync, source, and match confidence prevent stale context from being mistaken for live truth.

In the streaming example, a spike in playback-api latency becomes much more actionable after enrichment: the API supports tv_streaming, is owned by the Playback SRE team, depends on entitlement and regional Redis services, and has a high criticality. Elastic still supplies the live anomaly, SLO burn, traces, and logs; the CMDB explains organizational impact and where responsibility sits.

Which keys survive async boundaries?

Which keys are the BCS spine:

• labels.critical_service — e.g. tv_streaming
• labels.bcs_capability — discover, play, stream_qoe, cdn_delivery, origin_package, network_path

Which joins work when there is no shared trace:

• Session bridge — UX play/QoE events and CDN logs share session.id / labels.session_id in a time window
• Content / entity bridge — origin packages content_id once; CDN serves many requests for the same ID or cache_key
• Message bridge — producers and consumers copy labels.correlation_id / message_id across queues
• Weak join — critical_service + region/PoP + service + tight time window (hypothesis, not proof)

Which role does trace.id keep? Valuable inside a hop (origin API, worker graph). Optional across layers. Do not force a synthetic global trace when systems cannot propagate it.

Field naming, cardinality notes, and ES|QL starters ship in the E2E Observability Correlation Keys pack (BCS edition).

How does a TV streaming incident actually unfold?

Walk a realistic Saturday-night failure with W-questions at each step.

1. What do customers feel? Play starts fail or rebuffer rates spike in one region.
2. Who notices first? CX dashboards and support tickets — not necessarily APM alerts.
3. Where is the BCS red? Open the multi-layer scoreboard for critical_service: tv_streaming. Suppose experience and CDN look bad; origin packaging looks green for the last hour.
4. Which join proves the boundary? Take failing session_id values from RUM/QoE. Bridge to CDN logs for the same sessions. If CDN shows origin-fetch 5xx for those content IDs, the story moved to edge-to-origin delivery — still without a shared frontend trace.id.
5. When did origin last package that content? Entity-bridge on content_id. Packaging may have succeeded hours earlier; the live failure is delivery or path, not “the encode job.”
6. Where next? If PoP or ASN patterns dominate, continue into Network Observability packs. If origin APIs are implicated, use hop-local APM. If pods are throttled under the implicated services, continue into Kubernetes Observability.
7. Why did DORA care? The composite play-success SLI burned error budget even though one microservice team’s nightly dashboard looked fine.

That investigation path is intentional: BCS scoreboard first, join keys second, hop-local traces third, layer packs fourth.

How do common failure modes look by layer?

How can AI help teams understand all these observability angles?

More signals do not automatically create more understanding. During an incident, one responder may see an SLO burn, another an ML anomaly, another a new log category, and the CMDB owner a dependency relationship. AI can reduce that cognitive and coordination cost by turning a bounded evidence bundle into a readable briefing. Its job is to connect and explain evidence, not to manufacture certainty.

What should the AI receive?

Give the agent the smallest useful set of already-computed findings for the affected BCS and time window:

• Customer impact: SLO status, burn rate, failed journey, and affected capability.
• Behavioral deviation: scoped Machine Learning anomalies and ES|QL change points.
• Failure signature: emerging log categories, unusual log-rate contributors, and relevant alerts.
• Runtime topology: observed callers and downstream services from traces or the APM service map.
• Declared context: CMDB owner, criticality, runbook, maintenance state, and dependency edges.
• Correlation bridges: the session, content, message, service, environment, region, and time keys used to connect the evidence.

This scope matters. Sending unrestricted raw telemetry to an agent is expensive, noisy, difficult to audit, and more likely to produce a plausible but unsupported story. Elastic detection features should first narrow the data to findings; the agent then explains relationships between those findings.

Which questions can AI answer well?

• What changed? Summarize the anomaly, change point, new pattern, or deployment-adjacent shift in plain language.
• Why does it matter? Connect the affected entity to the BCS, SLO, criticality, and customer-facing capability.
• Who is affected and who should respond? Use region, environment, callers, CMDB dependencies, and ownership to describe likely blast radius and route the investigation.
• Which independent signals agree? Explain that an SLO burn establishes impact while ML, change points, logs, and topology strengthen or weaken a hypothesis.
• What is missing? State that no network evidence, no matching CMDB CI, insufficient ML history, or no cross-layer join was available.
• What should happen next? Recommend verification steps in evidence order: inspect affected sessions, compare dependency latency, open hop-local traces, check change history, then involve the owning team.

How do you keep the briefing trustworthy?

Require the output to separate three levels:

1. Observed: directly supported facts such as “the checkout availability SLO is burning at 3.2×” or “a latency anomaly began at 14:32.”
2. Correlated: multiple facts aligned by service, environment, region, dependency, or time — useful evidence, but not automatic causality.
3. Hypothesis: a possible explanation with explicit uncertainty and a concrete verification step.

Keep the underlying dashboard panels and links visible so a responder can verify every important statement. Use a read-only, least-privilege agent; do not give it broad connector administration or automatic remediation rights by default. If the model or connector is unavailable, deterministic SLO, ML, ES|QL, alert, pattern, and dependency findings must remain usable.

What does this look like in the streaming incident?

An AI-supported briefing might say: Observed: play-start success is burning budget in one region; CDN origin-fetch failures changed sharply at 20:03; a new timeout log category appeared in entitlement-api. Correlated: failing sessions share the same region and the CMDB declares entitlement-api as a high-criticality dependency of playback-api. Hypothesis: entitlement latency may be propagating through origin fetches; verify with hop-local traces and dependency latency before declaring root cause. Missing: no matching network degradation was found.

That is useful because it combines customer impact, statistical deviation, logs, runtime topology, and CMDB context while showing its reasoning boundary. It is not useful if it simply says “entitlement is the root cause” without evidence.

The Global IT Health — Native Findings package demonstrates this findings-first composition. It installs SLO, ML, change-point, log-pattern, alerting, workflow, service-map, and AI briefing layers so readers can see how deterministic Elastic features feed a bounded synthesis. Use it for the prioritization and briefing layer; use the BCS correlation packs below when an investigation must cross browser, CDN, async worker, network, and platform boundaries.

How should you model a BCS in your org?

1. What is the Business Critical Service name? Pick one vocabulary (tv_streaming), not five synonyms.
2. Which capabilities matter for customer outcome? Keep the list short enough for a scoreboard.
3. Who owns each capability’s signals and SLIs?
4. Which join keys survive your async boundaries? Prefer session, content/entity, message — document renames.
5. Which CMDB identities and dependency relationships add ownership, criticality, and declared blast-radius context?
6. What composite SLIs roll into the DORA view? Example: play start success, CDN error rate for the BCS, origin packaging/API budget, synthetic path availability.
7. How do responders drill down? Write the runbook before the incident: scoreboard to red capability to join to CMDB context to hop-local evidence.
8. Where can AI reduce cognitive load? Define a bounded evidence bundle, observed/correlated/hypothesis output, and a deterministic fallback before enabling synthesis.

Where else does the same pattern apply?

Streaming is the teaching example because async edge and “package once, serve many” make the anti-pattern obvious. The same W-questions remap cleanly:

• Retail / ecommerce — BCS checkout; capabilities browse, cart, pay, fulfill; joins order_id, session_id, payment correlation_id; fulfillment may be hours later with no shared checkout trace.
• Financial services — BCS mobile_banking or card_auth; joins account/session and message IDs across fraud, ledger, and notification workers.
• Telco — BCS subscriber_attach or voice_quality; joins IMSI/session-like identifiers across RAN/core-ish telemetry, CRM, and app experience (adapt fields; keep the layer model).

Do not force every industry into a checkout funnel demo. Force every industry into the same layer contracts and join discipline.

How does Elastic Content Share package this?

Composition, not a single mega-app:

• End-to-End Observability hub — BCS-oriented entry point and investigation path
• Correlation keys (BCS / async joins) — field map, personas, anti-patterns, samples
• Business Critical Service health dashboard — multi-layer scoreboard plus demo events joined by session and content IDs
• ES|QL cross-layer / async queries — session, entity, and message bridges; multi-layer failure sessions
• Synthetics critical journey templates — proactive UX path checks (one layer of the BCS)
• User journey / SLO templates — building blocks for layer and composite error budgets
• Global IT Health — Native Findings — findings-first prioritization that combines SLOs, ML, change points, patterns, alerts, topology, Workflows, and an evidence-grounded AI briefing
• Layer drill-downs: APM, OpenTelemetry, Network Monitoring Suite, Kubernetes OTel SLO Templates

Use synthetics and journey SLOs as experience-layer controls. Use Global IT Health to prioritize and summarize findings across Elastic features. Use CMDB lookups to add declared business and ownership context. None of those replaces the correlation keys needed to prove an async cross-layer path.

How do you investigate in practice? (checklist)

1. Open the BCS health view for critical_service (composite and per capability).
2. Identify which capability is red (experience vs CDN vs origin vs network vs platform).
3. Join with session, content, or message IDs across that boundary — not only trace.id.
4. Enrich the affected service or finding with CMDB owner, criticality, runbook, and declared dependencies; check match confidence and freshness.
5. Use hop-local APM traces inside the failing layer.
6. If path or PoP related, continue into Network packs; if capacity, into Kubernetes packs.
7. Let AI synthesize the bounded SLO, anomaly, change, pattern, topology, and CMDB evidence into observed facts, correlations, hypotheses, missing signals, and next checks.
8. Verify the briefing against deterministic panels, then update the composite SLI / error budget narrative for stakeholders who will never open Discover.

Related downloads

Global IT Health — Native Findings for Elastic Observability

Turn existing ECS and OpenTelemetry data into prioritized operational findings. This zero-effort package discovers services, evaluates bootstrap SLOs, learns ML baselines, detects changes and log patterns, triggers alerts, and produces an evidence-grounded AI briefing in one Global IT Health dashboard.

Download Now

Observed Assets Inventory Dashboard for Elastic

See every host, service, and pod your cluster has actually observed from ECS or OpenTelemetry — identity, signal coverage, and health on each asset, not event counts. Import the Lens ES|QL dashboards and they light up on logs-*, metrics-*, and traces-* with no extra mapping.

Download Now

Distributed Tracing Investigation Pack for Elastic APM

Hop-local APM/OTel investigation runbook and ES|QL: dependency errors, slow spans, and entity-scoped traces after a BCS scoreboard marks origin red.

Download Now

RUM User Journey and Conversion Pack for Elastic

Real-user journey funnel, Core Web Vitals, and session-aware RUM conventions for Business Critical Services—built to stitch with CDN and origin via session IDs.

Download Now

Network Engineering Topology for Kibana

Import a clickable Kibana topology dashboard with Vega zone and adjacency views, plus an Agent Builder readiness agent that checks whether discovered devices have enough evidence for reliable map placement.

Download Now

Business Critical Service Health Dashboard for Elastic

Multi-layer BCS scoreboard (TV streaming example): experience, CDN, origin, network, platform—with demo events joined by session and content IDs.

Download Now

What should you do next?

1. What BCS will you name first? Pick one revenue-critical capability (streaming, checkout, attach) and write its capabilities on one page.
2. Which join keys will you standardize? Adopt the BCS correlation keys pack and ban silent field synonyms.
3. How will the scoreboard look? Import the BCS health dashboard; validate with demo data, then wire live labels.
4. When will responders practice the path? Run one game day: red capability to session/content join to hop-local trace.
5. Which CMDB context changes decisions? Synchronize a small, governed lookup of identities, ownership, criticality, runbooks, and dependency edges; test unmatched and stale records.
6. How will AI stay grounded? Start with the Global IT Health package, keep deterministic findings visible, and require observed/correlated/hypothesis separation plus missing-signal disclosure.
7. Where do layer packs plug in? Link Network and Kubernetes runbooks from the BCS investigation steps so the story does not stop at the app team.

End-to-end observability is not a longer waterfall. It is a shared answer to the only question that matters under load: is the Business Critical Service healthy across every layer that makes it work?