Rethinking Observability Architecture in Cyber Security
The Executive Leaders Network Cyber Security Online Summit brought together industry practitioners to discuss evolving challenges in security operations: despite significant investments in tools, dashboards, and telemetry, security teams are not getting faster at investigations, in many cases, they are getting slower.
The session, moderated by Peter Dorrington, Strategic Advisor, Behavioural Analytics Pioneer, explored why this gap persists.
The featured presentation was delivered by Peter Marshall, Director of Developer Relations at Imply is the company behind Lumi, the modern data layer for Observability, Security, and AI. Founded by the original creators of Apache Druid®.
Marshall’s central argument was clear: The challenge is not simply data growth. It is an architectural mismatch between how observability systems are built and how security teams actually investigate incidents.
The Growing Pressure on Observability Systems
Security teams are collecting more telemetry data than ever before. Infrastructure changes such as Kubernetes environments, microservices architectures, and cloud-native platforms generate vast volumes of logs, metrics, and traces.
Marshall highlighted three major pressures currently shaping observability systems:
- Exploding Data Volume: Telemetry systems now ingest billions of events per minute in some environments. As organisations adopt distributed systems, the amount of operational data increases dramatically.
- Expanding Retention Expectations: Regulatory requirements and internal governance policies often require organisations to retain data for extended periods. In some cases, organisations want to retain logs indefinitely to support compliance investigations. However, long-term retention introduces cost challenges. As Marshall noted, storing everything indefinitely effectively requires an “infinite budget”.
- Architectural Mismatch: Most importantly, observability platforms were not designed to handle the combination of massive scale and diverse workload patterns.
This mismatch can lead to performance degradation, operational inefficiencies, and escalating infrastructure costs.
The Core Problem: Detection vs Investigation
Marshall emphasised a key distinction that is often overlooked in security architecture: detection and investigation are fundamentally different workflows.
Detection Workflows (Steady-State)
- Detection typically involves continuous monitoring:
- Dashboards tracking system health
- Automated alerts
- Predictable query patterns
- Short time ranges
- Structured analytics
These workloads are optimized for reliability and low-latency monitoring..
Investigation Workflows (Bursty & Exploratory)
Investigations are fundamentally different.
- Ad hoc queries across large time ranges
- Correlation across multiple datasets
- Run complex joins
- Adjust queries dynamically
- High concurrency during incidents
When an incident occurs, analysts may query months or years of historical data simultaneously. These bursts of activity create sudden spikes in compute demand.
Why This Breaks Traditional Systems
Most observability platforms were designed for detection workloads, not investigative ones.
As a result:
- Investigative queries compete with dashboards and alerts
- Performance degrades during incidents
- Teams restrict queries to protect system stability
- Data is downsampled or dropped to control cost
The system becomes optimized for monitoring but at the expense of investigation.
The Limitations of “Black Box” Platforms
Many modern observability platforms follow a vertically integrated architecture. In these systems:
- Data collection
- Storage
- Indexing
- Computation
- Visualisation
are bundled together within a single platform.
This approach offers convenience but also introduces constraints.
Marshall described how these systems effectively function as “black boxes”, where storage, compute, and processing scale together. As data volumes grow, organisations must scale the entire platform even if only one component requires additional capacity.
This leads to two common outcomes:
- Overprovisioning infrastructure, which increases cost.
- Performance contention, where investigative queries disrupt monitoring workloads.
Many organisations experience both problems simultaneously.
Observability Is Following the Path of Analytics
Marshall argued that this challenge is not unique. Similar architectural problems were previously encountered in business intelligence and analytics systems.
Earlier enterprise systems often bundled:
- Databases
- Reporting tools
- Compute resources
- Storage systems
into a single stack.
Over time, these systems evolved into more modular architectures. Reporting tools, data warehouses, and storage platforms became separate layers, allowing organisations to optimise each component independently.
This shift introduced greater flexibility, scalability, and cost control.
The Case for Decoupling Observability Architecture
The architectural principle at the centre of Marshall’s argument is decoupling.
Decoupling separates key components of the observability stack so that each can scale independently. In practice, this means separating:
- Data ingestion
- Storage
- Compute resources
- Analytics interfaces
By doing so, organisations can optimise infrastructure based on workload characteristics.
For example:
- Detection workflows can continue using highly optimised monitoring systems with a limited window of recent data.
- Investigation workflows can query historical data stored in large-scale object storage systems.
This separation prevents investigative workloads from disrupting operational monitoring.
The Case for Decoupling Observability Architecture
The architectural principle at the centre of Marshall’s argument is decoupling.
Decoupling separates key components of the observability stack so that each can scale independently. In practice, this means separating:
- Data ingestion
- Storage
- Compute resources
- Analytics interfaces
By doing so, organisations can optimise infrastructure based on workload characteristics.
For example:
- Detection workflows can continue using highly optimised monitoring systems with a limited window of recent data.
- Investigation workflows can query historical data stored in large-scale object storage systems.
This separation prevents investigative workloads from disrupting operational monitoring.
Introducing the Observability Warehouse
Imply introduced Lumi as an example of this emerging architectural pattern: The observability warehouse.
Rather than replacing existing observability platforms, the system introduces a separate storage and compute layer designed specifically for telemetry analysis.
The concept draws inspiration from traditional data warehouses used in business intelligence but is purpose-built for logs, metrics, and traces.
Key design principles include:
- Decoupled storage and compute
- Support for large-scale historical data
- Elastic scaling for investigative workloads
- Integration with existing observability tools
Importantly, Marshall emphasised that the goal is not to introduce new interfaces or languages.
In many deployments, analysts continue using existing platforms such as:
- Splunk
- Grafana
- Elastic Stack
Queries written in familiar languages (such as SPL or SQL) can continue to function while being executed against a different underlying infrastructure layer.
Interoperability as a Core Requirement
Marshall highlighted interoperability as a critical design principle.
Security teams often rely on multiple tools across the observability ecosystem. Introducing a completely new interface or workflow can create operational disruption.
Instead, decoupled architectures aim to preserve existing workflows while improving performance and scalability behind the scenes.
For analysts, the user experience remains largely unchanged. The improvements occur within the underlying infrastructure.
Data Retention and the AI Factor
A discussion between Marshall and moderator Peter Dorrington also explored the question of how much data organisations should retain.
With the emergence of artificial intelligence and machine learning, data that previously appeared unimportant may later become valuable for model training or retrospective analysis.
However, retaining large datasets also introduces risk. Excessive data collection can increase the attack surface if sensitive data is stored without adequate controls.
Marshall noted that determining which data to retain ultimately depends on the organisation’s use cases and governance policies.
Key Takeaways
Several important themes emerged from the session:
- Telemetry data volumes are increasing rapidly due to cloud-native infrastructure and distributed systems.
- Traditional observability platforms often struggle because they attempt to support both monitoring and investigative workloads simultaneously.
- Detection and investigation workflows require different architectural approaches.
- Vertically integrated “black box” platforms can lead to cost inefficiencies and performance contention.
- Decoupling storage, compute, and analytics layers enables greater scalability and flexibility.
- Observability architectures are beginning to adopt patterns that already exist in data analytics and business intelligence.
Frequently Asked Questions (FAQ)
What challenges are modern observability systems facing in cyber security?
Modern observability systems are struggling to keep up with the rapid growth of telemetry data generated by cloud-native infrastructure, microservices, and containerised environments. These systems must process massive volumes of logs, metrics, and traces while maintaining performance. At the same time, organisations are expected to retain data for longer periods to meet governance, compliance, and investigative requirements. Traditional architectures often cannot scale efficiently to meet both demands.
Why is telemetry data volume increasing so quickly?
Telemetry data volume is increasing due to the widespread adoption of distributed systems, Kubernetes environments, and microservices architectures. These technologies generate continuous streams of operational data describing system behaviour, application performance, and user interactions. As organisations scale their digital infrastructure, the number of events captured across platforms grows exponentially.
What is the difference between detection workflows and investigation workflows?
Detection workflows are designed for continuous monitoring and alerting. They typically involve predictable queries, dashboards, and automated alerts that monitor recent system activity. Investigation workflows are different because they require analysts to explore historical data, run complex queries, and correlate information across multiple datasets. These investigative tasks are more unpredictable and often require significantly more compute resources.
Why can traditional observability platforms struggle with investigation workloads?
Many observability platforms use vertically integrated architectures where data ingestion, storage, computation, and visualisation are tightly coupled. When investigation workloads introduce large queries across long time ranges, these systems can become overloaded. As a result, monitoring performance may degrade or organisations may be forced to overprovision infrastructure to handle occasional spikes in demand.
What does decoupling mean in observability architecture?
Decoupling refers to separating different layers of the observability stack so they can scale independently. Instead of bundling storage, compute, and analytics together, a decoupled architecture allows organisations to optimise each component based on the specific workload. For example, monitoring systems can handle real-time detection while large-scale storage and compute systems support deeper investigations.
What is an observability warehouse?
An observability warehouse is a data platform designed specifically for storing and analysing large volumes of telemetry data. It separates storage and computation so organisations can run complex queries across long-term datasets without affecting operational monitoring systems. This architecture enables security teams to retain more data, perform large-scale investigations, and scale resources dynamically when needed.
How does decoupling improve scalability and cost efficiency?
Decoupling allows organisations to align infrastructure resources with specific workload requirements. Detection workflows can run on smaller, optimised systems that manage recent data, while investigative workloads can access larger datasets stored in cost-effective object storage. Compute resources can scale up during investigations and scale down afterwards, reducing the need for constant overprovisioning.
Should organisations store all security telemetry data indefinitely?
Not necessarily. While storing large volumes of telemetry data can support investigations and emerging use cases such as AI analysis, organisations must balance retention with cost, governance, and security risk. Decoupled architectures make long-term storage more feasible by using scalable object storage while still allowing analysts to query the data when required.
Watch the Full Session On-Demand
If you would like to hear the full discussion between Peter Dorrington and Peter Marshall, the session is available to watch on demand:
CyberSecure Online Summit | Virtual Event for Cybersecurity Leaders
The replay includes additional insights from cybersecurity experts discussing identity risk, cloud security and emerging threat trends.
If you would like to understand more about Imply Lumi contact: Request a Lumi Demo
#cybersecurityobservability #observabilityarchitecture #telemetrydata #realtimeanalytics #ApacheDruid #ImplyPolaris #observabilitywarehouse #detectionworkflows #investigationworkloads #telemetrydataretention #cloudnativesecurity #Kubernetesobservability