Cloud-Native SIEM and the Security Data Lake: AWS Security Lake, Databricks Lakewatch, and Why Your Splunk Bill Is About to Change Forever

I have been buying and operating SIEM platforms since the mid-2000s, when the choice was essentially ArcSight or Splunk and the main differentiator was how fast each one could bankrupt you. Twenty years later the category is finally undergoing a genuine architectural transformation, not a rebrand. The security data lake pattern, backed by open standards like the Open Cybersecurity Schema Framework (OCSF), is forcing a reckoning with every assumption that legacy SIEM was built on. In 2026, if you are still making licensing decisions the old way, you are probably spending three to five times what you need to.

This is the piece I wish someone had handed me in 2024 when I started evaluating AWS Security Lake, Microsoft Sentinel’s new data lake tier, and the freshly announced Databricks Lakewatch. The terminology is slippery, the vendor marketing is aggressive, and the right architecture depends heavily on your existing data platform investments. Let me cut through it.

The Legacy SIEM Tax

Traditional SIEM was designed around two assumptions that made sense in the on-premises era: you owned a fixed set of systems, and log volume was predictable enough to capacity-plan against. Both assumptions are wrong in a cloud-native environment.

I ran a Splunk deployment for a mid-size financial services company a few years back. We were ingesting about 80 GB per day, mostly from on-prem network devices, Active Directory, and a few Linux servers. We had our licensing bill, it was painful but manageable. Then the team started deploying on AWS. VPC Flow Logs alone added 200 GB per day. CloudTrail added another 50 GB. Guard Duty findings, EKS audit logs, WAF logs: suddenly we were at 700 GB per day. The renewal quote arrived. I called the account rep. That conversation did not go well.

The fundamental problem with legacy SIEM is that it conflates storage, search, and analytics into a single proprietary system and then charges you for all three at once. You cannot separate cheap cold storage from expensive hot search. You cannot bring your own analytics engine. You cannot normalize data once and query it with three different tools. Everything flows through the vendor’s pipeline, and every gigabyte pays the toll.

The security data lake pattern fixes this by decoupling storage from compute, the same insight that Apache Iceberg and the data lakehouse model brought to analytics workloads. You land data once, in open formats on cheap object storage, and you layer query engines and detection tools on top without re-ingesting anything.

What OCSF Actually Is (and Why It Matters)

The biggest practical problem with building your own security data lake used to be normalization. Every vendor has its own log format. Palo Alto Networks firewall logs look nothing like CrowdStrike endpoint telemetry, which looks nothing like AWS CloudTrail events. Before you can run any cross-source correlation, you have to build and maintain a normalization pipeline. That is miserable, fragile work.

The Open Cybersecurity Schema Framework, originally developed by Splunk and AWS and now governed by the OCSF community, is an attempt to solve this once. OCSF defines a taxonomy for security events. Every event has a category (Network Activity, Findings, Identity and Access Management, and so on), a class within that category, and a set of required and optional attributes. A network connection event from CrowdStrike and a VPC Flow Log from AWS map to the same OCSF class with the same field names.

OCSF is not perfect. Version 1.x coverage is strong for common event types and gets thin for niche ones. Some vendors implement it loosely. But it is good enough that AWS built AWS Security Lake on top of it, and Datadog added an OCSF processor to its cloud SIEM in 2025. When the hyperscalers converge on a schema, it tends to become the standard. I am comfortable betting on it.

The schema is hierarchical. At the top you have the base event with common fields: time, severity, message, metadata. Then you have the category object, then the class object, each adding more specific fields. OCSF also handles extensions gracefully: vendors can add proprietary fields without breaking parsers that do not know about them, because everything is structured rather than free-form.

In practice, this means a properly normalized security data lake can answer cross-source questions that would have required expensive SIEM correlation rules or manual analyst work:

• Show me all privilege escalation events (any source) that occurred within 10 minutes of a lateral movement detection on the same host
• Find all authentication events where the user logged in from a country they have never used before, correlated with data exfiltration findings from the same account

These queries, run against normalized OCSF data in something like Athena, DuckDB, or ClickHouse, can complete in seconds and cost fractions of a cent.

AWS Security Lake: Architecture and Reality

AWS Security Lake, GA’d in 2023 and significantly matured by 2026, is the most prescriptive of the three architectures I’m going to cover. It is also the easiest on-ramp if you are already heavily AWS-native.

Security Lake automatically pulls in logs from the following AWS sources with zero configuration beyond enabling it: CloudTrail management events, CloudTrail S3 data events, VPC Flow Logs, Route 53 resolver query logs, Security Hub findings, Guard Duty findings, AWS WAF logs, and Lambda execution logs. It normalizes all of them to OCSF and writes them to your designated S3 bucket in Apache Parquet format, partitioned by region, account, event class, and day.

The subscriber model is clever. You designate Security Lake as the central store, and then downstream consumers subscribe to specific event classes. A SIEM like Microsoft Sentinel, a detection platform like Panther, or your own Athena queries can subscribe and be notified when new data arrives. You do not pay for Security Lake to replicate data to each consumer; each subscriber reads from S3 directly. This is fundamentally different from the legacy model where the SIEM was the store.

The operational reality is that third-party source normalization is where Security Lake earns its complexity badge. Getting CrowdStrike or Palo Alto data in requires a custom source configuration and, frequently, a Lambda function or Firehose transform to normalize non-AWS sources to OCSF. AWS has a partner ecosystem for this, and it is growing, but plan for integration work.

The cost model is dramatically better than legacy SIEM for high-volume data. You pay for:

• S3 storage (roughly $0.023 per GB per month for standard, much less for Glacier Instant Retrieval)
• Security Lake service fee (a per-event normalization charge, usually pennies per million events)
• Query costs when you actually run Athena queries

For a customer generating 500 GB of security logs per day, the total Security Lake cost typically runs $3,000 to $8,000 per month depending on retention and query frequency. A comparable Splunk deployment at that volume would run $40,000 to $80,000 per month. That delta pays for a lot of detection engineering.

The piece that Security Lake does not provide is a detection engine. Security Lake stores and normalizes data. What actually finds threats is your query layer. For automated detection, you need Athena scheduled queries, Amazon OpenSearch, a third-party platform like Panther, or Databricks on top of the lake.

Microsoft Sentinel’s Data Lake Tier

In early 2026, Microsoft made the strategic move that security architects had been waiting for: Sentinel introduced a built-in security data lake tier that stores logs at dramatically lower cost, separate from the expensive hot-tier Analytics workspace.

The architecture now has three tiers within Sentinel. Hot Analytics logs are the traditional Sentinel experience, fully indexed, available for KQL queries in seconds, priced at the legacy per-GB rate. The new Basic logs tier is a cheaper, slower tier for verbose operational data where you want to retain it but rarely query it. The Auxiliary logs tier (the new data lake tier) writes directly to ADLS Gen2 storage behind the scenes, at object storage prices, and you run ad-hoc queries against it with Sentinel’s search job capability.

Sentinel’s advantage is the integrated SIEM/SOAR experience. If your security operations team lives in Sentinel, adding the data lake tier for cheap high-volume retention while keeping the expensive hot tier for critical events is an evolution, not a migration. You get unified dashboards, analytics rules, automated playbooks, and Copilot for Security integration, all from one platform.

The disadvantage is lock-in. Sentinel’s native schema (Log Analytics schema plus ASIM) is not OCSF. OCSF support is on the roadmap but not generally available as of mid-2026. If you want to federate your security data lake with non-Microsoft tooling, you will be doing schema translation work that pure OCSF deployments avoid.

Microsoft’s strength is also in the identity and productivity correlation layer. Tying Entra ID sign-in patterns to Exchange Online anomalies to Defender for Endpoint findings within a single Sentinel workspace is genuinely easier than trying to accomplish the same federation across three separate OCSF sources. If you are a Microsoft-heavy shop, especially if you have E5 licenses, Sentinel is probably still your anchor.

Databricks Lakewatch: The Wildcard

In May 2026, Databricks announced Lakewatch: an agentic SIEM built natively on the Databricks lakehouse platform. This is the most interesting new entrant, and I have been watching it carefully since the announcement.

Lakewatch is built on Delta Lake (which has strong Apache Iceberg interoperability via UniForm), runs detection as Python or SQL notebooks, and adds an agent layer that can investigate findings autonomously, correlate across data sources, and generate remediation suggestions. The core pitch: if your security team is already doing data engineering in Databricks, your threat detection can live in the same platform as your data pipelines.

The OCSF normalization story is strong here. Databricks has a mature partner ecosystem for security data ingestion, and Lakewatch ships with OCSF normalization connectors for the major sources. You can point Lakewatch at an existing AWS Security Lake S3 bucket and have it query native OCSF data without re-ingesting.

The agentic angle is genuine, not marketing. I have seen demos where Lakewatch takes a detection alert, uses Genie (Databricks’ NLP-to-SQL engine) to query related events across the lake, and assembles a timeline of the attack chain without analyst prompting. Whether this is ready for production SOCs today is debatable, but the direction is clearly right. Security analysts spend most of their time on mechanical correlation work. An agent that does the first-pass investigation and surfaces only the events that require human judgment changes the economics of a SOC significantly.

Building Your Own: The Security Lakehouse Pattern

If you have an existing data platform investment and do not want to pay a SIEM vendor to manage storage you can manage yourself, the DIY security lakehouse is the approach. Here is how I would build it in 2026.

Ingestion layer. Use a streaming pipeline to collect logs from all sources. Kafka or AWS Kinesis Firehose for high-volume sources. AWS EventBridge Pipes for native AWS events. Vector (from the observability pipeline space) works well for host-based log shipping and has OCSF processors.

Normalization. Write OCSF normalization transforms in Spark or in a Flink streaming job. Run them as the data lands, not at query time. Store normalized Parquet in S3 or GCS, partitioned by OCSF category, class, and event day. Use Apache Iceberg as the table format for time-travel, schema evolution, and partition pruning.

Query layer. For interactive threat hunting, ClickHouse on a hot copy of the last 30 days is excellent. ClickHouse’s columnar storage makes security correlation queries extremely fast. For long-term retention and ad-hoc queries, Athena or Trino on the cold Iceberg tables.

Detection. Run scheduled detection queries on the normalized data. For simple pattern-based detections, Athena scheduled queries writing findings to a DynamoDB findings table work fine. For complex statistical detections (anomaly detection on authentication patterns, network baseline deviations), a small Spark job or a Databricks notebook running hourly is more appropriate.

Alerting. Write findings to a centralized findings table in OCSF Findings class format. Route high-severity findings to PagerDuty or your incident management platform. Lower severity goes to a Slack channel or a queue for analyst review.

The total cost of a well-built DIY security lakehouse at 500 GB/day of logs, 1-year retention for hot data and 7-year for cold, typically runs $5,000 to $12,000 per month. You are trading vendor convenience for cost control and avoiding the normalization tax you pay every time you want to switch tools.

One thing the DIY approach requires that the managed platforms handle for you: compliance chain-of-custody. Your data catalog needs to know what data landed, when, and whether it was tampered with. Use S3 Object Lock (WORM) for compliance retention, log inventory reports for completeness checking, and OpenMetadata or DataHub to document data lineage from source to detection. Auditors will ask for this, particularly under data residency and sovereignty requirements.

Threat Hunting Patterns on Normalized Data

Once you have OCSF-normalized data in a queryable store, the threat hunting capability is genuinely transformative. Here are the query patterns I use regularly.

Credential stuffing detection. Query authentication events grouped by source IP, bucket by 5-minute windows, flag any IP with more than 50 login attempts across more than 10 distinct usernames. With OCSF’s Authentication class, the relevant fields (src_endpoint.ip, user.name, status) are standardized across your identity providers, VPN authentication logs, and application SSO events. One query, all sources.

Lateral movement correlation. Join Network Activity events (class 4001) with Authentication events (class 3002) on time and host. A sequence of successful authentication to a host followed within 5 minutes by outbound connections to new internal subnets is a classic lateral movement signal. Window functions over Iceberg tables handle this cleanly.

Data staging detection. Join DNS resolution events with S3 API events. An EC2 instance that has never made S3 API calls starting to make high-volume PutObject calls to a bucket it has never accessed before, combined with an outbound DNS lookup to an unrecognized domain, is a data exfiltration precursor. Again: one query across normalized OCSF data, sources that would have required three separate SIEM datasources before.

The constraint is detection latency. In a managed SIEM, detections often run continuously against a streaming index. In a data lake, you are typically running batch queries every 5 to 15 minutes against landed data. For many detection use cases, 15-minute latency is fine. For active ransomware or wiper activity, it is not. Hybrid architectures that use a streaming detection layer (Kafka Streams, Flink, or AWS Kinesis Data Analytics) for high-severity real-time patterns, combined with a batch lake for retrospective hunting, thread the needle.

Cost Comparison: Legacy SIEM vs. Security Data Lake

I want to give you a concrete cost comparison because the numbers are the most convincing part of the argument.

Assume 500 GB per day of log volume, 90-day hot retention, 7-year cold retention for compliance.

Splunk Cloud at this volume: roughly $70,000 to $120,000 per month depending on license tier and your negotiating leverage. I have seen contracts at both ends of that range.

AWS Security Lake plus Athena for queries: approximately $4,000 to $7,000 per month including the Security Lake service fee, S3 storage costs for both hot and cold tiers, and realistic Athena query volumes. Add $3,000 to $5,000 per month for a third-party detection platform like Panther that sits on top.

Databricks Lakewatch: pricing is still maturing, but based on early customer quotes, expect $8,000 to $15,000 per month for the same volume including the Databricks compute and the Lakewatch platform fee. More expensive than the DIY approach but cheaper than Splunk, and you get the agentic investigation capabilities.

Microsoft Sentinel with the new data lake tier: highly variable because it depends heavily on which data sources you already have covered by E5 licensing. If your Entra ID and M365 data is free under your license deal, and you are only paying Sentinel rates for third-party sources, the comparison gets more complex. For AWS-heavy organizations, Sentinel tends to run $15,000 to $30,000 per month at this volume.

None of these numbers are warranties. They depend heavily on your specific log mix (verbose data like packet captures or full request/response API logs changes the math dramatically), query patterns, and vendor negotiation. But the direction is consistent: the security data lake approaches are significantly cheaper than monolithic SIEM for high-volume cloud-native environments.

What This Means for Your Security Stack

The practical decisions you need to make depend on your organizational context.

If you are AWS-native and you have no existing SIEM contract: start with AWS Security Lake for normalization and cheap storage. Layer Panther or Detective on top for detection. Do not build a custom SIEM unless you have the engineering capacity to maintain it.

If you are Microsoft-heavy with E5 licenses: maximize what Sentinel gives you for free via the license. Adopt the new data lake tier for high-volume verbose logs. Do not migrate away from Sentinel just because of the OCSF narrative; the integrated identity correlation is worth more than the normalization portability for most Microsoft shops.

If you have a data platform team that already runs Databricks: Lakewatch deserves a serious evaluation. Having your security data co-located with your business analytics data in the same lakehouse is a genuine advantage for insider threat detection and fraud use cases that require joining security events with business context.

If you are mid-size, resource-constrained, and need a SIEM that actually works out of the box: do not over-engineer this. A managed SIEM is still a reasonable choice. The security data lake pattern requires engineering capacity to implement and maintain, and a poorly maintained cheap SIEM is worse than an expensive one that actually generates alerts your team reviews.

For any of these paths, the integration with your zero-trust security architecture matters. Your SIEM or security data lake should be receiving identity provider logs, network policy enforcement logs, and workload identity events. The OCSF Findings class is particularly useful here: your CNAPP tools, your cloud-native application protection platform, and your runtime security tools should all be emitting OCSF Findings that land in the same lake.

The Role of Secrets and Credential Telemetry

One category of security telemetry that most organizations underinvest in: secrets and credential lifecycle events. Your secret management platform should be emitting every secret access, rotation, and revocation event to your security lake. HashiCorp Vault’s audit device, AWS Secrets Manager CloudTrail events, and GCP Secret Manager audit logs all have OCSF mappings.

Why does this matter? The sequence “secret accessed, secret not rotated in 30 days, outbound connection to unfamiliar IP” is a high-fidelity indicator of compromised credentials. You can only detect this sequence if the secret lifecycle events are in the same query space as the network activity events.

Where This All Goes

I expect the security data lake to become the default architecture for mid-to-large organizations within three years. The economics are too compelling, the OCSF standard is reaching critical mass, and the managed on-ramps (AWS Security Lake, Sentinel data lake tier, Lakewatch) are removing the engineering complexity that made DIY security lakehouses impractical.

What does not go away: the SIEM experience layer. Analysts need dashboards, correlation rules they can write without SQL, automated playbooks for common response actions, and case management. The fight in this market is whether that experience layer will be provided by cloud vendors (AWS, Microsoft, Databricks) or by specialist platforms (Panther, Chronicle, Exabeam) sitting on top of open storage. My bet is both: Microsoft Sentinel survives because of the M365 integration moat, and open platforms like Panther on AWS Security Lake grow for organizations that value vendor independence.

For cloud architects, the immediate implication is design. Stop sending all logs to a single SIEM. Start designing data flows that land in normalized object storage first, and layer detection on top. Your storage bill will drop. Your detection flexibility will increase. And when the next SIEM vendor has a pricing crisis or an acquisition, you will have data you actually own.

Twenty years of SIEM wars, and finally the data is free.