Cloud Architecture

Webhook Infrastructure at Scale: Svix, Hookdeck, Convoy, and Building Reliable Event Delivery for Your Platform

How to build production-grade outbound webhook infrastructure using Svix, Hookdeck, and Convoy, covering delivery guarantees, signing, retries, and when to roll your own.

Diagram showing webhook delivery pipeline with retry queues, signing, and consumer endpoints

I have watched engineers build the same webhook system from scratch no fewer than a dozen times over twenty years, and it always goes the same way. It starts with a simple POST in a background job. Then a customer complains they missed an event. Then someone realizes the background job silently dropped failures. Then someone adds a retry table to the database. Then that table becomes the slowest thing in the system, and the whole mess gets handed off to a newer team member who eventually asks why there is a webhook_deliveries table with eleven million rows in production.

Webhooks are deceptively simple and operationally brutal. Getting them right is not about the HTTP call itself. It is about every failure mode around it: the customer’s endpoint being down, their load balancer returning a 429, your app server being mid-deploy when you send, the event queue backing up, customers wanting replays, your security team wanting audit logs of every delivery. This is a full infrastructure problem that deserves a real infrastructure answer.

This guide covers what production webhook delivery actually requires, the tools that have emerged to solve it without building from scratch, and the architectural decisions that determine whether your platform’s event delivery is a competitive advantage or a continuous source of support tickets.

Why Webhooks Are Harder Than They Look

The naive implementation fires an HTTP POST to the subscriber URL inline or in a background worker. This breaks in ways that are not obvious until production. Customer endpoints return 500s intermittently. They return 200s but are actually dropping the event on their side. They go down for thirty minutes during a maintenance window. Your deployment introduces a brief period where workers are restarting. Any of these conditions silently loses events.

The fundamental problem is that HTTP is not a reliable transport. It is a request-response protocol with no built-in delivery guarantee. When you use it as a pub-sub mechanism, you inherit all the failure modes of TCP connections plus all the failure modes of the customer’s application stack, with no retry semantics at the protocol level.

A production webhook system needs to handle several distinct concerns simultaneously:

Delivery guarantees. At-least-once delivery is the right target. Exactly-once is theoretically impossible without coordination on both sides and practically expensive even with it. At-least-once means you must retry on failure, and consumers must be built to handle duplicates. This is not optional. Any webhook system that does not retry on failure is at-most-once delivery, which means it silently drops events on any transient failure.

Backpressure and rate limiting. Customers who implement tight rate limits on their own endpoints will start returning 429s the moment you send a burst. Your retry logic needs to respect Retry-After headers, implement exponential backoff, and not hammer a struggling endpoint into the ground.

Ordering guarantees. Most webhook consumers do not require strict ordering, but some do. A payment platform where customers listen for payment.created, payment.captured, and payment.refunded genuinely cannot process events out of order. Providing per-consumer FIFO ordering without blocking delivery to other consumers is a non-trivial queuing problem.

Security. Every webhook payload needs an HMAC signature so the receiver can verify it came from you and was not tampered with. The Standard Webhooks specification, an open-source community initiative now supported by over a hundred SaaS platforms, standardizes the headers (webhook-id, webhook-timestamp, webhook-signature) and the HMAC-SHA256 signing scheme. Without signatures, your customers have no way to authenticate that an incoming request actually came from your platform, which is the same problem every NHI credential has.

Observability and debugging. When a customer says “I never received the payment captured event from last Thursday,” you need to be able to prove that you sent it, show them the payload, the response code, every retry attempt, and what their endpoint returned. This is not a nice-to-have. It is a customer support and compliance requirement.

Customer-facing management. Customers need to rotate their signing secrets, enable and disable endpoint URLs, test with synthetic payloads, and filter which event types they receive. Building a UI and API for this is weeks of product work that has nothing to do with your core product.

Architecture diagram: event queue, webhook dispatcher, retry scheduler, and customer endpoints

The Standard Webhooks Specification

Before evaluating tooling, it is worth understanding the Standard Webhooks spec because it has become the common foundation. The spec defines three required HTTP headers that every webhook delivery should include:

  • webhook-id: A stable, unique identifier for this event. The same ID is used on every retry, allowing consumers to deduplicate.
  • webhook-timestamp: Unix timestamp in seconds when the event was created.
  • webhook-signature: A comma-separated list of HMAC-SHA256 signatures over the concatenation of the message ID, timestamp, and body.

The signature construction is ${msgId}.${timestamp}.${body}, hashed with the consumer’s signing secret. The timestamp is included in the signed content specifically to defeat replay attacks: a consumer should reject any webhook with a timestamp outside a reasonable window (Svix’s SDK defaults to five minutes in each direction).

The rationale for a comma-separated list of signatures is key rotation. When a consumer needs to rotate their signing secret, you can include signatures for both the old and new secret during the transition window. The consumer accepts delivery as long as any of the listed signatures is valid, then transitions fully to the new secret without any missed events.

This specification is what replaced the chaotic landscape where every SaaS had its own signing scheme, its own header names, and its own signing documentation. If you are building outbound webhooks today and you are not following this spec, you are creating unnecessary integration friction for your customers.

The Core Architecture: What You Are Actually Building

Whether you use a managed service or build your own, the architecture you need has the same logical components.

An event store that records every event you intend to deliver, durably, before attempting delivery. This is the classic transactional outbox pattern. The event record and the domain state change must be committed atomically, or you will have race conditions where your database transaction commits but the event is never sent because the process crashed before enqueuing.

A fanout layer that takes a single event from your domain and creates one delivery attempt record per registered endpoint. If a customer has three endpoint URLs registered, you create three delivery attempts. This decouples the number of registered consumers from the domain event production path.

A delivery worker that attempts the HTTP POST, records the response (status code, response body, latency), and determines whether to mark the attempt successful or schedule a retry. The standard backoff schedule is roughly: 5 seconds, 30 seconds, 2 minutes, 5 minutes, 10 minutes, 30 minutes, 2 hours, with some jitter added to avoid thundering herds.

A retry scheduler that picks up failed delivery attempts that have been waiting long enough for their next retry and feeds them back to the delivery worker. This needs to be separate from the initial delivery path so a backlog of retrying events does not delay fresh event delivery.

An endpoint management API that handles customer configuration: registering endpoints, setting event type filters, rotating secrets, manually disabling/enabling endpoints, and retrieving delivery history.

A consumer-facing portal that wraps the management API in a UI for customers who do not want to manage this via API.

The code volume here is substantial. The delivery worker alone needs to handle connection timeouts, TLS certificate errors, DNS failures, response parsing, 1xx through 5xx handling (treat 408, 429, 5xx as retriable; treat 4xx except 429 as permanent failure), and customer-defined rate limits. I have seen teams spend three months building this correctly, only to discover they also need to solve the portal problem.

Svix: The Managed Outbound Webhook Platform

Svix is the most polished managed solution for outbound webhooks. The core server is open source under the MIT license, but the production value comes from the managed cloud service, which adds infrastructure reliability, geographic distribution, and a genuinely good embeddable portal.

The core concept in Svix is the Application. Each Application represents one of your customers, and you create one per customer when they sign up. Applications contain endpoints, each with its own filtering, signing secret, and delivery history. Svix’s API surface is clean enough that creating an application and registering an endpoint is about five lines of code in any of their SDKs.

Svix handles a cluster of concerns well:

Message ordering. Svix supports FIFO ordering per application, which means per customer. Events sent to a given customer are delivered in order, and a failed delivery does not block delivery to other customers. For teams that need total ordering within a customer’s stream, this is valuable.

Transformations. You can run serverless JavaScript functions on Svix’s infrastructure to transform the payload before delivery. This matters when customers have webhook consumers that expect a slightly different schema than your canonical event format, without requiring them to run their own translation layer.

Polling. For customers who cannot accept inbound HTTP connections, Svix offers a polling API that exposes the same event stream over a standard GET endpoint. This is surprisingly useful for enterprise customers with strict firewall policies that block inbound webhooks from cloud IPs.

The embeddable portal. Svix’s AppPortal is an embeddable UI that you can drop into your own dashboard with a few lines of code. Customers get a full management interface for their endpoints, complete with event replay, delivery logs, and signing secret rotation, all scoped to their own Application. This is typically weeks of product work that Svix eliminates entirely.

The tradeoffs are that you are trusting Svix with every webhook payload you produce, which matters if you handle regulated data. For HIPAA-regulated healthcare platforms, Svix’s BAA availability is a requirement before you can use the managed cloud. Their open-source server allows full self-hosting, which solves the data residency concern but puts the operational burden back on you.

Svix’s pricing is message-based at higher tiers. The entry paid plan starts at a price point that makes sense for established SaaS products but can be steep for a startup evaluating the product, so check their current pricing at svix.com before budgeting. The free tier exists for initial integration and testing.

Hookdeck: Bidirectional Webhook Management

Hookdeck takes a different approach. Where Svix is squarely focused on outbound webhook delivery (your platform sending webhooks to customers), Hookdeck is built for the full picture: receiving webhooks from third-party services AND sending them to your customers.

The receiving side of Hookdeck is genuinely useful. If your platform needs to ingest webhooks from Stripe, GitHub, Shopify, Salesforce, and a dozen other third-party services, managing the inbound side is its own operational problem. Hookdeck gives you a stable inbound URL, handles the fan-out to your internal services, adds filtering and transformations, and maintains a complete log of everything received. This solves the problem of a third-party service retrying a webhook that your internal service already processed, or debugging why an event from Stripe is missing from your system.

Hookdeck added an outbound product called Outpost specifically to compete with Svix on the sending side. It covers the Standard Webhooks spec, delivery guarantees, and customer management. If you need both inbound and outbound webhook management and want a single vendor, Hookdeck is worth evaluating.

One place Hookdeck genuinely stands out is its observability model. The event inspector gives you a timeline view of every webhook event, the original payload, any transformations applied, every delivery attempt, and the full HTTP response from the destination. When a customer opens a ticket saying they did not receive an event, the investigation workflow in Hookdeck is significantly faster than digging through raw delivery logs.

The bidirectional architecture also makes Hookdeck a natural fit if your platform acts as both a consumer and a producer of events, which describes most modern integration platforms.

Hookdeck bidirectional webhook flow: inbound from third parties, transformation layer, outbound to customers

Convoy: The Self-Hosted Option

Convoy is a source-available webhook gateway under the Elastic License 2.0 (ELv2). The ELv2 license allows self-hosting and modification but prohibits offering Convoy itself as a managed service to others. If your compliance requirements, data residency constraints, or cost structure make managed services difficult to justify, Convoy is a strong self-hosted option. If you need a fully open-source (Apache 2.0) alternative, Hookdeck Outpost is now available as a standalone open-source outbound webhook service.

Convoy’s feature set covers the core requirements: fan-out to multiple endpoints, automatic retries with exponential backoff, signature verification, event types and filtering, and a management UI. It supports both REST and message queue sources (you can feed Convoy from a Kafka or Amazon SQS topic, which fits naturally into event-driven architectures built around stream processing).

The Kubernetes deployment path for Convoy is well-documented. You bring your own PostgreSQL instance for event storage, Redis for the retry queue, and a worker pool sized to your delivery throughput. This is a real operational commitment. You need to size the worker pool for peak delivery load, monitor queue depth, manage Postgres storage growth as delivery history accumulates, and handle the operational overhead of Redis. Teams that underestimate this maintenance burden typically end up spending more engineering time on Convoy operations than they would have spent on a managed service’s subscription fee.

Where Convoy genuinely wins is total control. Your webhook payloads never leave your infrastructure. You can audit the delivery code. You can tune the retry policy, the timeout behavior, the concurrent delivery limit per endpoint, and the storage retention policy exactly to your needs. For a highly regulated platform, this control is worth the operational cost.

One practical consideration: Convoy does not include a fully-featured embeddable customer portal out of the box. If your customers need a UI for managing their webhook endpoints, you will be building that yourself. Svix’s portal is probably the strongest single argument for the managed service over self-hosted alternatives.

Sequin: The Database-Centric Alternative

Sequin deserves a mention as a different conceptual model. Rather than treating webhook delivery as a separate queue service, Sequin exposes database tables as event streams and delivers changes via webhooks. If you have an existing PostgreSQL database and want to add webhook delivery for record changes without building an event sourcing layer, Sequin uses logical replication to capture changes and deliver them.

This is a niche fit but a strong one. For platforms where the events you want to deliver are fundamentally database row changes, Sequin eliminates the need to both write to the database and publish to a separate event queue. The tradeoff is coupling your event schema to your database schema, which works well for simple CRUD event streams but becomes limiting as event semantics get more complex.

Building Your Own: When It Makes Sense

After twenty years, I am not reflexively opposed to building webhook infrastructure in-house. There are specific conditions where it makes sense:

You are a large platform with event volume that makes per-message pricing economically irrational. You have specific delivery semantics that no managed service supports (unusual ordering guarantees, custom retry schedules driven by customer-tier SLAs, exotic authentication requirements). You are operating in a regulated environment where you cannot send payload data to a third party regardless of BAA availability.

If you build your own, the architecture guidance earlier in this article is the right foundation. The specific mistakes I see teams make repeatedly:

Using the application database as the retry queue directly. The webhook_deliveries table pattern fails at scale because every delivery worker poll is a full table scan on the failed_pending rows. Use Redis or a proper job queue with priority support.

Not separating initial delivery from retry delivery. Mixing fresh events with retrying events in the same queue means a customer whose endpoint has been failing for two hours can slow down fresh event delivery to working endpoints. Separate queues with separate workers.

Storing full payloads in the retry queue instead of just IDs. Large payloads in Redis are expensive. Store the event ID in the retry queue, fetch the payload from the event store when processing.

Not enforcing a maximum retry window. Retrying indefinitely is an antipattern. Set a maximum age (seven days is a common default) after which delivery attempts are permanently failed. Customers need to know there is a window for endpoint recovery, not an infinite safety net.

Skipping the delivery history API. The hardest customer support tickets are “I never received that event from two weeks ago.” If you do not retain delivery history with full payloads and response codes, you cannot help them. Retention cost is real but unavoidable.

For event-driven architectures that use Kafka or similar streaming infrastructure internally, the webhook delivery layer is an outbound adapter that translates from your internal stream to customer-facing HTTP. This mapping is clean and testable. The AsyncAPI specification is a useful tool for documenting what event types your platform emits and what the payload schema is for each type, which is exactly the documentation your customers need when implementing their webhook consumers.

Delivery Semantics and Consumer Requirements

One thing managed services cannot solve for you: your customers’ webhook consumers must be idempotent. At-least-once delivery is not a limitation, it is a correct behavior. The webhook-id header from the Standard Webhooks spec gives consumers a stable deduplication key. A consumer that processes order state changes should check whether it has already processed a given webhook-id before applying the state transition.

If your customers routinely complain about duplicate event processing, the problem is usually that their consumers are not checking the webhook-id. The right fix is documentation and SDK examples that show deduplication, not attempting to build exactly-once delivery into your infrastructure.

For compliance and audit requirements, consider treating your webhook delivery history as an append-only log. Some regulated platforms need to demonstrate to auditors that every event was delivered successfully or that every failed delivery was investigated. An immutable delivery log with timestamps, response codes, and payload hashes satisfies this without requiring you to retain full payloads indefinitely.

Webhook consumer implementation showing idempotency check with webhook-id deduplication key

Choosing Between Managed Service and Self-Hosted

The right answer depends on where you are in the product lifecycle and what your constraints are.

For most teams building a SaaS product, Svix or Hookdeck will save months of engineering time and produce a better customer experience than an in-house implementation. The portal alone is worth the cost for a product team that does not want to build webhook management UI. The managed reliability, the monitoring, and the customer support tooling are real value.

For teams with data residency requirements, regulated data handling, or extreme cost sensitivity at high event volumes, Convoy’s self-hosted model is the right starting point. The operational cost is real but predictable once the system is running.

For teams using complex internal event-driven architectures built around Apache Kafka or similar systems, the webhook delivery layer is a relatively thin outbound adapter, and building it in-house is more practical because you are integrating it with existing streaming infrastructure rather than adding a new system entirely.

The Standard Webhooks specification is worth adopting regardless of which path you take. Standardizing your signing headers, your retry behavior, and your deduplication strategy means your customers’ integration code is portable across services that follow the spec, and your documentation has a canonical reference rather than custom signing instructions that every integration has to decipher.

Security Hardening for Production Webhook Delivery

Webhook delivery has a surface area that is easy to overlook. A few specific requirements:

TLS verification. Always verify the customer endpoint’s TLS certificate. Never offer an option to disable certificate verification, regardless of how many enterprise customers claim they cannot fix their self-signed certificates. If they can configure a custom CA, offer that path instead.

IP allowlisting. Publish a stable list of source IPs from which your webhook deliveries originate. Enterprise customers routinely need this for firewall configuration. Managed services like Svix provide static source IPs; if you are running your own delivery workers, you need Elastic IPs or similar static addressing on your worker pool.

Payload size limits. Enforce a maximum payload size (1MB is common) and document it. Customers whose events grow over time will eventually hit this limit, and hitting it silently is worse than refusing it at event publication time with a clear error.

Signing key rotation. Implement signing key rotation from day one. The Standard Webhooks spec’s multi-signature support handles this cleanly. Customers who have been compromised need to be able to rotate secrets without missing events during the rotation window.

Delivery confirmation vs. processing confirmation. A 200 response from the customer’s endpoint means they accepted the HTTP request. It does not mean they successfully processed the event. Some teams want an acknowledgment API where consumers explicitly confirm processing. This is valuable for critical workflows but significantly complicates both sides of the integration, so it is worth careful consideration before committing.

Practical Guidance on Event Schema Design

The hardest part of webhook infrastructure is not the delivery mechanics but the event schema. I have seen platforms ship outbound webhooks with event schemas that made breaking changes in minor releases, with no versioning story, and watched as it destroyed customer confidence in the platform’s reliability.

Version your event schemas from the start. The simplest approach is an api_version field in every payload header or in the payload itself. When you need to make a breaking schema change, you can offer a migration window where both the old and new schema are available, then deprecate the old version.

API contract testing applies here. If your customers implement consumers that you want to avoid breaking, you need automated tests that verify your event production matches the documented schema. Generating events from a schema definition and validating them against that schema before delivery is straightforward and catches a large class of regression bugs early.

Document your event catalog clearly. Each event type, its trigger conditions, what fields are always present versus conditionally present, and what the payload schema is. The Standard Webhooks spec does not cover event catalog documentation, but the OpenAPI or AsyncAPI ecosystem provides good tooling for this.

The Bottom Line

Webhook infrastructure is one of those problems that gets worse the more you delay solving it properly. The initial naive implementation works for the first hundred customers and falls apart under the operational demands of a real production SaaS. When you are debugging missed events at midnight because a customer’s checkout flow depends on your payment.captured webhook, you will wish you had spent two weeks on the infrastructure rather than two hours.

My recommendation: start with Svix or Hookdeck unless you have specific reasons not to. The managed services are mature, the Standard Webhooks compatibility means your customers get a consistent experience, and the embeddable portal eliminates weeks of product work. If you hit the ceiling on cost, data residency, or customization requirements, migrate to Convoy at that point with the architecture patterns already established.

Building webhook infrastructure yourself is not wrong, but it is a larger investment than it looks, and the operational surface area grows continuously. Every team that has built it in-house has eventually built something that looks very much like what Svix or Hookdeck already ship.


For more on the event-driven patterns that underpin good webhook systems, see The Transactional Outbox Pattern and Idempotency in Distributed Systems. If you’re building the broader event-driven architecture your webhooks live inside, Event-Driven Architecture on AWS and AsyncAPI for Event-Driven APIs are good companions to this piece.