A webhook can make two business systems feel connected: a payment service reports a successful transaction, a CRM announces a new lead, or a project tool tells an internal application that a task changed. The receiving system gets an HTTP request and can react without repeatedly asking the other system whether anything is new.
That convenience can hide a difficult operating question: what happens when the delivery is forged, arrives twice, takes longer to process than the sender will wait, or fails halfway through the workflow? A dependable webhook integration treats receiving an event as one step and processing it as another. The endpoint verifies and records the delivery quickly; a worker handles the business action with a clear path for retry and review.
Start with the delivery contract
Before writing a handler, document the provider’s contract. Identify which event types the business actually needs, how the provider authenticates deliveries, how long your endpoint has to respond, what retry behavior the provider uses, and whether events can arrive out of order. Also record the provider’s unique delivery or event identifier and the fields that indicate the event type and action.
Subscribe only to useful events. Receiving every event increases work, storage, and the number of cases the application must understand. A small, explicit list is easier to test and easier for an operations team to explain.
Do not put API keys or webhook secrets in a URL. Use the provider’s supported signing mechanism, store its secret outside source code, and keep the endpoint on HTTPS with certificate verification enabled. These controls do not make the business action automatically safe, but they establish a sensible boundary for deciding whether a request is authentic and untampered with.
Verify the request before trusting the payload
Signature verification generally depends on the exact bytes the sender delivered. Framework middleware that parses and re-serializes JSON can change whitespace or ordering and cause a valid signature check to fail. Capture the raw request body as required by the provider, then verify the signature before treating the payload as an instruction.
Use a constant-time comparison where the provider recommends it, reject missing or invalid signatures, and keep secrets in a protected configuration mechanism. Log a safe reason such as signature_invalid, not the secret, authorization header, or complete request body. If a provider includes a timestamp in its signed header, check that it falls within a reasonable freshness window and account for limited clock skew.
After authentication, validate the event structure. Check the event type, action, required identifiers, and expected data shape. An authenticated request can still be malformed, unsupported, or inconsistent with the version your application understands.
Acknowledge quickly, process asynchronously
A webhook sender usually wants a prompt 2XX response. The endpoint should do the minimum work needed to verify the request and durably record the event or place it on a queue. It should not wait for a slow CRM update, a large file operation, or a chain of downstream API calls before acknowledging the delivery.
A useful receipt record can include the provider name, delivery ID, event type, received timestamp, validation outcome, processing status, attempt count, and a redacted reference to the business object. Store the payload only where the retention and access rules make sense. Do not copy passwords, tokens, payment details, or unnecessary personal data into a convenient event table.
A queue gives the worker room to retry temporary failures without forcing the sender to keep the HTTP connection open. It also makes workload visible: an operations lead can see whether events are waiting, processing, completed, or held for review.
Expect duplicate deliveries
Webhook delivery is commonly at-least-once: the provider may send an event again when it did not receive a response, when a network path failed, or when a redelivery was requested. Your handler should assume duplicates are normal rather than treating them as exceptional.
Use the provider’s event or delivery ID as an idempotency key when that ID is guaranteed to be unique. Record it before processing, with a uniqueness constraint that prevents two workers from accepting the same delivery at the same time. If the provider can create separate event records for the same underlying object and action, combine the event type with the provider’s object ID where the documentation supports that approach.
Idempotency is not just a database lookup. The business action must also be safe. If processing sends an email, creates an invoice, or changes a customer record, decide how a repeated worker attempt is prevented or detected. For operations that cannot be made safely repeatable, route the event to a review state before taking the irreversible action.
Make failure and replay deliberate
Separate temporary failures from permanent ones. A short network timeout or unavailable downstream service may be retryable. Invalid data, an unsupported event type, or a failed authorization usually needs correction or review instead. Use bounded retries with backoff, record the next attempt time, and stop retrying when the budget is exhausted.
Keep failed events available for investigation without making the replay button a dangerous shortcut. A replay operation should show the event ID, prior attempts, last error category, and intended destination. Require an authorized operator or a controlled job to select the event. Before replaying, check whether the business action partially completed and whether the current application version still understands the event.
When a provider offers redelivery, understand whether it keeps the original event identifier. Your local duplicate logic should recognize that a redelivery may be the same event, while still allowing an operator to re-run a failed business action through an explicit, auditable path.
Test the cases that matter
A basic test suite should cover a valid delivery, a missing signature, a modified body, an unknown event type, a duplicate delivery, a worker restart, a downstream timeout, a permanently invalid record, and a controlled replay. Test the endpoint’s response time separately from the worker’s processing time.
Monitor more than HTTP errors. Track accepted deliveries, rejected signatures, queue age, processing duration, retry count, duplicate count, permanently failed events, and events waiting for human review. These measures help distinguish a provider outage from a bug in one event handler or a growing backlog.
A practical first implementation
- Choose one business-critical event and document the provider’s delivery rules.
- Build an HTTPS endpoint that preserves the raw body and verifies signatures before parsing trusted data.
- Record a minimal receipt with a unique event ID and a safe processing status.
- Enqueue accepted work and return a prompt 2XX response.
- Make the worker idempotent, validate event types, and use bounded retries.
- Add a reviewable failed state and an auditable replay procedure.
- Test duplicate, outage, malformed, and partial-completion scenarios before connecting a second event.
Webhooks are small HTTP endpoints with large operational consequences. The reliable design is usually modest: verify the sender, record the receipt, queue the work, prevent duplicate effects, and give people a clear way to understand and replay failures. That structure lets a small business gain timely system-to-system updates without turning every delivery problem into an emergency.
Next step: Schedule a short consultation to identify the next useful improvement.