Your Webhook Is Already Broken
You’ve done it. You finished the feature, wrote the curl -X POST, pushed to production. Somewhere downstream a service is waiting for that event to trigger a job, sync a record, send a notification. Everything looks fine, until your downstream endpoint is in the middle of a deploy, times out at 30 seconds, and your event vanishes into the void.
No retry. No log. No evidence it ever happened. Just a support ticket three days later asking why their data didn’t sync.
That’s not a bug in your code. That’s the fundamental problem with naive webhook delivery: fire and forget is great until the receiver isn’t there to catch.
Convoy is the fix. It’s an open-source, self-hostable webhook gateway from Frain Technologies that adds retries, exponential backoff, dead-letter queues, HMAC signing, fan-out delivery, and event replay, the stuff every serious webhook system needs and almost nobody builds themselves the first time.
Full example: Clone the working Compose setup at github.com/KingPin/sumguy-examples/devops/convoy-self-hosted-webhooks/
What Convoy Actually Does
Think of Convoy as a middleware layer between your app that produces events and the endpoints that consume them. Instead of your service firing HTTP calls directly, it publishes events to Convoy. Convoy takes on the delivery responsibility: it retries on failure, signs payloads, routes to multiple subscribers, and keeps a full audit trail.
The core concepts:
- Projects: logical groupings (one per app or team)
- Sources: where events come in (your app, or inbound HTTP for ingress routing)
- Subscriptions: which endpoints get which events, with optional filter rules
- Endpoints: the downstream URLs that receive events
- Events: the payloads; each one has a full delivery attempt history
That’s the mental model. Now let’s get it running.
Deploy with Docker Compose
Convoy needs Postgres and Redis. The Compose file is straightforward:
services: postgres: image: postgres:16-alpine restart: unless-stopped environment: POSTGRES_USER: convoy POSTGRES_PASSWORD: convoy_secret POSTGRES_DB: convoy volumes: - pgdata:/var/lib/postgresql/data healthcheck: test: ["CMD-SHELL", "pg_isready -U convoy"] interval: 5s timeout: 5s retries: 5
redis: image: redis:7-alpine restart: unless-stopped volumes: - redisdata:/data
convoy: image: getconvoy/convoy:latest restart: unless-stopped depends_on: postgres: condition: service_healthy redis: condition: service_started ports: - "5005:5005" environment: CONVOY_DB_SCHEME: postgres CONVOY_DB_HOST: postgres CONVOY_DB_PORT: "5432" CONVOY_DB_DATABASE: convoy CONVOY_DB_USERNAME: convoy CONVOY_DB_PASSWORD: convoy_secret CONVOY_REDIS_SCHEME: redis CONVOY_REDIS_HOST: redis CONVOY_REDIS_PORT: "6379" CONVOY_NATIVE_REALM_ENABLED: "true" CONVOY_SIGNUP_ENABLED: "true" PORT: "5005"
volumes: pgdata: redisdata:docker compose up -d# give it 10-15 seconds for migrations to rundocker compose logs convoy --tail=20Hit http://localhost:5005 and you’ll land on the signup page. Create your admin account, create a project, and you’re in the dashboard.
Send Your First Event
Convoy has a REST API. After creating a project you’ll get an API key from the dashboard. Send an event like this:
# Replace PROJECT_ID and API_KEY with your valuescurl -X POST http://localhost:5005/api/v1/projects/PROJECT_ID/events \ -H "Authorization: Bearer API_KEY" \ -H "Content-Type: application/json" \ -d '{ "event_type": "user.created", "data": { "id": "usr_123", "email": "[email protected]" } }'That’s it from the producer side. Convoy queues the event, looks up which subscriptions match user.created, and fires delivery to each one.
For Python shops, you’ll probably wrap this in a helper:
import httpximport json
CONVOY_URL = "http://localhost:5005"PROJECT_ID = "your-project-id"API_KEY = "your-api-key"
def send_event(event_type: str, data: dict) -> dict: resp = httpx.post( f"{CONVOY_URL}/api/v1/projects/{PROJECT_ID}/events", headers={ "Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json", }, json={"event_type": event_type, "data": data}, timeout=10, ) resp.raise_for_status() return resp.json()
# Usagesend_event("order.completed", {"order_id": "ord_456", "total": 89.99})Go version, because some of you are contractually obligated to write everything in Go:
package main
import ( "bytes" "encoding/json" "fmt" "net/http")
const ( convoyURL = "http://localhost:5005" projectID = "your-project-id" apiKey = "your-api-key")
type Event struct { EventType string `json:"event_type"` Data map[string]any `json:"data"`}
func sendEvent(eventType string, data map[string]any) error { payload := Event{EventType: eventType, Data: data} body, _ := json.Marshal(payload)
req, _ := http.NewRequest( "POST", fmt.Sprintf("%s/api/v1/projects/%s/events", convoyURL, projectID), bytes.NewBuffer(body), ) req.Header.Set("Authorization", "Bearer "+apiKey) req.Header.Set("Content-Type", "application/json")
client := &http.Client{} resp, err := client.Do(req) if err != nil { return err } defer resp.Body.Close()
if resp.StatusCode >= 400 { return fmt.Errorf("convoy returned %d", resp.StatusCode) } return nil}Retries, Backoff, and Dead-Letter Queues
Here’s where Convoy earns its keep. When a delivery fails (non-2xx, timeout, connection refused), it doesn’t just give up. You configure retry strategy per endpoint, exponential backoff is the default, and it’s the right call:
- Attempt 1: immediate
- Attempt 2: 10s
- Attempt 3: 100s
- Attempt 4: ~17 minutes
- …up to your configured max attempts
After the final attempt, the event lands in the dead-letter queue (Convoy calls it “dead endpoints” in the UI). From there you can replay individual events or bulk-replay an entire window, say, everything that failed between 2 AM and 3 AM when your database was doing maintenance. This is the feature that makes 2 AM your friend instead of your enemy.
The retry config lives in the endpoint settings in the dashboard, or via the API:
curl -X POST http://localhost:5005/api/v1/projects/PROJECT_ID/endpoints \ -H "Authorization: Bearer API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "billing-service", "url": "https://billing.internal/webhooks", "support_email": "[email protected]", "rate_limit": 100, "rate_limit_duration": "1m", "advanced_signatures": true, "retry_config": { "type": "exponential", "retry_count": 5, "interval_seconds": 10 } }'HMAC Signing and Verification
Convoy signs every outbound payload with HMAC-SHA256 using a per-endpoint secret. Your receiver should always verify this, it’s the difference between “we got a webhook” and “we got a webhook we trust.”
Convoy sends the signature in X-Convoy-Signature. Here’s a receiver that verifies it:
import hashlibimport hmacfrom fastapi import FastAPI, Request, HTTPException
app = FastAPI()WEBHOOK_SECRET = "your-endpoint-secret-from-convoy"
@app.post("/webhooks")async def receive_webhook(request: Request): body = await request.body() sig_header = request.headers.get("X-Convoy-Signature", "")
# Convoy may send multiple signatures (key rotation), comma-separated expected = hmac.new( WEBHOOK_SECRET.encode(), body, hashlib.sha256 ).hexdigest()
sigs = [s.strip() for s in sig_header.split(",")] if not any(hmac.compare_digest(expected, s) for s in sigs): raise HTTPException(status_code=401, detail="Invalid signature")
payload = await request.json() event_type = payload.get("event_type") data = payload.get("data", {})
# handle your event print(f"Got {event_type}: {data}") return {"status": "ok"}Always use hmac.compare_digest for timing-safe comparison. Do not use ==. You’ve seen the talks; you know why.
Fan-Out and Subscription Rules
One event, multiple consumers, this is where Convoy really shines over a hand-rolled solution. Add multiple subscriptions to a source, point them at different endpoints, and optionally filter by event type or payload content.
Subscription filter example: billing service only cares about order.* events, notification service only wants user.*:
# Create subscription for billingcurl -X POST http://localhost:5005/api/v1/projects/PROJECT_ID/subscriptions \ -H "Authorization: Bearer API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "billing-sub", "type": "api", "endpoint_id": "BILLING_ENDPOINT_ID", "filter_config": { "event_types": ["order.created", "order.completed", "order.refunded"] } }'
# Create subscription for notificationscurl -X POST http://localhost:5005/api/v1/projects/PROJECT_ID/subscriptions \ -H "Authorization: Bearer API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "notify-sub", "type": "api", "endpoint_id": "NOTIFY_ENDPOINT_ID", "filter_config": { "event_types": ["user.created", "user.deleted"] } }'Now order.completed goes to billing only. user.created goes to notifications only. Neither service gets events it doesn’t care about. You just saved yourself from writing a routing layer.
Inbound Webhooks (Ingress Mode)
Convoy handles the other direction too. Got a payment provider, GitHub, Stripe, or any third party firing webhooks at you? Point them at a Convoy inbound source instead of directly at your service. Convoy receives the event, fans it out to your internal subscribers, and you get the same retry/audit benefits on the consumer side.
The inbound URL looks like https://convoy.yourdomain.com/ingest/SOURCE_ID. Configure that as your webhook endpoint with Stripe or GitHub, then subscribe your internal services to it. Now when Stripe retries because it thought your endpoint was slow, Convoy handles the deduplication. Your services receive each event exactly once.
Convoy vs. The Alternatives
| Convoy | Hookdeck | Svix | Roll-your-own | |
|---|---|---|---|---|
| Self-hostable | Yes | No | No | Yes |
| Retries + DLQ | Yes | Yes | Yes | You build it |
| Fan-out | Yes | Limited | Yes | You build it |
| HMAC signing | Yes | Yes | Yes | You build it |
| Audit log | Yes | Yes | Yes | You build it |
| Cost at scale | Infra only | $$$ | $$$ | Engineering time |
Hookdeck and Svix are cloud-only SaaS. They’re excellent and save you ops work, but you’re paying per event volume and you’re not in control of the data. For home labs, internal tooling, and anyone who’d rather own their stack, Convoy is the obvious move.
The roll-your-own path is genuinely fine for simple cases, but it almost always starts as “just retry three times” and ends six months later as a Redis queue with dead-letter handling, a signature library, a replay endpoint, and a dashboard someone built in a weekend. Convoy is that thing you’d have built, pre-built.
When You Don’t Need Convoy
Honestly? If you have exactly one consumer, your receiver is rock-solid (internal service, same datacenter, not going anywhere), and you genuinely don’t care if an event gets lost once a quarter, a direct HTTP call is fine. Don’t add infrastructure for the sake of adding infrastructure.
Similarly, if you’re already running something like RabbitMQ or Kafka, you might already have the retry and fan-out story covered at the queue layer. Convoy makes the most sense when you’re delivering webhooks to external or semi-reliable endpoints, or when you need the audit trail for compliance or debugging.
But if you’ve ever had a “why didn’t the webhook fire?” conversation with a teammate, you know the value of a full delivery log. Convoy ends those conversations.
Wrapping Up
Convoy is one of those tools that’s almost invisible when everything works, events flow, endpoints get hit, everyone’s happy. But when something breaks (and something always breaks), you’ve got retries in progress, a dead-letter queue to replay from, and a full audit trail to explain exactly what happened and when.
That beats staring at application logs trying to reconstruct whether your curl actually fired.
Your 2 AM self will appreciate it.