How to let your system grow without letting it collapse under its own weight — and why every place you open up for extension is also a place something can go wrong.
This chapter is about one of the most tempting and most dangerous things you can do in software: design your system so that other people can extend it. We will look at the three main ways to build extension points — webhooks, event buses, and pipeline patterns — and we will be honest about what each one costs you. We will cover the Open/Closed Principle as it applies not just to classes but to entire distributed services, the security attack surface that opens up every time you create an extension point, and the single most important question to ask before you build any of this: do you actually need it?
When a system works well, people want more from it. They want to connect it to other tools, add behavior that the original authors never planned, and customize it for their specific needs. This is a good problem to have. But it creates a real design pressure: how do you let people extend your system without giving them the keys to break it?
The naive answer is to just keep adding features. Someone asks for a new hook, you add it. Someone asks for a callback, you wire it up. This works for a while, but it has a ceiling. Every feature you add to the core is code you own, maintain, and carry forever. At some point the core becomes too large to understand, and every change risks breaking something three layers away.
The smarter answer is to design your system so that new behavior can be added outside the core, by other people, without touching your code. This is what extension points are for. But extension points are not free. They are contracts, and contracts have obligations.
The more you open your system for extension, the harder it becomes to reason about what your system does. A closed system is easy to understand and hard to customize. An open system is easy to customize and hard to understand. The goal is to find the right place on that spectrum for your use case.
Before we get into the mechanics, let's be clear about what you are signing up for when you build an extension point. These costs are easy to miss when you are designing the system in a quiet meeting room.
None of this means you should never build extension points. It means you should build them deliberately, and only when the value is clear.
Most engineers know the Open/Closed Principle from object-oriented design: a class should be open for extension but closed for modification. The idea is that you can add new behavior by subclassing or implementing an interface, without changing the existing code.
The same idea applies to entire services in a distributed system, but the stakes are much higher. When a class breaks its contract, the compiler catches it. When a service breaks its contract, you find out at 2am when production is on fire.
In a distributed system, "open for extension" means: you can change what the system does by plugging something in from the outside, without deploying new code to the core service. This is what webhooks, event buses, and plugin systems all try to give you.
"Closed for modification" means: the core contract — the API, the data model, the guarantees — does not change out from under the things that depend on it. This is the hard part. It requires you to think carefully about what is stable and what is flexible before you expose it.
A stable contract is one that changes as rarely as possible. It hides the implementation details of the core system and only exposes what external code genuinely needs.
Think of GitHub's webhook events. They fire an event when a pull request is opened, closed, merged. The event payload contains the PR number, the author, the target branch. This contract has been stable for years. GitHub can completely rewrite how pull requests are stored internally, and nobody's webhook integration breaks, because the contract is between the event and the consumer, not between the consumer and the internals.
Expose domain events, not implementation events. "A pull request was merged" is a domain event — it describes something meaningful in your domain. "Row 4872 was updated in the pull_requests table" is an implementation event — it describes a database operation. Domain events make stable contracts. Implementation events leak internals and break constantly.
A webhook is an HTTP callback. You register a URL with a service, and when something happens in that service, it sends an HTTP request to your URL. That's the whole idea.
Stripe sends a webhook when a payment succeeds. GitHub sends a webhook when someone pushes to a repository. Shopify sends a webhook when an order is placed. This pattern is everywhere because it is simple to understand, simple to implement, and requires no persistent connection.
Webhooks look simple but they have a real operational problem: you are sending an HTTP request to a server you do not control. That server might be down. It might be slow. It might return an error. What do you do?
The standard answer is at-least-once delivery with retries. You keep sending the webhook until you get a successful response (HTTP 2xx), using exponential backoff to avoid hammering a struggling consumer. You put a limit on retries — usually 24-72 hours — and then mark the delivery as failed and move on.
This sounds fine until you think through the consequences. At-least-once means the consumer might receive the same event more than once, especially around retries. If your consumer charges a customer's credit card when it receives a "payment succeeded" event, and it receives the event twice, you have a big problem. The consumer must be idempotent.
Always include a unique event ID in your webhook payload. Consumers use this ID to deduplicate events — they check if they have already processed this ID before acting. Without this, you are shipping consumers an at-least-once delivery mechanism with no way to make it safe.
Because webhooks are retried independently, they do not arrive in order. A "subscription cancelled" event might arrive before the "subscription created" event it follows, if the first delivery of "subscription created" failed and was retried later. Consumers who assume ordering will build bugs that are very hard to reproduce.
The safest design is to include a sequence number or timestamp in the event payload, and to put the timestamp of the state at the time the event fired, not the time it was delivered. If ordering matters, the consumer can discard events that arrive out of sequence.
Anyone who knows a webhook URL can POST to it. If your consumer trusts any POST to that URL, an attacker can forge events — trigger fake "payment succeeded" events, fake "user verified" events, whatever your consumer acts on.
The standard defense is an HMAC signature. When you fire a webhook, you compute
a hash of the request body using a shared secret, and include it in a header (Stripe uses
Stripe-Signature, GitHub uses X-Hub-Signature-256). The consumer
recomputes the hash and checks that they match. If someone tampers with the body or forges a
request without the secret, the hash will not match.
import hmac
import hashlib
def verify_webhook(payload_body: bytes, secret: str, signature_header: str) -> bool:
# Signature header format: "sha256=abc123..."
expected = hmac.new(
secret.encode(),
payload_body,
hashlib.sha256
).hexdigest()
received = signature_header.removeprefix("sha256=")
# Use compare_digest to avoid timing attacks
return hmac.compare_digest(expected, received)When one event needs to be sent to many subscribers, you have a fan-out problem. If you have 10,000 webhook subscribers for a "new post published" event, you cannot send 10,000 HTTP requests synchronously in the path of the user who published the post. It will be too slow, and if any subset of those requests is slow or fails, it blocks the rest.
The solution is to put webhook delivery into a background job queue. The publish action writes a job for each subscriber, and workers process those jobs asynchronously. This decouples the user-facing action from the delivery, and lets you scale delivery workers independently.
A webhook is a point-to-point mechanism. Your service calls a specific URL registered by a specific consumer. If you want five different systems to react to the same event, you call five different URLs. This works, but it means your core service has to know about all its consumers, which is exactly the coupling you wanted to avoid.
An event bus solves this. Publishers write events to a central topic. Subscribers read from that topic independently. The publisher does not know who is subscribed. It does not care. It just fires and forgets.
This is a big improvement. Adding a new consumer — say, a fraud detection service that wants to analyze every order — requires no changes to the Order Service. You just add a new subscriber. The publisher and all existing subscribers are unaffected.
Decoupling is genuinely valuable, but it comes with a real cost that is easy to miss when you are designing the system and feels very real when you are debugging it at midnight.
When everything was tightly coupled, you could trace a request from beginning to end in a single log. "User clicked checkout. Order Service created the order. Inventory Service decremented the count. Notification Service sent the email." Clean, linear, visible.
With an event bus, the causal chain is broken. You can see that an order was placed. You can see that the inventory was decremented. But connecting those two facts requires distributed tracing infrastructure — correlation IDs threaded through every event payload and every downstream service. Without that infrastructure, debugging becomes archaeology.
Before you adopt an event bus, make sure you have distributed tracing in place. A correlation ID should travel with every event from the moment it is created until every downstream effect has been produced. Without this, "why did this order not get an email?" becomes a multi-hour detective investigation across three different log systems.
One of the hardest problems with event buses is that events are shared across teams, and
teams move at different speeds. The Order Service team wants to add a new field to the
order.placed event. The Inventory Service has not deployed yet. What happens?
If you add a new field, old consumers that do not understand it should just ignore it. This is forward compatibility, and it means consumers must tolerate unknown fields. If you remove a field that old consumers depend on, those consumers break. This is why you can add fields to an event schema, but you cannot remove them or change their meaning without a migration plan.
The practical approach is a schema registry — a central store of event schemas with explicit version numbers. Producers register their schema before publishing. Consumers declare which version they support. The registry enforces that producers only make compatible changes.
| Change Type | Safe? | Notes |
|---|---|---|
| Add an optional field | Yes | Old consumers ignore it. New consumers can use it. |
| Add a required field | No | Old consumers that validate schema will reject the event. |
| Remove a field | No | Any consumer reading that field breaks silently. |
| Rename a field | No | Same as remove + add. Treat as a breaking change. |
| Change a field's type | No | Will silently corrupt data or cause parse failures. |
| Change a field's meaning | No | The most dangerous change — passes all schema checks, wrong behavior. |
Modern event buses like Kafka and Google Pub/Sub support the concept of consumer groups. Multiple instances of the same service share a consumer group, and each event is delivered to exactly one instance in the group. This gives you horizontal scalability: add more instances to process events faster.
But different services that each want all events — the Inventory Service, the Notification Service, the Analytics Service — are in separate consumer groups. Each group gets a full copy of every event. The bus fans out between groups, not within them.
Webhooks and event buses are about connecting separate services. The pipeline pattern is about composing behavior within a single processing flow. You break a complex operation into a sequence of discrete stages. Each stage does one thing, takes some input, and produces some output (or modified input) for the next stage.
You have seen this pattern if you have used HTTP middleware in any web framework. In Express, each middleware function receives a request object and either modifies it and passes it forward, or halts the chain and sends a response. In Django, middleware wraps the request processing in layers.
The pipeline pattern is powerful because stages are reusable and independently testable. You can add a new stage — say, request logging — without touching any existing stage. You can reorder stages. You can swap out the rate limiter implementation. The core logic does not care what happened before it.
Every pipeline stage needs to answer one question when something goes wrong: do I halt the chain, or do I pass the request forward? This decision is the most consequential contract a stage has, and it is often underdocumented.
A stage that should halt but does not creates security holes. An auth check stage that swallows exceptions and passes the request forward — even if the check failed — means unauthenticated requests reach your core logic. This happens more often than you would think, especially when developers add try/catch blocks around "unreliable" external calls without thinking through the security implications.
Never silently pass a request forward when a security-critical stage fails. If the auth check throws an exception — maybe the auth service is down — the right behavior is to halt with a 503, not to let the request through unauthenticated. An availability penalty is acceptable. A security hole is not.
The pipeline pattern is best for operations that are synchronous, sequential, and share context. HTTP request processing is a natural fit. Message processing before writing to a database is another.
It is a bad fit for operations that are parallel, asynchronous, or independent. If you need to send an email, update a search index, and refresh a cache after creating an order, those are not steps in a pipeline — they are independent side effects. Use an event bus for those.
Webhooks and events are about reacting to things that happen. Plugins are different: they run code inside your system's execution context. They can change how your system behaves, not just observe it.
Think of a content management system that lets you install plugins to add new field types, custom workflows, or integrations. Or a data pipeline that lets you add custom transformation steps. The plugin registers itself against a contract — an interface, a set of hooks — and the host system calls it at the right moments.
A plugin contract is the interface between your core system and the plugin code. It defines what the plugin is expected to do, what data it receives, and what it is allowed to return. This is the most important thing you will design when building a plugin system.
A narrow contract is better than a wide one. If your plugin contract only exposes the fields a plugin genuinely needs, then plugins cannot accidentally (or maliciously) read data they should not have access to. It also means you can change the internals of your system without breaking the contract.
# Bad: exposes the entire request context — too much surface area
class TransformPlugin:
def transform(self, request_context: RequestContext) -> RequestContext:
...
# Better: expose only what the plugin needs for this operation
class TransformPlugin:
def transform(self, record: dict, config: dict) -> dict:
...If a plugin crashes, what happens to your system? If a plugin takes 30 seconds to run, what happens to your latency? If a plugin writes to a global variable, can it corrupt other plugins? These are not edge cases — they happen with real third-party code.
The options for isolation range from "trust the plugin" (no isolation) to "run the plugin in a separate process with strict resource limits" (strong isolation). Where you land depends on who writes the plugins.
WebAssembly is emerging as a serious option for plugin sandboxing. Wasm modules run in a memory-isolated sandbox, cannot access the host filesystem or network unless explicitly granted, and have near-native performance. Systems like Cloudflare Workers, Envoy proxy filters, and several data processing tools use this approach.
Every extension mechanism is a new way for something to go wrong. Security is not the last thing to think about with extension points — it needs to be baked into the design from the start.
Server-Side Request Forgery (SSRF) is one of the most common webhook vulnerabilities.
It works like this: you let a user register a webhook URL. They register
http://169.254.169.254/latest/meta-data/ — the AWS instance metadata endpoint.
Your server faithfully sends a POST to that URL, which is an internal IP address only your server
can reach. The response comes back to your server, and if you log it or return it in an error
message, the attacker has just read your AWS credentials.
The defense is to resolve and validate the webhook URL before calling it. Block requests to private IP ranges (10.x.x.x, 172.16.x.x, 192.168.x.x), loopback addresses (127.x.x.x), and cloud metadata endpoints (169.254.169.254). Do this at request time, not just at registration time, because DNS can resolve to different IPs than it did when the URL was registered — a technique called DNS rebinding.
Never trust a user-supplied URL and call it from your server without first checking that it resolves to a public IP address. Do this check immediately before making the request, not at registration time. Use a dedicated HTTP client with SSRF protections baked in, not a raw HTTP library.
An event bus that allows broad subscriptions can become a data exfiltration path.
If any authenticated user can subscribe to any event topic, and events contain sensitive data,
then a compromised account can subscribe to payment.processed and silently
receive every payment event in your system.
The fix is fine-grained subscription permissions. A service should only be able to subscribe to event topics that are relevant to its function. The principle of least privilege applies to event subscriptions as much as it does to API access.
A plugin that runs in the context of your service inherits its permissions. If your service has read access to a database, a plugin can read that database. If your service can call an internal API that is not exposed externally, a plugin can call it too.
This is why isolation matters beyond just crash safety. Even a non-malicious plugin might access data it should not, simply because it was the easiest way to solve a problem. The principle of least privilege should apply to what plugins can see and call, not just what the host service can do.
Here is the thing nobody tells you when you build your first extension system: the moment someone external depends on your extension API, you have two APIs. The one you show to your users, and the one you show to your extenders. Both need to be versioned. Both need documentation. Both will break in different ways.
The extension API is often harder to maintain than the user-facing API, because it is less visible. When a user-facing API breaks, users complain immediately. When an extension API breaks, a webhook payload changes shape, a plugin interface signature shifts — it might take days or weeks before a third-party integration surfaces the breakage.
Every webhook payload should include a version field. Every plugin interface should have a version. Not because you plan to break things, but because you need the ability to make changes without breaking existing integrations.
The strategy that works: support old versions for a defined deprecation window. Announce the new version. Give integrators 6-12 months to migrate. Then sunset the old version. Do this in writing, with a clear timeline, every time.
{
"api_version": "2024-11-01",
"id": "evt_3P9dK2EKj1234",
"type": "payment.succeeded",
"created": 1714392000,
"data": {
"object": {
"id": "pay_9dK23",
"amount": 9900,
"currency": "usd",
"status": "succeeded"
}
}
}Notice how Stripe dates their API versions. A consumer specifies which version they built against when they register their webhook. Stripe sends them events in that format, even after the format has changed for newer versions. This is how you deprecate without breaking.
The wrong time to build an extension system is when you have zero concrete use cases and someone in a planning meeting says "we should make this extensible."
Extensibility is not inherently good. It is a feature that costs engineering time to build, adds operational complexity, creates security surface area, and locks you into a contract you must maintain. It is worth paying that cost when you have a real need. It is waste when you are designing for hypothetical futures.
Build a plugin system when you have at least two concrete, different use cases that cannot be served by extending the core code. One use case is a feature. Two different use cases reveal the pattern that the extension mechanism should support. "We might want this later" is not a use case.
Ask these questions before building:
Let's bring the three mechanisms together and be explicit about when to use each one.
| Mechanism | Best For | Main Cost | Security Concern |
|---|---|---|---|
| Webhooks | Notifying external systems of events. Simple integrations. Third-party consumers you do not control. | Delivery reliability. You must retry, deduplicate, handle dead endpoints. | SSRF if you call user-supplied URLs. Forgery if you do not sign payloads. |
| Event Bus | Internal fan-out to multiple services. Decoupling producers and consumers. High-throughput event streams. | Observability complexity. Schema evolution. Operational overhead of the bus itself. | Data exfiltration via over-broad subscriptions. Schema injection. |
| Pipeline | Composable request processing. Sequential stages that share context. Middleware-style extensibility. | Order dependency. A halting stage blocks everything after it. | A stage that swallows exceptions and passes requests through creates auth bypasses. |
| Plugin System | Third-party code that changes system behavior. Marketplace-style extensibility. | Isolation complexity. Two APIs to maintain. Plugin support burden. | Privilege escalation. Malicious code in your process if not sandboxed. |
The honest summary: prefer the simplest mechanism that solves your problem. A webhook is simpler than an event bus. An event bus is simpler than a plugin system. Go up the ladder only when the simpler mechanism genuinely does not fit.