Progressive Extraction: Start Simple, Complicate by Requirement

The most expensive architecture mistake in software engineering is not building something too simple. It is building something too complex before you have evidence that complexity is needed.

The Problem with Architecture Astronautics

The standard consulting playbook for new systems goes something like this: draw a microservices diagram on a whiteboard, label each box with a domain boundary from the DDD book, add message queues between everything, sprinkle in an API gateway, and declare the architecture "enterprise-ready."

Six months later, the team has built the inter-service communication framework, the distributed tracing pipeline, the service mesh configuration, and the deployment orchestration. The actual business logic fits in about 200 lines of code that nobody has had time to write because they were too busy building infrastructure.

This is architecture astronautics: designing for problems you do not have, at a scale you have not reached, for requirements that have not been articulated.

The Progressive Extraction Pattern

Progressive extraction inverts the approach. Start with the simplest architecture that could possibly work. Run it in production. Observe where it actually strains. Extract components only when you have evidence that extraction is needed.

The pattern follows a simple decision tree:

Build a monolith (or modulith). Single deployable, single database, single process.
Observe production behavior. Where are the bottlenecks? Which components scale differently? Where are the deployment conflicts?
Extract when evidence demands it. The payment processing module needs to scale independently? Extract it. The search functionality has different latency requirements? Extract it. The reporting module causes deployment delays? Extract it.
Do not extract speculatively. "This might need to be separate someday" is not evidence. "This component's scaling profile is 10x different from the rest of the system" is evidence.

Why This Works

You Cannot Predict Boundaries

Domain-Driven Design provides excellent tools for understanding business domains. It does not tell you where to draw service boundaries in a distributed system. Service boundaries should be drawn where independent scaling, independent deployment, or independent technology choices are required by observed production behavior -- not where a domain expert drew bubbles on a diagram.

In our experience, at least 30% of the service boundaries we drew on whiteboards turned out to be wrong when we observed real production traffic. Modules we expected to be hot were cold. Modules we expected to be independent were tightly coupled in ways the domain model did not predict. Data that we thought lived in one domain was accessed by three.

Distributed Systems Are Expensive

Every service boundary you introduce adds:

Network hops -- Latency, failure modes, retry logic
Data consistency challenges -- Distributed transactions, eventual consistency, saga patterns
Operational overhead -- Independent deployment, monitoring, alerting, logging per service
Debugging complexity -- Distributed tracing, request correlation, cross-service error propagation

These are not theoretical costs. They are engineering hours spent on infrastructure instead of features, and they compound with every service you add.

Extraction Is Easier Than Merging

If you start simple and need to extract, the extraction is mechanical: identify the boundary, create the new service, migrate the API calls, verify behavior. It is work, but it is well-understood work with clear success criteria.

If you start distributed and need to merge (because you drew the boundaries wrong), the merge is a nightmare: data migration, API consolidation, behavior reconciliation, and untangling distributed transaction patterns.

Starting simple preserves optionality. Starting complex does not.

The Modulithic Architecture

We use the term "modulithic" to describe the middle ground: a single deployable with clear internal module boundaries. Each module has its own package structure, its own API surface, and its own data access patterns -- but they all deploy together and communicate through in-process function calls, not network requests.

The modulithic architecture gives you:

Logical separation without distributed systems complexity
Module-level testing without inter-service contract testing
Single deployment without coordinated multi-service releases
In-process communication without network latency and failure modes
Clear extraction seams when (if) you need to distribute

This is not a compromise. It is a deliberate architectural choice that optimizes for the most common reality: the system does not need to be distributed yet.

When to Extract

We use three criteria to decide when extraction is justified:

Criterion 1: Scaling Divergence

Module A needs 10 instances to handle load. Module B needs 1 instance. Deploying 10 instances of Module B wastes resources. Extract Module A.

Criterion 2: Deployment Independence

Module A changes daily. Module B changes monthly. Every Module A deployment requires testing Module B (because they are co-deployed), creating unnecessary deployment friction. Extract Module A.

Criterion 3: Technology Mismatch

Module A is CPU-bound computation best served by Go or Rust. Module B is I/O-bound orchestration best served by Python or Java with virtual threads. The language/runtime mismatch justifies separation.

Note what is absent from these criteria: organizational structure, domain boundaries, and "best practices" from conference talks. Those are inputs to the analysis, not decision criteria.

A Real Example

We built our AI platform as a modulithic Python application. Meeting processing, financial analytics, email management, and security monitoring all lived in one process.

Over time, we observed:

Meeting transcript processing was CPU-intensive and spiked during specific hours
Financial analytics needed access to a different set of secrets than email processing
Security scanning needed to run continuously, independent of user-facing services

These observations led to targeted extractions. The meeting pipeline became its own service. Security scanning became its own service. Financial analytics stayed in the main application because its scaling profile matched the rest of the system.

We extracted three services over six months. If we had started with the microservices architecture we initially designed on the whiteboard, we would have built eight services. Five of them would have been unnecessary complexity that slowed development and increased operational burden.

Progressive extraction saved us months of unnecessary engineering and produced a better architecture than we could have designed upfront.

David Boggess is Managing Partner and CEO of Boggess Advisory Group. BAG advocates for the modulithic architecture pattern and progressive extraction as defaults for new systems.