Evolutionary architecture pitfalls and antipatterns

We identify two kinds of bad engineering practices that manifest in software projects — pitfalls and antipatterns. Many developers use the word antipattern as a jargon for "bad", but the real meaning is more subtle. A software antipattern has two parts. First, an antipattern is a practice that initally looks like a good ideia, but turns out to be a mistake. Second, better alternatives exist for most antipatterns. Architects notice many antipatterns only in hindsight, so they are hard to avoid. A pitfall looks superficially like a good idea but immediately reveals itself to be a bad path.

Technical architecture

Antipattern: vendor king

Some large enterprises buy Enterprise Resource Planning (ERP) software to handle common business tasks like accouting, inventory management, and other common chores. However, many organizations become overambitious with this category of software, leading to the vendor king antipattern, an architecture built entirely around a vendor product that pathologically couples the organization to a tool.

To escape this antipattern, treat all software as just another integration point, even if it initially has broad responsibilities. By assuming integration at the outset, developers can more easily replace behavior that isn't useful with other integration points, dethroning the king.

Pitfall: leaky abstractions

Increased tech stack complexity has made the abstraction distraction problem worse recently. Not only does the ecosystem change, but the constituent parts become more complex and interwined over time as well. Our mechanism for protecting evolutionary change — fitness functions — can protect the fragile join points or architecture. Architects define invariants at key integration points as fitness functions, which run as part of a deployment pipeline, ensuring abstractions don't start to leak in undesirable ways.

Antipattern: last 10% trap

Another kind of reusability trap exists at the other end of the abstraction spectrum, with package software, platforms, and frameworks.

In 4GLs, 80% of what the client wants is quick and easy to build. These environments are modeled as rapid application development tools, with drag-and-drop support for UIs and other niceties. However, the next 10% of what the client wants is, while possible, extemely difficult — because that functionality wasn't built into the tool, framework, or language. So clever developers figure out a way to hack tools to make things work: adding a script to execute where static things are expected, chaining methods, and other hacks. The hack only gets you from 80% to 90%. Ultimately the tool can't solve the problem completely — a phrase we coined as the Last 10% Trap — leaving every project a disappointment. While 4GLs make it easy to build simple things fast, they don't scale to meet the demands of the real world.

Antipattern: code reuse abuse

Ironically, the more effort developers put into making code reusable the harder it is to use. Making code reusable involves adding additional options and decisions points to accomodate the different uses. The more developers add hooks to enable reusability the more they harm the basic usability of the code.

Microservices eschew code reuse, adopting the philosophy of prefer duplication to coupling: reuse implies coupling, and microservices architectures are extremely decoupled. However, the goal in microservices isn't to embrace duplication but rather to isolate entities within domains. Services that share a common class are no longer independent. In a microservices architecture, Checkout and Shipping would each have their own internal representation of Customer. If they need to collaborate on customer-related information, they send the pertinent information to each other. Architects don't try to reconcile and consolidate the disparate versions of Customer in their architecture. The benefits of reuse are illusory and the coupling it introduces comes with its disadvantages. Thus, while architects understand the downsides of duplication, the offset that localized damage to the architectural damage too much coupling introduces.

We're not suggesting teams to avoid building reusable assets, but rather evaluate them continually to ensure they still deliver value. When coupling points impede evolution or other important architectural characteristics, break the coupling by forking or duplication.

All too often architects make a decision that is the correct decision at the time but becomes a bad decision over time because of changing conditions like dynamic equilibrium. Architects must continually evaluate the fitness of the "-ilities" of the architecture to ensure they still add value and haven't become antipatterns.

Pitfall: resume-driven development

Don't build architecture for the sake of architecture — you are trying to solve a problem. Always understand the problem domain before choosing an architecture rather than the other way around.

Incremental change

Antipattern: inappropriate governance

In modern environments, it is inappropriate governance to homogenize on a single technology stack. This leads to the inadvertent overcomplication problem, where governance decisions add useless multipliers to the effort required to implement a solution. When developers build monolith architectures, governance choices affect everyone. Thus, the architect must look at the requirements of every project and make a choice that will serve the most complex case. A small project may have simple needs yet must take on the full complexity for consistency.

With microservices, because none of the services are coupled via technical or data architecture, different teams can choose the right level of complexity and sophistication required to implement their service. This partitioning tends to work best when the team wholly owns their service, including the operational aspects.

Building services in different technology stacks is one way to achieve technical architecture decoupling. Many companies try to avoid this approach because they fear it hurts the ability to move employees across projects. However, Chad Fowler, an architect at Wunderlist, took the opposite approach: he insisted that teams use different technology stacks to avoid inadvertent coupling. His philosophy is that accidental coupling is a bigger problem than developer portability.

From a practical governance standpoint in large organizations, we find the Goldilocks Governance model works well: pick three technology stacks for standardization — simple, intermediate, and complex — and allow individual service requirements to drive stack requirements. This gives teams the flexibility to choose a suitable technology stack while still providing the company some benefits of standards.

Pitfall: lack of speed to release

While the extreme version of Continuous Delivery, continuous deployment, isn't required for an evolutionary architecture, a strong correlation exists between the ability to release software and the ability to evolve that software design.

If companies build an engineering culture around continuous deployment, expecting that all changes will make their way to production only if they pass the gauntlet laid out by the deployment pipeline, developers become accustomed to constant change. On the other hand, if releases are a formal process that require a lot of specialized work, the chances of being able to leverage evolutionary architecture diminishes.

Cycle time is therefore a critical metric in evolutionary architecture projects — faster cycle time implies a faster ability to evolve.

Business concerns

Most of the time, business people aren't nefarious characters trying to make things difficult for developers, but rather have priorities that drive inappropriate decisions from an architectural standpoint, which inadvertently constrain future options.

Pitfall: product customization

Salespeople want options to sell. The caricature of sales people has them selling any requested feature before determining if their product actually contains that feature. Thus, sales people want infinitely customizable software to sell. However, that capability comes at a cost along with a spectrum of implementation techniques, like unique build for each customer, permanent feature toggles, and product-driven customization.

Customization also impedes evolvability, but this shouldn't discourage companies from building customizable software, but rather to realistically asses the associated costs.

Antipattern: reporting

Reporting is a good example of inadvertent coupling in monolithic architectures. Architects and DBAs want to use the same database schema for both system of record and reporting, but encounter problems because a design to support both is optimized for neither. A common pitfall developers and report designers conspire to create in layered architecture illustrates the tension between concerns. Architects build layered architecture to cut down on incidental coupling, creating layers of isolation and separation of concerns. However, reporting doesn't need separate layers to support its function, just data. Additionally, routing requests through layers adds latency. Thus, many organizations with good layered architectures allow report designers to couple reports directly to database schemas, destroying the ability to make changes to the schema without wrecking reports. This is a good example of conflicting business goals subverting the work of architects and making evolutionary change extremely difficult. While no one set out to make the system hard to evolve, it was the cumulative effect of decisions.

Many microservices architectures solve the reporting problem by separating behavior, where the isolation of services benefits separation but not consolidation. Architects commonly build these architectures using event streaming or message queues to populate domain "system of record" databases, each embedded within the architectural quantum of the service, using eventual consistency rather than transactional behavior. A set of reporting services also listens to the event stream, populating a denormalized reporting database optimized for reporting. Using eventual consistency frees architects from coordination — a form of coupling from an architectural standpoint — allowing different abstractions for different uses of the application.

Pitfall: planning horizons

The Sunk Cost Fallacy describes decisions affected by emotional investment. Put simply, the more someone invests time or effort into something, the harder it becomes to abandon it. In software, this is seen in the form of the irrational artifact attachment — the more time and effort you invest in planning or a document, the more likely you will protect what's contained in the plan or document even in the face of evidence that it is inaccurate or outdated.

Beware of long planning cycles that force architects into irreversible decisions and find ways to keep options open. Architects should avoid following technologies that require a significant upfront investment before software is actually built (e.g., large licenses and support contracts) before they have validated through end-user feedback that the technology actually fits the problem they are trying to solve.

Neal Ford, Rebecca Parsons & Patrick Kua, "Evolutionary Architecture Pitfalls and Antipatterns", in Building Evolutionary Architectures: Support Constant Change, 12-139.