Powering Autonomous Capability Domains using an Enterprise Event Stream

Disclaimer – Copyright Rocket Mortgage, LLC [2022 – 2023]. The author (Nish Nishant) has been authorized to post this article under license from Rocket Mortgage. This article is is the property of Rocket Mortgage, LLC and is not authorized for redistribution.

Author’s Note – I wrote this article for my employer’s public facing technology blog, but for various reasons, that blog was retired. Since I did not want to lose the content forever, I am re-publishing it here with explicit approval, and hence the disclaimer above.

Abstract

As software systems grow in number and complexity, they end up being inadvertently dependent on each other. This slows down the delivery velocity across technology companies. Teams are forced to delay deliverables as they wait for other cohorts to finish functionality that they’re reliant on. This delay also results in teams not evolving their legacy components.

Solutions to these challenges include defining capability domains, creating boundary APIs and using an enterprise-wide event stream to achieve clean separation across capability domains. This article focuses on the core patterns required to achieve this solution and does not recommend any particular tool-stacks other than as coincidental examples.

Capability Domains

Defining and mapping a technology enterprise’s software systems into domains based on capability is a common first step in avoiding dependencies between systems. Typically, these domains match the technology enterprise’s organizational structure, though it isn’t required. Each capability domain owns the functionality to implement its specific capability, and ideally, there wouldn’t be any overlap across domains.

Increasing the autonomy of the domain is one of the goals of capability domains, which means that systems owned by the domain are considered internal to each domain. As a result, the teams building those systems have the freedom to dictate their technology roadmaps.

A common anti-pattern here is when internal systems integrate directly with other systems from another domain. Doing this is a form of negative coupling that negates the advantages of defining and mapping enterprise-wide capability domains. The preferred way to expose a domain’s functionality to another with minimal coupling is through domain boundaries.

Domain Boundaries

Domain boundaries are public interfaces on the edge of the domain. Typically, these public interfaces are implemented as public REST APIs with published service contracts. Domains within an enterprise need to set mutual expectations that they will only integrate with other domains via that domain’s boundary APIs. Doing so solves the problem of negative coupling that arises as a side effect of directly integrating with internal systems outside their domain.

The boundary APIs also need to be backwards compatible, and it’s good practice to maintain a formal API versioning system. Boundary APIs themselves lack the autonomy internal domain systems enjoy, but this is a very small price to avoid roadmap-hindering dependencies.

Enterprise Event Streams

An event stream is a continuous stream of events broken down into named event topics. An enterprise event stream is a shared event stream where all capability domains publish and receive messages.

Event streams are typically built on data streaming platforms such as Confluent Kafka, Amazon Kinesis or Azure Event Hubs. Regardless of the vendor you choose, they all have a standard, core set of features to stream event messages and are all built for high scalability, concurrency, resiliency and performance. However, more critical than your vendor choice is standardizing event schemas, capability owners and controlling the access to those events.

Combining Domains and Event Streams

Figure 1 - Domains Sitting Around An Enterprise Event Stream

Once implemented together, capability domains and an enterprise event stream can help an enterprise achieve a high degree of independence across domains. Figure 1 shows multiple domains (X, Y, Z and W); none of those have direct coupling to another domain. They only integrate through the enterprise event stream. There are also multiple consumers (A, B, C and D) and those consumers either integrate with the enterprise event stream or with the domain API, but never directly with an internal domain component such as the App APIs inside domain X.

There are two fundamental tenets to this pattern:

1. Domains directly integrate with other domains solely through domain boundary APIs (often referred to as domain APIs).
2. Domains produce and consume enterprise events via the enterprise event stream.

The enterprise events produced by a domain and their domain API are the only public contracts exposed. They’re the only aspects of their domain that must follow enterprise standards and be backwards compatible. Everything else that powers that domain’s capability is internal to the domain, which gives the domain owners a high level of freedom concerning their choice of implementation languages, libraries, container technologies, cloud resource preferences and software standards.

For example, one domain may use .NET and C# on Microsoft Azure because they have team members proficient in those technologies. Even if the rest of the enterprise uses Java on Amazon Web Services, both domains publish standardized events to the enterprise event stream and expose a public REST API, which are agnostic to their technology stack of choice.

Event Payload Patterns

Figure 2 - Common Payload Patterns

Different enterprises choose approaches that work best for their business requirements. In general, there are three types of payload patterns that are used for enterprise events. Figure 2 illustrates three different consumers: A, B and C, each of them using one of the payload patterns. Consumer A uses the Lightweight pattern, consumer B uses the Full Payload pattern, while Consumer C uses the Hybrid approach.

Lightweight Events

For lightweight events, the event payload is often a single identifier or a URI (uniform resource identifier). Consumers receive the event and then make an API call to a domain API to retrieve the current state of the data associated with that event. The benefits of lightweight events are that you save on data size as the events are extremely minimal and you do not need to worry about event ordering since the consuming systems always retrieve the latest state of the data from the source domain. The disadvantage of lightweight events is that the consumer must make an API call to retrieve the data it needs.

Full Payloads

In full payloads, the event payload is a complete snapshot of the state of the event data at a fixed point in time. Consuming systems have all the information they need when they receive the event. These systems save on having to make a call to a domain API. The disadvantage of full payloads is that the order in which the events are processed matters, which is a solved problem as all of the popular vendors support guaranteed message ordering.

Hybrid Approach

In the hybrid approach, the event payload is a partial dataset of the event state. The payload includes the source identifiers and URIs, and a subset of the state of the event. This allows some consumers to determine if they need to retrieve further data from the domain API while allowing other consumers to operate off the subset state of the data in the event payload. The hybrid approach is considered the best of both worlds by many.

There is no perfect way to do this, and it’s a sign of a mature enterprise if they choose to use more than one of these approaches to fit specific requirements.

Conclusion

Highly independent and fast-delivering technology companies use an enterprise event stream and cataloging of multiple capability domains. They couple this with domain owners who follow standards, and the company grows through the additive effects of the capability domains’ advancements.