Polyglot Persistence Using DDD and Repository Unit of Work

• Download source code from Github

Introduction

Traditional business applications usually use a relational database to store data and use this database as an integration point. That entails designing sets of relational tables and accessing them with a single data access layer in code, as well as using an ORM to convert relational tables to an OOP structure .

Nowadays, that might not be optmized for each business use case scenario, considering some challenges encountered with relational database design. It's unnecessary here to list all the disadvantages of such design we can find a more articles about. But let's give a hint:

Agility: Mismatch between data structure in the database and the application object model has an adverse productivity.

Performance: Relational models were designed to take minimum disk space and to consume less resources, which has some side effects and performance limitations.

Availability and price: Increasing availability in relational model complicates the consistency and has its price.

Flexibility: Not all developers are familiar with relational models and schemas, in some situations, complex business models require a thorough knowledge of SQL language and relational models.

In a number of enterprise applications that capture a huge amount of data and numbers of concurrent requests, the relational model did not provide the requirements. So looking for other alternatives to solve a specific set of persistence problems leads many designers to select a non-relational systems referred to as NoSQL databases.

Non-relational systems or NoSQL databases can be categorized by a set of functional areas: Key/Value, document databases, column family databases, graph databases. Each category can fit one or more business use case scenarios and it's for a designer to choose the most appropriate system. Some NoSQL databases use filesystem for persistence while some of them are in memory.

Background

The advent of various NoSQL databases and the abundance of cheap storage spaces (disk and memory) will be a momentum for designers and developers to switch to non-relational models.

However, an obvious question will occupy the mind. How to design an application so that it will dialog with different data stores?

What Are We Going To Achieve At The End?

We are going to design and build a simple application using DDD and Repository Unit of Work pattern. This application will persist data in relational (SQL) and non relational (NoSQL) data stores.

As a real world example, we will use an enterprise product store, let's say an enterprise (customer) with many organizations, departments and employees managing a huge amount of products and each employee can search, list, add, update and remove products. Also, the application needs to store the users' traffic to help administrators to address possible performance issues. To meet the requirement, the application will be deployed on the cloud (Windows Azure) and on-premises. Finally, for demos and testing purposes, the application will store all the data in memory.

Selecting Databases

For our example, products store application we can recognize different types of data requirements. For each requirement, we will use a specific database type as follows:

• Customers, organizations, departments and users necessitate a safe reliable database to keep the hierarchical structure safe. And those entities don't have very frequent changes. So, we will persist them in relational database, let's say SQL Server.
• The application is supposed to hold a large number of products, and it has to filter, list and search products in acceptable performance.The information held in each product may be different. So, for those reasons, we will persist products in NoSQL document database. Here, we will use MongoDB.
• The application will allow the users to take some notes and share them with other users. Notes are just text that can be listed, searched and filtered. For Note, text will be stored in Lucene index.
• The application will store products images in cloud blob storage as key/Value NoSQL database.
• The application will log errors and store change history in cloud table storage as key/Value NoSQL database.

Designing and Creating a Solution

Before we start coding, let's give a brief high level design structure to our application.

Here is the layered diagram of our solution:

By the diagram here above, we can examine the principal layers used in the application:

• Presentation: ASP.NET MVC5 application with HTML5 AngularJS
• Distributed Services: MVC 5 Web API application exposes REST services
• Application management: Manage the business rules
• Domain Objects: Core Application business definitions, Interfaces, entities, aggregates
• Data: Repositories and a specific unit of work implementations. Repository for each domain entity or aggregate and Unit of Work for each data store type.
• Data stores: At the bottom of the figure above, we see the different data stores that we will use in application SQL server, MongoDB, Azure Storage, Lucene index and in-memory fake store

To examine the interactions between components, let's examine the component diagram below:

Presentation: In the presentation layer, we can see that ASP MVC 5 project will serve as SPA with HTML5 angularJS. That can also consume a direct Web API services in the distributed services using HTTP/HTTPS

Distributed Services: Web API exposing Rest Services over HTTP/HTTPS

Application Management: Class library used as .NET reference for Web API and MVC application.

Domain Objects: Class library used as .NET reference for application management and infrastructure data

Infrastructure data: Class library referencing Domain object and some libraries as connectors for data stores

• Entity framework 6 for SQL Server
• Mongocsharpdriver for mongoDB
• Lucene.net for Lucene index
• Windows Azure graph API for Azure storage

Now let's zoom on the Domain objects layer and see the entities definitions in this map diagram:

You can find this diagram in the attached solution.

Working with the Code

Domain Entities

After the general design description of our application example, let's delve into some lines of code. Mainly focus on the domain objects definitions and the infrastructure data layer. For the other layers, you can download the attached example.

Starting with domain objects definitions: Here, we will first define a base class for all entities witch contains common properties decorated with some libraries attributes. Here is the code of our base class.

/// <summary>
/// Base class for entities
/// </summary>
public abstract class EntityBase
{
    #region Members
      Guid _Id;
    #endregion
   #region Properties

     /// <summary>
    /// Get or set the persisted object identifier
    /// </summary>
    [Key]
    [Field(Key = true)]
    [BsonId]
    public virtual Guid Id
    {
        get
        {
            return _Id;
        }
        set
        {
            _Id = value;
        }
    }
    /// <summary>
    /// Get or set the Date of Creation
    /// </summary>
    [DataType(DataType.DateTime)]
    public DateTime CreationDate { get; set; }
    /// <summary>
    /// Get or set the Date of LastUpdate
    /// </summary>
    [DataType(DataType.DateTime)]
    public DateTime LastUpdateDate { get; set; }
    #endregion
 }

This class defines the Id property decorated by some attributes:

• Key: Attribute for entity framework so that the property will be mapped to the SQL primary key
• BsonI: Attribute for mongoDB so the Id property will be mapped as document identifier
• Filed(Key=true): This for Lucene.net to be used as document identifier on the index

Now this is for entities. Let’s see how to define a contract to our repositories. Here is the definition:

    /// <summary>
    /// Base interface for implement a "Repository Pattern", for
    /// </summary>
    /// <remarks>
    /// </remarks>
    /// <typeparam name="TEntity">Type of entity for this repository </typeparam>
    public interface IRepository<TEntity> 
        where TEntity : EntityBase
      {
        /// <summary>
        /// Get the unit of work in this repository
        /// </summary>
        IUnitOfWork UnitOfWork { get; }
       
         /// <summary>
        /// Add item into repository
        /// </summary>
        /// <param name="item">Item to add to repository</param>
        Task AddAsync(TEntity item);

        /// <summary>
        /// Delete item 
        /// </summary>
        /// <param name="item">Item to delete</param>
        Task RemoveAsync(TEntity item);

        /// <summary>
        /// Get element by entity key
        /// </summary>
        /// <param name="id">entity key values, the order the are same of order in mapping.</param>
        /// <returns></returns>
        Task<TEntity> GetElementByIdAsync(Guid id,
                                       CancellationToken cancellationToken = default(CancellationToken));

        /// <summary>
        /// Get all elements of type {TEntity} in repository
        /// </summary>
        /// <param name="pageIndex">Page index</param>
        /// <param name="pageCount">Number of elements in each page</param>
        /// <param name="orderBy">Order by expression for this query</param>
        /// <param name="ascending">Specify if order is ascending</param>
        /// <returns>List of selected elements</returns>
        Task<IEnumerable<TEntity>> GetPagedElementsAsync<T>(int pageIndex, int pageCount,
                             Expression<Func<TEntity, T>> orderBy, bool ascending,
                             CancellationToken cancellationToken = default(CancellationToken));
}

All entities and aggregates used in the application will define a repository class that implement the contract above as we can see the repository define its unit of work that will be injected at run time. So for each store, we will define its unit of work and inject it to the repository. Here is a contract for unit of work.

 public interface IUnitOfWork
        : IDisposable
    {
        /// <summary>
        /// Commit all changes made in a container.
        /// </summary>
        ///<remarks>
        /// If the entity have fixed properties and any optimistic concurrency problem exists,  
        /// then an exception is thrown
        ///</remarks>
        void Commit();
   
......
         /// <summary>
        /// Commit all changes made in a container Async.
        /// </summary>
        ///<remarks>
        /// If the entity have fixed properties and any optimistic concurrency problem exists,  
        /// then an exception is thrown
        ///</remarks>
        Task CommitAsync( CancellationToken cancellationToken = default(CancellationToken));
        
        ......

         /// <summary>
        /// Commit all changes made in  a container Async.
        /// </summary>
        ///<remarks>
        /// If the entity have fixed properties and any optimistic concurrency problem exists,
        /// then 'client changes' are refreshed - Client wins
        ///</remarks>
        Task CommitAndRefreshChangesAsync(CancellationToken cancellationToken = default(CancellationToken));

Accessing Data Stores

For each data store, we will define its unit of work so we can then inject it in a specific repository. First, in domain object layer, we have defined a contract for unit of work so the unit of work here will implement it.

Starting with relational database here SQL server, maybe it's the most usual. Here, we will use Entity framework and create a unit of work that implements a base contract and inherits entity framework DBContext. Above is the code used in this example.

 public class MainBCUnitOfWork : DbContext, IMainBCUnitOfWork 
    { 
        #region Fileds
        private IDbSet<Customer> _customers;
        private IDbSet<Address> _addresses;
        private IDbSet<Department> _departments;
        private IDbSet<Organization> _organizations;
   ....... 
        
         #region Properties
        public IDbSet<Customer> Customers
        {
            get
            {
                if (this._customers == null)
                    this._customers = (IDbSet<Customer>)this.Set<Customer>();
                return this._customers;
            }
        }
 
        public IDbSet<Department> Departments
        {
            get
            {
                if (this._departments == null)
                    this._departments = (IDbSet<Department>)this.Set<Department>();
                return this._departments;
            }
        }

         ..............
         #endregion
.........
       public virtual IQueryable<TEntity> CreateSet<TEntity>() where TEntity : class,new()
        {
            return (IDbSet<TEntity>)this.Set<TEntity>();
        }

        public virtual void Commit()
        {
            try
            {
                this.SaveChanges();
            }
            catch (DbEntityValidationException ex)
            {
                throw this.GetDBValidationExptions(ex);
            }
        }
         ...............

       public async Task CommitAsync(CancellationToken cancellationToken = default(CancellationToken))
        {
            cancellationToken.ThrowIfCancellationRequested();
            try
            {
                await this.SaveChangesAsync(cancellationToken);
            }
            catch (DbEntityValidationException ex)
            {
                throw this.GetDBValidationExptions(ex);
            }
        }
 ........................
 }

The full definition of the class can be found in the attached code.

For each entity stored in SQL server database, an IDBSet property is defined. The create set method will be used to retrieve data.

By the same way, we will define a unit of work for mongoDB using mongoDB C# driver that you can add as Nuget package. Here is the code used:

 public class MongoUnitOfWork : IMongoUnitOfWork
    {
        #region Fields
        string _dbHostName;
        string _dbName;
        MongoDatabase _database;
       ......

         #region properties
        public string DbName
        {
            get {
                if (string.IsNullOrEmpty(this._dbName))
                {
                    this._dbName = "polyGlotDemo";
                }
                return _dbName;
            }
            set { _dbName = value; }
        }

       public string DbHostName
        {
            get {
                if (string.IsNullOrEmpty(this._dbHostName))
                {
                    this._dbHostName = "127.0.0.1";
                }
                return _dbHostName;
            }
            set { _dbHostName = value; }
        }

        public MongoDatabase Database
        {
            get { return _database; }
            set { _database = value; }
        }

        public IDbSet<DepartmentAggregate> Departments { get; set; }
        #endregion
        #region Ctor
        public MongoUnitOfWork()
        {
            var pack = new ConventionPack();
            pack.Add(new CamelCaseElementNameConvention());
            ConventionRegistry.Register("MongoUnitOfWorkPack", pack, (t) => true);
            string connectionString = "mongodb://" + t
            MongoClientSettings settings = MongoClientSettings.FromUrl(new MongoUrl(connectionString));
            settings.WriteConcern.Journal = true;
            var mongoClient = new MongoClient(settings);
            var mongoServer = mongoClient.GetServer();
           if (!mongoServer.DatabaseExists(this.DbName))
            {
                throw new MongoException(string.Format
                (CultureInfo.CurrentCulture, Messages.DatabaseDoesNotExist, this.DbName));
            }
             this.Database = mongoServer.GetDatabase(this.DbName);
            var coll = this.Database.GetCollection<
            			DepartmentAggregate>("DepartmentAggregate");
            //coll.RemoveAll();
            foreach (var dep in data.DepartmentAggregates)
            {
                if (!coll.AsQueryable().Any(x => x.Id == dep.Id))
                {
                    coll.Insert<DepartmentAggregate>(dep);
                }
            }

           this.Departments = new MemorySet<DepartmentAggregate>();
        }
        #endregion
   .........
}
       #endregion

The full definition of the class can be found in the attached code.

For the other stores and unit of work, you can browse the code. The logic is the same.

Running the Example

Here are some prerequisites to work with the attached example.

• Download mongoDB from https://www.mongodb.org/downloads
  • Start mongo and create a database and collection.
• Visual Studio ultimate if you would like open the modeling project
• Azure SDK 2.5: Emulate Azure storage
• Run Visual studio as administrator
• Run Azure solution
• Use Nuget to restore any missing package

Points of Interest

Application example managing multiple stores and multiple environments using DDD, Windows Azure, MongoDB, Lucene.net, ...

Hope this example will help and any comments are welcome.