Data Models Challenges in Microservices architecture

One of the primary benefit of the microservices model is its ability to organize development teams according to bounded contexts(sub-models).

In monolithic applications, shared database creates high coupling between different domains, if data is wrongly architected, it will be highly coupled, which may require downtime to upgrade.

While Monolithic application depends on on ACID principles to resolve consistency issues, Microservice approach is 'Share-Nothing', which may lead to de-normalized data models that cause duplicates and inconsistency.

So, How to deal with such challenge in microservices

Dealing with challenges can have one of the following approaches:

• Accept: Which means, no worry about data duplication as architecture trade-off, which applying models and design patterns to handle inconsistencies

• Mitigate/Prevent: Which is not applicable in our case

In order to set defensive model to handle the challenge we should implement the following practices, as required within the solution:

1. API aggregators: Check for data from the other services

Pros: Direct

Cons: Additional complexity, Service coupling, which is against microservice architecture.

2. CQRS & EDA: Which you should notify the object changes due to the CQRS pattern, and to notify listeners by changes using the EDA, which guarantee 'At least once' publishing to interested services

Pros: Support the integrity by adding database objects (Views, Functions, Procedures) with defined responsibility.

Cons: Complexity, and code duplicates

3. SAGA: relies on orchestrating a sequence of local transactions, which is responsible for building local transactions, but the revert of the transactions in case of failure will.

Pros: Good for services provided by different solution providers

Cons: Higher complexity

4. Building Anti-Corruption Layer:  Which take responsibility for data quality practices(Correctness/consistency/completeness) across different systems.

Finally,

As an Architect, you are responsible about data integrity, using different technology domains(Microservice/relational databases/non-relational databases/AI, etc.), that direct you according to each context (Single/Multiple service provider) to implement the best fit approach to achieve the objective with minimal overheads and drawbacks.

This may be challenging, to have fully independent domains, that destroy relationships between objects, accept inconsistencies, require building healthy checkers to resolve it.

In addition, you should think about evaluating any approach, not only micro-service architecture, rather than applying everything, without considering the architecture trade-offs like:

• Complexity

• Consistency

• Fit for purpose

• Fit for use

As reports and analytics are core features in software, so most of the time you should create reports and analytics from different business functions, which can be created as following:

1. Database Level Reports: When connectivity to databases are feasible(If you are the owner of both solutions), reporting database/data warehouse should take place to facilitate reporting requirements

2. Microservices level Reports: By creating an aggregator layer for the reports, the purpose is to present data from multiple functions, within the frontend  

3. Anti-Corruption Layer and Data Duplication

   1. Request the data you need for reports within your data models

   2. Apply an anti-corruption layer to keep data consistency, as discussed in microservice architecture design patterns

4. Apply CQRS design pattern, by segregating read and write operations, let reporting operations connected to data warehouse, instead of operation database for reporting purposes