Embedded Files
04-Data Architecture
04-Data Architecture
Introduction
Introduction
A data architecture describes how data is managed during its journey, from collection through to transformation, distribution, and consumption. It sets the blueprint for data and the way it flows through data storage systems. It is foundational to data processing operations and (AI) applications.
In includes the following:
Data Mesh
Conceptual data models: Database level:
They are also referred to as domain models and offer a big-picture view of what the system will contain, how it will be organized, and which business rules are involved
Logical data models: Schema and relations level:
They are less abstract and provide greater detail about the concepts and relationships in the domain under consideration.
Integration between different data models
Physical data models: Entities level:
They are less abstract and provide greater detail about the concepts and relationships in the domain under consideration. One of several formal data modeling notation systems is followed. These indicate data attributes, such as data types and their corresponding lengths, and show the relationships among entities. Logical data models don’t specify any technical system requirements.
What Is a Data Pipeline?
What Is a Data Pipeline?
Data has use case, since it represent a solution that have specific objectives, which pass through the following lifecycle:
- Data Collection: Collect : How to We collect data?
User inputs
Integration landscape
Content Crawling
- Date refinement: How do We refine data?
Anti-Corruption Layer
In order to keep smooth running between different co-worker independent databases, anti-corruption layer should take place
Integration consideration should take place
Data healthy checker
How do you incrementally check the integrity beween your data
How do you report it
How do you resolve it
Business rules
Validation rules
Consistency check
Data quality techniques
- Data engine: How data is stored?
Data format:
Relational: like Postgresql, Oracle, SQL Server, and MySQL
Non-Relational: like Mongodb, DynamoDB, and Oracle Nosql
Hybrid model
Location
Cloud
On prime
Hybrid mode
Architecture Model
Single database
Multiple distributed database
Containerized DB
- Data Analysis: How data analyzed?
Data analysis:
What are expected tools, reports and analytics types
Or Will design built-in business components in application architecture phase
How to support the decision making using these data
ML/DL Models
Analysis types:
Realtime: so it will need different tools
Periodic: to consider the data warehouse design
- Data delivery: How data is accessed/delivered?
In order to access data models, We have different approaches:
Create views on the database levels, what ever the technical methodology, using views, functions, procedures, or synonyms
APIs: Which provide communication layer over the database, in order to be exposed to the external parties.
Exported Format: like tab and comma delimiter
Web services
- Data Security Model : How data is secured?
Data access and security model from both application layer and database layer
Encryption for sensitive data
Connection security
API Security
Crawling prevention
- Data Versioning: Backup and restore strategy
How do We keep versions of data considering the growth over time?
How do We test it
Do We use only backup and restore
Or keep historical changes for the data within the logs database
Data architecture model
Database types
8. Open data consideration
8. Open data consideration
Open data governance
Open data execution approach
Open data format
9. Data democratization
9. Data democratization
Means, how to enable the access for smooth and easy access for different personas, to meet business objectives.
So: it needs, self-service analytics tools, education about the data by enabling different data dictionaries for the model, and building data over business objectives, not to present the complexity of the data models.
09. Data documentation
09. Data documentation
Well-defined objects
Well-defined data models details:
Object type
Attributes
Data types
Domain models
Sample
09. Data Quality
09. Data Quality
Data model attributes:
Business Rules Management:
Anti-Corruption Layer:
Data Quality practices:
Completeness
For text values
For numeric
For dates
Correctness
Domains
Anomaly detection
Consistency
Cross models validation
Cross Modules validation
Data Democratization
Tools
Tools
Enterprise Architect
Archimate
Dr. Ghoniem Lawaty
Tech Evangelist
Page updated
Google Sites
Report abuse