There are 2 data processing solutions:
- Transaction Systems (OLTP)
- Analytical Systems (OLAP)
Online Transaction Processing AKA OLTP is a process where data from different sources such as banking activity, retail checkout etc., and stores the transaction data into a database.
Online Analytical Processing AKA OLAP is a process applies complex queries to large amounts of historical data, aggregated from OLTP databases and other sources, for data mining, analytics, and business Intelligence projects
|Handles day to day transactions that results from enterprise operations||Analysis of information in the database for the purpose of making management decisions.|
|Small, discrete unit of work||Big picture view of the information held in DB|
|Often high volume||Can be high/low depending on the requirement|
|Data is processed very quickly||Seconds, minutes, or hours depending on the amount of data to process|
|Control and run essential business operations in real time||Plan, solve problems, support decisions, discover hidden insights|
|Normalized databases for efficiency||Denormalized databases for analysis|
- Getting the data from multiple data sources and storing it into one centralised location is called DATA INGESTION. It is the process of obtaining and importing data.
- This data can arrive as continuous stream or batches.
- Raw data can be stored in DBMS as files or other forms for fast, easily accessible storage.
- Data Ingestion might perform
- Filtering: Eg, Reject suspicious, corrupt or duplicated data
- Simple Transformation: Converting data into standard form
NOTE: During ingestion we can make only simple transformations and not complex critical transformation
- Data processing takes data in raw form, cleans it and converts it into a more meaningful format.
- The result is a databases /data warehouse that you can use to perform queries and generate visualisations.
- Data Processing might perform
- Data Cleaning: Removing anomalies, applying filters and transformations
- Data Wrangling: Capture, filter, clean, combine and aggregate data
NOTE: We can make more complex transformation. Eg: Azure synapse Analytics is used to store cleaned and transformed data
- Data exploration is the process of trying to put together the pieces of puzzle in the journey to find a message in the data working with feedback cycles of defining hypotheses, analyzing data, and visualizing results.
- Exploration is a deep dive analysis of data in search for better insights
- The explored data can be visualized using tools such as Tableau, Power BI etc.,