A data engineer with focus on building scalable distributed data systems.
Designed, developed and implemented high-volume event pipelines and analytics platforms processing 100s of Terabytes of data daily.
Ritu Gupta.
Page 02
About.
Python, Spark, AWS, Airflow
Customers servedAdvertising, IoT, Telecom
01
Background
02
Foundations in Quality Engineering
03
Building Automation and Distributed Systems
04
Designing Large-Scale Reliable Data Pipelines
Page 03
Key Problems Solved.
Ad Clickstream Analytics - Data Lakehouse
Large-Scale Data Processing - ETL Pipeline Design
Enhanced Quality - Data Observability & Reliability
Scaling and Optimization - Distributed Systems
Petabytes of Event-driven data processing systems
~100TBs of data ingested, transformed and analyzed daily
90% less customer reported Data Completness issues
Resources and processing times optimized by 35%
Page 04
Deep Dive I - System Architecture.
Distributed Data Lakehouse
- Event-driven ingestion
- Micro-batch Spark processing
- Layered storage and transformation zones
- Athena and API-based consumption
Source
Event Sources
Ingestion
Data Stream Ingestion
Processing
Raw Micro-Batch Processing
Storage
Transformations
Transform
Aggregations
Analytics
Analytics & APIs
Consumption
Athena + API
Page 05
Deep Dive II - ETL Pipelines.
Raw Events
Schema Validation
Preprocessing & Filtering
Explode Nested Data
Aggregations
Partitioned Datasets
Analytics Tables
ETL Transformations
- Schema-first validation before transformations
- Progressive refinement from raw to analytics-ready data
- Nested event flattening for downstream aggregation
- Partitioned outputs optimized for query performance
Page 06
Deep Dive III - Data Observability.
Platform for Completeness, Reliability
Monitoring - Metrics, alerts, dashboards
Page 07
Distributed Systems at Scale.
Scaling & Optimization
Built around measurable objectives and a system design that keeps data quality visible, actionable, and resilient.
Operational Excellence
Operational Excellence
CI/CD
CI/CD
Page 08
Impact.
100TB+
of data processed daily
90%
less data defects reported
1 hour
detection time reduced from 2 days
35%
optimization in resources and processing times
Petabytes
of events data extracted, transformed and loaded
1000s
of customers served with analytics and insights