Sales ETL Pipeline
ONGOING
End-to-end data engineering pipeline on Databricks using Medallion Architecture (Bronze/Silver/Gold) on a retail sales dataset. Implemented Delta Lake tables with ACID transactions and schema evolution, PySpark DataFrame transformations and window functions, and a 53-check automated data quality validation framework with severity-based pipeline gating. Modeled a star schema (fact and dimension tables) and scheduled the pipeline via Databricks Jobs with failure alerting. Currently extending with AWS S3 integration, Docker, CI/CD, and Airflow orchestration.