Data Engineering Project

Showcasing Our Data Engineering Project

Santanu Ghosh's photo
Santanu Ghosh
·

1 min read

In today's data-driven world, efficient data management and processing are crucial for deriving actionable insights. Our latest data engineering project exemplifies this by focusing on robust data ingestion, validation, and storage techniques. Here's an overview of the project's scope and key areas of focus:

Project Scope

1. Data Sources: We will receive four JSON files from telematics devices, which will contain information about the device and vehicle status, such as trip details, engine status, fuel status, tire pressure, and driving behavior:

  • DeviceData

  • TripDetails

  • StatusFile

  • FuelInfo

2. Data Ingestion: Utilizing Databricks, we will ingest data from these complex nested JSON files stored in Azure Data Lake Storage. During the ingestion process, we will explode and flatten the nested JSON and implement data validation and transformation to ensure the highest data quality.

3. Data Storage: The ingested data will be stored in Delta Lake format using Databricks Cloud Files (Autoloader) Streaming. This approach allows us to leverage ACID transactions and scalable metadata handling, with incremental data read only once using autoloader checkpointing. This structured format will be suitable for both analysis and reporting.

By focusing on these key areas, our project aims to deliver a seamless and efficient data pipeline that ensures data integrity and accessibility. Stay tuned for more updates as we progress with this exciting project!