Project Description: Utilized a Spotify dataset to demonstrate proficiency in data engineering within an AWS cloud environment. Employed a range of AWS services including Amazon S3, Glue, Athena, and QuickSight to streamline the data processing pipeline. Uploaded the dataset to S3 buckets, implementing a staging folder structure. Leveraged AWS Glue to orchestrate a seamless data transfer from staging to a data warehouse, granting S3 access to Glue for pipeline monitoring. Established a database and catalog using Glue's crawler, enabling metadata creation for efficient querying through Athena. Utilized Athena to execute SQL queries on the database generated by Glue's crawler. Finally, leveraged QuickSight to visualize insights from the processed data, creating interactive dashboards for comprehensive analysis. This project showcases hands-on experience in cloud-based data engineering and analytics.
