Lightweight library to write, orchestrate and test your SQL ETL. Writing ETL with data integrity in mind.
-
Updated
Jan 11, 2024 - Python
#
data-engineering-workflows
Here are 7 public repositories matching this topic...
Data Engineering portfolio projects, resources used to study data tools...
python sql apache-spark databases data-engineering apache-kafka data-pipelines apache-airflow data-engineering-workflows etl-automation data-portfolio
-
Updated
Mar 25, 2024 - Jupyter Notebook
A first course for data engineering workflows
-
Updated
May 21, 2018 - Python
Examples that I use to learn and show Apache Beam
etl examples data-engineering apache-beam data-processing etl-pipeline data-engineering-workflows data-engineering-pipeline
-
Updated
Oct 24, 2018 - Python
This repository is the collection point for all of the projects completed during the Udacity Data Engineering Nano Degree program.
-
Updated
May 30, 2021 - Jupyter Notebook
Automated trade data pipeline for analyzing the impact of Free Trade Agreements (FTAs) and tariffs on U.S. agricultural exports. Integrates global datasets from USDA, WTO, World Bank, and IMF.
python selenium data-engineering economics selenium-webdriver usda api-integration selenium-python wto gravity-models data-engineering-workflows data-engineering-pipeline gravity-model data-automation data-engineering-team economics-and-computation trade-analysis data-engineering-project api-integration-and-data-retrieval economics-research
-
Updated
Mar 19, 2026 - Python
Leveraging AWS Cloud Services, an ETL pipeline transforms YouTube video statistics data. Data is downloaded from Kaggle, uploaded to an S3 bucket, and cataloged using AWS Glue for querying with Athena. AWS Lambda and Glue converts to Parquet format and stores it in a cleansed S3 bucket. AWS QuickSight then visualizes the materialised data.
python aws spark aws-lambda etl aws-s3 pandas pyspark data-engineering aws-iam aws-cloudwatch data-pipeline etl-pipeline aws-glue data-engineering-workflows data-engineering-pipeline aws-lambda-layers aws-data-engineering-project data-engineering-project
-
Updated
May 30, 2024 - Python
Improve this page
Add a description, image, and links to the data-engineering-workflows topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the data-engineering-workflows topic, visit your repo's landing page and select "manage topics."