Databricks-code

This repository contains sample code for working with Databricks.

File Index

• README.md: Overview and guide to the repository.
• Cheat Sheets/PySpark_SQL_Cheat_Sheet.pdf: Quick reference for PySpark SQL commands.
• Cheat Sheets/Scikit-Learn_Cheat_Sheet.pdf: Quick reference for Scikit-Learn.
• SQL/sql_cheat_sheet_EN.md: Summary of common SQL commands in English.
• SQL/sql_cheat_sheet_ES.md: Summary of common SQL commands in Spanish.
• sql/create_materialized_view.sql: Example showing how to create a Delta materialized view.
• notebooks/PySpark_Databricks_Notebook.ipynb: Notebook with key PySpark commands for Databricks.
• notebooks/PySpark_Databricks_Notebook.py: Script version of the previous notebook.
• notebooks/pyspark_dataframe_api_commands.ipynb: Examples of the PySpark DataFrame API.
• notebooks/pyspark_delta_commands.ipynb: Commands for working with PySpark and Delta Lake.
• notebooks/pyspark_pipeline_with_cache.ipynb: PySpark pipeline demonstrating the use of .cache().
• notebooks/scikit_learn_commands.ipynb: Essential steps for training models with scikit-learn.
• notebooks/data_quality_best_practices.ipynb: Best practices for data quality pipelines.
• notebooks/delta_lake_interview_commands.ipynb: Common Delta Lake commands for interviews.
• notebooks/pandas_data_processing_commands.ipynb: Typical data processing commands with Pandas.
• notebooks/spark_sql_commands.ipynb: Summary of Spark SQL commands.
• notebooks/Medallion/bronze_layer_processing.ipynb: Bronze layer processing.
• notebooks/Medallion/silver_layer_processing.ipynb: Silver layer processing.
• notebooks/Medallion/gold_layer_processing.ipynb: Gold layer processing.