Labs Overviews:

Module 3: Data exploration and transformation in Azure Databrick

• Describe Azure Datbrick
  

• Read and write data in Azure Databricks
  

• work with DataFrames in Azure Databricks
  

• Work with DataFrames in Databricks
  

• Work with DataFrames advanced methods in Azure Databrick
  

Module 4:

• Understand big data engineering with Apache Spark in Azure Synapse Analytics
  

• Ingest data with Apache Spark notebooks in Azure Synapse Analytics
  

• Transform data with DataFrames in Apache Spark Pools in Azure Synapse Analytics
  

• Integrate SQL and Apache Spark Pools in Azure Synapse Analytics
  

Module 5:

• Use data loading best practices in Azure Synapse Analytics
  

• Petabyte-scale ingestion with Azure Data Factory or Azure Synapse Pipline
  

Course Overviews:

• Explore Azure Synapse serverless SQL pools capabilities: Data Exploration, Data transformation, Logical data warehouse

• user can use Azure Synapse Analytics dedicate SQL pool, which has the SQL authentication/AD authentication to access the date in the storage (for different level of credential)

• Differnt level of Credentials (authentication from low to high): user/group, service principal/app registration, manage identity (system assigned, user assigned[the highest])

  • ETC. VM asigned to managed identity -> read the databse (for authentication of db)
    

• Securing access to data in a data lake when using Azure Synapse Analytics: Authentication -> Authorization -> Access to storage accounts
  

• What can Azure Synapse serverless & SQL pools do? https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/on-demand-workspace-overview

• Different type of Query data(etc. Parquet, Json, csv…)

• What is Azure Synapse Studio platform?

• How can the user add role signment to storage (datalake/SQL pools)? : https://learn.microsoft.com/zh-tw/azure/role-based-access-control/role-assignments-portal
  
  

• Manage user access to data lake files: https://docs.informatica.com/data-integration/powerexchange-adapters-for-informatica/h2l/1357-prerequisites-to-create-a-microsoft-azure-data-lake-storage/prerequisites-to-create-a-microsoft-azure-data-lake-storage-gen2/setting-permissions-for-microsoft-azure-data-lake-store-gen2--ac.html
  

• How to change the setting for “Manage ACL”? https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-acl-azure-portal

• Uderstand Azure Databricks:
  

• Introduction to big data engineering with Apache Spark in Azure Synapse Analytics:
  

• How do Apache Spark pools work in Azure Synapse Analytics

• Create a new serverless Apache Spark pool using the Azure portal:
  

• Transform data with DataFrames in Apache Spark Pools in Azure Synapse Analytics:

  • Step1.Define a function for flattening
  • Step2. Flatten nested schema
  • Step3. Explore arrays
  • Flatten child nested schema
  • Quick knowledge & practice: https://learn.microsoft.com/en-us/training/modules/understand-big-data-engineering-with-apache-spark-azure-synapse-analytics/

• Integrate SQL and Apache Spark pools in Azure Synapse Analytics

  • Quick knowledge & practice:https://learn.microsoft.com/en-us/training/modules/integrate-sql-apache-spark-pools-azure-synapse-analytics/2-describe-integration-methods-between-sql
  • 

• Dedicated SQL Pool architecture
  

• Azure Data Factory/Synapse pipline revision
  

• Understand integration: Azure Integration Runtime , Self-hosted Integration Runtime