Labs Overviews:

Module 6: Transform data with Azure Data Factory or Azure Synapse Pipelines

• Data integration with Azure Data Factory or Azure Synapse Piplines
  

• Code-free transformation at scale with Azure Data Factory or Azure Synapse
  

• Pipline
  

Module 7:Integrate data from notebooks with Azure Data Factory or Azure Synapse Pipelines

• Orchestrate data movement and transformation in Azure Data Factory or Azure Synapse Piplines
  

Module 8: End-to-end security with Azure Synapse Analytics

• Secure a data warehouse in Azure Synapse Analytics
  

• Configure and manage secretes in Azure Key Vault
  

• Implement compliance controls for sensitive data
  

Extra Lab: Movies Analytics using Azure Data Factory

• https://github.com/djpmsft/adf-ready-demo/blob/master/ADF_Lab_Instructions.md
  

Course Overviews:

• Linked services in Azure Data Factory and Azure Synapse Analytics: https://learn.microsoft.com/en-us/azure/data-factory/concepts-linked-services?tabs=data-factory
• Ingest and Transform Data with Azure Synapse Analytics: https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/ingest-and-transform-data-with-azure-synapse-analytics-with-ease/ba-p/1975563
• Mapping data flows in Azure Data Factory: https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-overview
• Integration data from Azure Data Factory or Azure Synapse Piplines:
  • 1. Use Storage Account
  • 2. Use Azure Synapse Pipline
  • 3. Create data workflow pipline
  • 4. Add a Notebook Activity to the pipline
  • 5. Set parameters & dependency condition
• Secure a data warehouse in Azure Synapse Analytics:
  • Network security
  • Identity and access management
  • Managin sensitive data
  • Encryption capabilities but into Azure
• Network Security: Securing your network from attcks and unauthorized access is an important part of any architecture
  • Internet protection
  • Firewall
  • DDos Protection
  • Networking security groups

What is NSG (Network Security Group)?

• "NSG" is the one that connect to the subnet and the nic.
• On the other hand, subnet need to use nic to connect Azure Serice. The the nic is known as "Private Endpoint".
• The connection between the Virtual Network is call "Peering"
• We can use "Express Route or VPN" to connect the Virtual Network and local Network.
• The connection between Azure Service and Virtual Macine(VM) is known as "Service Connection"

 Virtual Network
    L
      subnet
             L
                nic
                     L
                        VM

Identity and access:

• Authentication vs. Authorization:
  

• Azure Active Directory features:

  • Single Sign-On
  • Apps & Device Management
  • Identity Service

• What is Intune (MAM & MDM)? : https://learn.microsoft.com/zh-tw/mem/intune/apps/app-management

  • Difference between MDM and MAM: https://learn.microsoft.com/en-us/microsoft-365/business/ui/mam-and-mdm?view=o365-worldwide

• Azure Key Vault protects:

  • Quickstart: Set and retrieve a secret from Azure Key Vault using the Azure portal: https://learn.microsoft.com/en-us/azure/key-vault/secrets/quick-create-portal
  • More details (Manage secrets in your server apps with Azure Key Vault): https://learn.microsoft.com/en-us/training/modules/manage-secrets-with-azure-key-vault/
  • Data engineers are typically concerned with accessing the data contained in Key Vault to apply to linked service.
    • 1. Secretes
    • 2. Keys
    • 3. Certificates