Labs Overviews:

Module 9 : Support Hybrid Transactional Analytical Processing (HTAP) with Azure Synapse Link

•     Design hybrid transactional and analytical processing using Azure Synapse Analytics
  

•     Configure Azure Synapse Link with Azure Cosmos DB
  

•     Query Azure Cosmos DB with Apache Spark for Azure Synapse Analytics
  

•     Query Azure Cosmos DB with SQL serverless for Azure Synapse Analytics
  

Module 10 : Real-time stream processing with Stream Analytics

•     Enable reliable messaging for Big Data application
  

•     Work with data streams by using Azure Stream Analytics
  

•     Ingest data streams with Azure Stream Analytics
  

Module 11:Create a stream processing solution with Event Hubs and Azure Databricks

•     Process streaming data with Azure Databrick structured streaming
  

Course Overviews:

Access Skillpipe:

If you join DP203 SYSTEX course, you can claim DP203 E-book from the "Skillpipe", the details will be sent to your E-mail.
To access Skillpipe, please click on the following link: www.skillpipe.com
If you are new to Skillpipe, please set up your Skillpipe account here: https://www.skillpipe.com/#/account/registration
IMAGE

Design hybrid transactional and analytical processing using Azure Synapse Analytics

More reference: https://k21academy.com/microsoft-azure/data-engineer/azure-synapse-link-hybrid-transactional-analytical-processing/

Query Azure Cosmos DB with Apache Spark for Azure Synapse Analytics

•     Step 1. Load the data in Spark
  

•     Step 2. Create a based DataFrame
  

•     Step 3. Flatten JSON data
  

•     Step 4. Create the final DataFrame
  

More reference: https://learn.microsoft.com/en-us/azure/synapse-analytics/synapse-link/how-to-copy-to-sql-pool

Multi Dimensional Model vs. Tabular Model

More reference: https://learn.microsoft.com/en-us/analysis-services/comparing-tabular-and-multidimensional-solutions-ssas?view=asallproducts-allversions

What is Power Pivot (in memory)

More reference: https://support.microsoft.com/zh-tw/office/power-pivot-%E6%A6%82%E8%A7%80%E8%88%87%E5%AD%B8%E7%BF%92-f9001958-7901-4caa-ad80-028a6d2432ed

What is Hybrid Connection?

More reference: https://learn.microsoft.com/en-us/azure/app-service/app-service-hybrid-connections

What is Azure Relay?

More reference: https://learn.microsoft.com/en-us/azure/azure-relay/relay-what-is-it

Azure Event Hubs:

Azure Event Hubs is a highly scalable publish-subscribe service that can ingest millions of event per second and stream them into multiple applications
more reference: https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-about

Configure applications to use Event Hubs

Mobile APP --- Receive ---> API -----Sent---> Event Hub (Queues streaming data)

• Quickstart: Create an event hub using Azure portal: https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-create

What are data streams?

• Data Stream : In the context of analytics, data streams are event data generated by sensors or other sources that can be analyzed by another technology.
• Data Stream processing approach: There are two approaches. Reference data is streaming data that can be collected over time and persisted in storage as static data. In contrast, streaming data have relatively low storage requirements. And run computations in sliding windows.
• Data stream are used to:
  * Analyza data
  * Understand systems
  * Trigger actions

Event processing:

The process of consuming data streams, analyzing them, and deriving actionable insights out of them is called Event Processing and has three distinct components.

• Event Producer
• Event processor
• Event consumer
  

Processing events with Azure Stream Analytics

Microsoft Azure Stream Analytics is an event processing engine. It enables the consumption and analysis of high volumes of streaming data in real time.

• Source: Sensor, Systems, Application
• Ingestion: Event Hubs , IOT Hubs, Azure a Blob Store
• Analytical engine: Stream Analytics Query, Language, .NET SDK
• Destination: Azure Data Lake, Cosmos DB, Blob Store, SQL Database, Power BI
  

Create streamAnalytics Service

• Job name
• Subscription
• Resource group
• Location
  

Using Windows function with Azure Stream Analytics

There are five kinds of temporal windows to choose from: Tumbling, Hopping, Sliding, Session, and Snapshot windows. You use the window functions in the GROUP BY clause of the query syntax in your Stream Analytics jobs. You can also aggregate events over multiple windows using the Windows() function.

Tumbling

Hopping

Sliding

Session Windows

Snapshot windows

• More reference: Introduction to Stream Analytics windowing functions
  

What is Event Grid?

More reference: https://learn.microsoft.com/en-us/azure/event-grid/overview

Stream data from a file and write it out to a distributed file system and connect to Event Hubs to read and write streams

Some how I　found a "Azure Developer College": have many great articles and challenge to practice ~ So if you want to know more about Azure, here is the one!