Load Testing Azure Event Hubs services with restricted public access
Load Testing Azure Event Hubs services with restricted public access
Scenario details
Potential use cases
- Anomaly detection (fraud/outliers)
- Application logging
- Analytics pipelines, such as Clickstream
- Live dashboards
- Archiving data
- Transaction processing
- User telemetry processing
- Device telemetry streaming
Architecture of the system we want to load test
Dataflow
Load testing Architecture
Load testing scenario
- Configure the Azure Event Hubs namespace firewall to accept connections from the different sources on Internet including the Azure Load Testing service adding the source IP addresses in the firewall configuration
- Launch the load testing service to send requests to the input Event Hub.
- The Azure Event Hubs collects the events from the different source.
- The service analyzes the events using different possible resources like Azure Data Factory, Azure Machine Learning, Azure Stream Analytics, Azure Synapse Analytics. If an anomaly is detected, the alert is sent to the Output Event Hub.
- The alert can trigger the transmission of an mail, notification. Moreover, the alert can be stored in Azure Storage and Azure SQL Server.
- While the load testing is still sending requests to the Azure Event Hub, the load testing pipeline sends events that should trigger an alert, the same pipeline reads events (alerts) from the output event hub to measure the latency of the system.
- When the load testing duration is reached, the service stops sending requests to the input Event Hub. The load testing results are successful if the alert latency is not degraded with the high ingress traffic.
Components
- Azure Load Testing: Enable developers and testers to generate high-scale load and run simulations that reveal actionable insights into app performance, scalability, and capacity with a fully managed load-testing service.
- Create tests quickly without prior knowledge of load testing tools, or upload your existing Apache JMeter scripts. Gain specialized recommendations backed by comprehensive metrics and analytics, and support continuous improvement through automated continuous integration and continuous delivery (CI/CD) workflows—all with a testing service built for Azure.
- Azure Event Hubs: Stream millions of events per second from any source to build dynamic data pipelines and immediately respond to business challenges.
- Azure Machine Learning: Empowers data scientists and developers to build, deploy, and manage high-quality models faster and with confidence.
- Azure Data Factory: Integrate all your data with Azure Data Factory—a fully managed, serverless data integration service.
- Azure Synapse Analytics: Limitless analytics service that brings together data integration, enterprise data warehousing, and big data analytics.
- Azure Stream Analytics: Go from zero to production in minutes using SQL—easily extensible with custom code and built-in machine learning capabilities for more advanced scenarios.
- Azure Kubernetes Service: The quickest way to start developing and deploying cloud-native apps in Azure, data centers, or at the edge with built-in code-to-cloud pipelines and guardrails.
- Azure Blob Storage: A massively scalable object storage for any type of unstructured data, including images, videos, audio, documents, and more. It's easy and cost effective.
- Azure SQL Database: Fully managed relational database service built for the cloud. Build your next app with the simplicity and flexibility of a multi-model database that scales to meet demand.
- Email: Create automated task and workflows with Azure Logic Apps and Microsoft 365 Outlook Connector to send an email.
Considerations
Cost optimization
Next steps
- A sample implementation load testing an Eventhubs based infrastructure from either a dev container in Visual Studio Code or an Azure DevOps pipeline or a Github Action is available here
Published on:
Learn moreRelated posts
PostgreSQL for your AI app's backend | Azure Database for PostgreSQL Flexible Server
Use Postgres as a managed service on Azure. As you build generative AI apps, explore advantages of Azure Database for Postgres Flexible Server...
Storage migration: Combine Azure Storage Mover and Azure Data Box
Migrating storage from on-premises can be challenging. That’s why we are on a mission to make your migrations as simple as possible. We've dev...
Loop DDoS Attacks: Understanding the Threat and Azure's Defense
In the realm of cybersecurity, Distributed Denial-of-Service (DDoS) attacks are a significant concern. The recent holiday season has unveiled ...
Azure Communication Services at Microsoft Build 2024
Join us in-person in Seattle or virtually for Microsoft Build 2024 from May 21 to 24. We're excited to share the latest updates from Azure Com...
Azure Developer CLI (azd) – May 2024 Release
This post announces the May release of the Azure Developer CLI (`azd`), which includes a new demo mode, various .NET Aspire enhancements, and ...
Join us at Build 2024: Get the latest on Azure Cosmos DB in Seattle or online!
Are you ready to dive into the future of AI and data innovation? Microsoft Build 2024 is just around the corner, taking place May 21–23 in Sea...
Announcing Data API builder General Availability for Azure Cosmos DB
Great news for developers out there! The Data API Builder for Azure Cosmos DB is now officially available, eliminating the need for writing co...
Organizing rule collections and rule collection groups in Azure Firewall Policy
Firewall Policy is the recommended method to manage Azure Firewall security and operational configurations. When using Firewall Policy, any ru...
Get Started with Azure AI Services | Open AI and Deployment Models
Table of Contents Overview - Azure AI Services Kind of Azure AI Services Responsible AI Services Limited Access Features Cognitive Acc...