Stream Data to Azure Event Hub with Mockaroo and Kafka API

In a recent project, I faced the need to generate randomized data for transmission to the Azure Event Hub. This hub is a key component of Microsoft Azure, used for real-time data ingestion and processing.

First, let’s take look at how I created this random data. I wanted to come up with a way to make data that looks like what we see in the real world, but without using any real information from users. This made-up data was really important for a bunch of things, like checking if our software works well.

Certainly, I’d be happy to provide you with a detailed explanation of how, I effectively utilised Mockaroo for generating random data for my project.

How Mockaroo Works?

• Sign Up and Grab Your API Key: The starting point is to register on Mockaroo’s platform and get your unique API key, which is needed for mockaroo api to generate your random data.

• Log In and Locate Your API Key: Once you’ve signed up, log in to your Mockaroo account. Head over to your account settings, and there you’ll find your API key. This key is akin to a secret handshake that grants you access to Mockaroo’s data-spawning capabilities.

• Design Your Data Blueprint: Visualise how you want your data to look. Think about what details you need, such as names, numbers, or other bits of information. This blueprint is essential because it guides Mockaroo in creating data that matches your needs precisely.

• Select Your Data’s Appearance: Mockaroo lets you choose how your data will be presented. It’s like picking a format that best suits your needs – you can go for a list-style format (JSON), a table format (CSV), or other options based on your project’s requirements.

• Simplify with APIs: If your project calls for generating the same type of data repeatedly, you’re in for a treat. Mockaroo allows you to set up a special link known as an API. This link acts as a virtual data generator, conjuring up the exact data you need whenever you ask for it.

• Specify Your Data Quantity: let Mockaroo know how much data you’re looking to generate. Whether it’s just a handful for testing purposes or a large batch to populate your database, Mockaroo has you covered.

By following these straightforward steps, Mockaroo generates customised and randomised data tailored precisely to your project’s needs. It’s like having a dedicated assistant that effortlessly crafts up the data you require, freeing you up to focus on your project’s core objectives without the hassle of data creation.

Moving forward, the next step involves sending data to our event hub. To achieve this, I’ve harnessed the power of Kafka API.

What is Kafka API here?

The Kafka API refers to the integration of Apache Kafka’s producer and consumer capabilities with Azure Event Hub, which supports Kafka’s protocol. This allows us to send and receive messages using Kafka APIs while Azure Event Hub handles the message processing infrastructure. The Kafka API acts as the communication layer for streaming Mockaroo generated data to Event Hub.

Configuring Kafka for Integration with Spring Boot

The initial phase involves configuring Kafka. This is conveniently accomplished by setting up Kafka properties within environment variables. Subsequently, these configurations are seamlessly incorporated into the application-properties.yml file.

Let’s explore the key configurations essential for configuring Kafka API. In my setup, I’m leveraging the Spring Boot Kafka template, which simplifies the process.

Here’s an overview of the configurations you’ll need to fine-tune when working with Kafka API:

1. Bootstrap Servers: These servers serve as the initial point of contact for Kafka clients. They enable efficient communication and data transfer within the Kafka cluster.
2. Security Protocol: Establishing a secure communication channel is crucial. This protocol ensures the integrity and confidentiality of data exchanged between the clients and Kafka brokers.
3. SASL Mechanism: This mechanism orchestrates the authentication process, ensuring that only authorised users can access the Kafka cluster.
4. SASL JAAS Configuration: Leveraging JAAS, this configuration outlines the specifics of the authentication process, including login module details.
5. Producer Key Serializer: For sending data to Kafka, this serializer converts message keys into a suitable format for transmission.
6. Producer Value Serializer: Similar to the key serializer, this component handles the formatting of message values prior to dispatch to Kafka.

In essence, these configurations serve as the bedrock for establishing a seamless connection between your application and Kafka API. They collectively empower you to efficiently channel and transmit your synthetic data to the Azure Event Hub, making use of the Spring Boot Kafka template to simplify the process.

To send data to the Azure Event Hub, I used Mockaroo and Kafka API for seamless integration. In the application.yml file, I configured Mockaroo’s API URL to streamline the process.

I specified the url in my code.

I integrated the URL into my Java code using a web client.

Conclusion

I established a connection between the Mockaroo mock API and my web client. This connection functions as a virtual pipeline, allowing my application to continuously request data from the API’s unique web address. The application retrieves the data and promptly sends it to the Azure Event Hub using Kafka API for efficient streaming.