Create dashboard for Apache Kafka data source using Data Integration Platform
Introduction:
This article explains how to create a dashboard by receiving the message from the Apache Kafka data source using the Syncfusion Data Integration Platform.
Steps Involved:
Using the Syncfusion Data Integration Platform:
- Send JSON data as a message to Kafka.
- Receive the JSON data from Kafka.
- Clean and process the consumed JSON data into a required flat schema.
- Move the final processed data into a SQL database.
Using the Syncfusion Dashboard:
- Create a dashboard to showcase the Patient health summary.
JSON file:
For steps 1 to 4, the data flow in the Data Integration Platform will be defined as shown in the following image.
Data flow in Data Integration Platform
Step 1: Send JSON data as a message to Kafka
Use the GetFile processor to fetch the file content that is to be sent to Kafka.
Use the PublishKafka processor to send the JSON data as a message to Kafka. Ensure that you have created the topic in Kafka to publish the message under the specific topics.
Step 2: Receive the JSON data from Kafka
The ConsumeKafka processor acts as a Kafka consumer and consumes the data received by the Kafka in the specified topic. Ensure that you get the Group ID from the Apache Kafka consumer properties file.
Step 3: Clean and process the consumed JSON data into a required flat schema
In the SplitJson processor, specify the JsonPath Expression. The SplitJson processor is used to split the JSON array data using the specified JSON path into separate flow files.
Convert the flat JSON flow files into SQL INSERT statements using the ConvertJSONToSQL processor.
Step 4: Move the final processed data into the SQL database
Use the PutSQL processor to execute the SQL INSERT commands from the incoming flow files to insert the consumed data into the SQL table.
Make sure you have created a SQL table (Patient_details_db) using the following query and create a controller service for it in the Data Integration Platform. For more details on controller settings, refer to our documentation.
Step 5: The final step is to create a Patient health summary dashboard like the following in the Syncfusion Dashboard Cloud.
To learn the basics of creating a dashboard, refer to these links:
- Create business dashboards online
- Creating a sales dashboard with SQL Server and Syncfusion dashboards
You can follow the steps covered in these links to create your dashboards easily.
Patient Health Summary Dashboard
