What’s up?
2021/03/05
This is another blog about Raspberry PI, and today I want to show how I did a simple Kafka cluster demo. It’s kind of a continuation from my two previous blogs, Kafka at the edge with Raspberry PI and Real-Time Locating System with Kafka.
If you’re not familiar with Kafka, I suggest you have a look at my previous post What is Kafka? before, and you can have a look at how I created the Kafka cluster here.
Idea
I found the idea of an OCI dashboard with smashing dashing in a colleague’s blog some time ago and I decided to pivot a little bit in a simple Raspberry Pi Kafka example where I can get OCI data and combine it with the smashing dashing dashboard idea.
For the producer part, you can use my Micronaut Sense hat example as a start point and just change to get the OCI data. You can get the full Micronaut Kafka Producer code on my GitHub.
https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/javasdk.htm
https://github.com/oracle/oci-java-sdk
For the consumer part, you can use my Quarkus example. You can get the full Quarkusl Kafka Consumer code on my GitHub.
clone the code and change the update.py file to read the value from the REST interface that Quarkus generates.
Results
Because this is a kind of Oracle Cloud solution you can use Oracle Streaming Service instead of hosting your Kafka.
Oracle Streaming Service (OSS) is a real-time, serverless, Apache Kafka-compatible event streaming platform for developers and data scientists, it provides a fully managed, scalable, and durable solution for ingesting and consuming high-volume data streams in real-time. To learn more about Oracle Streaming Service, see the documentation overview.
You can check my GitHub with some OSS examples and check the Micronaut one.
You just need to change the “application.yml” file from my Micronaut sense hat idea.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | kafka: bootstrap: servers: streaming.{your-region}.oci.oraclecloud.com:9092 security: protocol: SASL_SSL sasl: mechanism: PLAIN key: serializer: org.apache.kafka.common.serialization.StringSerializer deserializer: org.apache.kafka.common.serialization.StringDeserializer value: serializer: org.apache.kafka.common.serialization.StringSerializer deserializer: org.apache.kafka.common.serialization.StringDeserializer retries: 5 max: request: size: 1048576 partition: fetch: bytes: 1048576 group: id: group-0 |
And you can use all solutions in the cloud.
The Kafka consumer and the dashboard can be hosted in a cloud VM as well.
OCI Logging service is a highly scalable and fully managed single pane of glass for all the logs in your tenancy. Logging provides access to logs from Oracle Cloud Infrastructure resources. These logs include critical diagnostic information that describes how resources are performing and being accessed. To learn more about the Logging service, see the documentation overview.
OCI Service Connector Hub moves data, such as logs from Logging, to services, such as Object Storage, Streaming, and Monitoring. It triggers functions for custom data processing and sends notifications about changes to cloud resources. To learn more about the Service Connector Hub, see the documentation overview.
Here using Service Connector Hub I can get the logs from my tenancy and send them to OSS in a simple way.
Links
https://www.oc-blog.com/2019/05/13/oci-dashboard-with-smashing-dashing/