{"id":1172,"date":"2021-04-27T10:09:54","date_gmt":"2021-04-27T09:09:54","guid":{"rendered":"http:\/\/www.igfasouza.com\/blog\/?p=1172"},"modified":"2021-04-27T10:41:23","modified_gmt":"2021-04-27T09:41:23","slug":"kafka-weather-station","status":"publish","type":"post","link":"http:\/\/www.igfasouza.com\/blog\/kafka-weather-station\/","title":{"rendered":"Kafka weather station"},"content":{"rendered":"

\"\"<\/p>\n

What\u2019s the craic?<\/b><\/p>\n

In this blog, I want to set up a base use case that I can explore in future blogs. The weather stations are a simple idea where I can use a different source of data to get temperature, pressure, or humidity, send this to a Kafka topic, add an aggregator to do some Kafka Streams, like calculate average per hour and on the end, I can create several topics to play around the idea. I can add some kind of visualization and so on. It is a simple idea, but it is an idea that I can explore a lot of scenarios and I can easily expand and add more things.<\/p>\n

I can explore not only Kafka, local, or cloud, but I can explore some frameworks, program languages, some tools for visualization, and of course I can add some hardware.<\/p>\n

\"\"
\nPicture 1: Hats that I can use to generate some data<\/small><\/p>\n

\"\"
\nPicture2: Hats that I can use for visualization<\/small><\/p>\n

I created a GitHub<\/a> where I will add all links and codes for my blogs related to this idea.<\/p>\n

\"\"
\nPicture3: Use Case<\/small><\/p>\n

Use Case<\/h2>\n

Submit temperature data to a Kafka topic every 5 seconds. Ideally, I\u2019ll add more than one source, like 2 different sensors or weather APIs and so on. Possibility to explore the Producer and Kafka connect APIs.<\/p>\n

Create an aggregator where I can explore the Stream APIs, I can use Kafka Stream or KSQL, and do some ETL.<\/p>\n

Create different topics for different sensor data and aggregation values.<\/p>\n

Use the Consumer Kafka APIs to read the values and create some visualization.
\nIt can send the data to some kind of storage, like a database, object store, or Hadoop.<\/p>\n

I can explore this idea using Kafka at the Edge or on Cloud, and I can combine this with IOT, MQTT, and Advanced Message Queuing Protocol (AMQP).<\/p>\n

I can combine with Spark, Storm Flink, and others Streaming tools.<\/p>\n","protected":false},"excerpt":{"rendered":"

What\u2019s the craic? In this blog, I want to set up a base use case that I can explore in future blogs. The weather stations are a simple idea where I can use a different source of data to get… Continue Reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":1179,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[25,24],"tags":[7,13],"class_list":["post-1172","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-kafka","category-raspberry-pi","tag-kafka","tag-raspberry-pi"],"_links":{"self":[{"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/posts\/1172","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/comments?post=1172"}],"version-history":[{"count":6,"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/posts\/1172\/revisions"}],"predecessor-version":[{"id":1186,"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/posts\/1172\/revisions\/1186"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/media\/1179"}],"wp:attachment":[{"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/media?parent=1172"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/categories?post=1172"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/tags?post=1172"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}