{"id":838,"date":"2020-10-09T12:18:29","date_gmt":"2020-10-09T11:18:29","guid":{"rendered":"http:\/\/www.igfasouza.com\/blog\/?p=838"},"modified":"2021-04-27T11:05:56","modified_gmt":"2021-04-27T10:05:56","slug":"raspberry-pi-kafka-cluster","status":"publish","type":"post","link":"http:\/\/www.igfasouza.com\/blog\/raspberry-pi-kafka-cluster\/","title":{"rendered":"Raspberry Pi Kafka cluster"},"content":{"rendered":"

\"\"<\/p>\n

How heya?<\/b><\/p>\n

This is another blog about Raspberry PI, and today I want to show how I did a simple Kafka cluster.<\/p>\n

If you’re not familiar with Kafka, I suggest you have a look at my previous post \u201cWhat is Kafka?<\/a>\u201d before.<\/p>\n

This is a really simple tutorial and you can find similar instructions over the internet.<\/p>\n

Pre-Requisites<\/h2>\n
    \n
  • Raspbian installed<\/li>\n<\/ul>\n

    Components<\/h2>\n
      \n
    • 3x Raspberry Pi3 B with power supply and micro SD card with Raspbian.<\/li>\n<\/ul>\n

      Despite every time that you look at a cluster tutorial they add a switch, router, and some internet cables I did use just simple wifi.<\/p>\n

      For the software part, we will download Zookeeper and Kafka, and there’s no installation. Is just about untar and changing some settings files.
      \nDownload latest Apache Zookeeper & Apache Kafka.<\/p>\n

      Steps<\/h2>\n

      You should do this in all Raspberry PIs.<\/p>\n

      1) Zookeeper<\/b><\/p>\n

      1
      <\/div><\/td>
      wget<\/span> https:\/\/<\/span>downloads.apache.org\/<\/span>zookeeper\/<\/span>zookeeper-3.6.1\/<\/span>apache-zookeeper-3.6.1-bin.tar.gz<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

      Modify the “conf\/zoo.cfg<\/em>” with;<\/p>\n

      1
      2
      3
      4
      5
      6
      7
      8
      9
      <\/div><\/td>
      dataDir<\/span>=\/<\/span>opt\/<\/span>zookeeper_data
      \ntickTime<\/span>=2000<\/span>
      \ninitLimit<\/span>=10<\/span>
      \nsyncLimit<\/span>=5<\/span>
      \ndataDir<\/span>=\/<\/span>var\/<\/span>zookeeper
      \nclientPort<\/span>=2181<\/span>
      \nserver.1=192.168.0.18:2888<\/span>:3888<\/span>
      \nserver.2=192.168.0.15:2888<\/span>:3888<\/span>
      \nserver.3=192.168.0.16:2888<\/span>:3888<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

      create a file “myid<\/em>” and the file should have only the id of the zookeeper node.
      \nI use 1, 2 and 3<\/p>\n

      Run this under the zookeeper\u2019s root folder to start the Zookeeper service.<\/p>\n

      1
      <\/div><\/td>
      .\/<\/span>bin\/<\/span>zkServer.sh start ><\/span> \/<\/span>dev\/<\/span>null 2<\/span>>&<\/span>1<\/span> &<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

      2) Kafka<\/b><\/p>\n

      I downloaded the most recent stable version
      \nModify the “config\/server.properties<\/em>” with:<\/p>\n

      1
      2
      3
      <\/div><\/td>
      broker.id=1<\/span>
      \nport<\/span>=9092<\/span>
      \nhost.name=192.168.0.10 zookeeper.connect=192.168.0.18:2181<\/span>,192.168.0.15:2181<\/span>,192.168.0.16:2181<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

      Boker.id should be 1,2 and 3 for each PI
      \nHost.name is the machine IP address
      \nzookeeper.connect should be equals in all<\/p>\n

      (Depends on each Raspberry PI version you are using)<\/p>\n

      Update the “bin\/kafka-server-start.sh<\/em>” with:<\/p>\n

      1
      2
      <\/div><\/td>
      export<\/span> JMX_PORT<\/span>=${JMX_PORT:-9999}<\/span>
      \nexport<\/span> KAFKA_HEAP_OPTS<\/span>="-Xmx256M -Xms128M"<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

      Otherwise, JVM would complain not able to allocate the specified memory.<\/p>\n

      Update “bin\/kafka-run-class.sh<\/em>” with:<\/p>\n

      1
      <\/div><\/td>
      KAFKA_JVM_PERFORMANCE_OPTS<\/span>="-client -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true"<\/span> # change -server to -client<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

      You can run this to start Kafka<\/p>\n

      1
      <\/div><\/td>
      \/<\/span>opt\/<\/span>kafka\/<\/span>bin\/<\/span>kafka-server-start.sh -daemon<\/span> \/<\/span>opt\/<\/span>kafka\/<\/span>config\/<\/span>server.properties ><\/span> \/<\/span>dev\/<\/span>null 2<\/span>>&<\/span>1<\/span> &<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

      And that’s it! Now you can do some basic tests.<\/p>\n

      Create a topic<\/p>\n

      1
      <\/div><\/td>
      \/<\/span>opt\/<\/span>kafka\/<\/span>bin\/<\/span>kafka-topics.sh --create<\/span> --zookeeper<\/span> 192.168.0.15:9092<\/span>,192.168.0.16:9092<\/span>,192.168.0.18:9092<\/span>  --replication-factor<\/span> 1<\/span> --partitions<\/span> 1<\/span> --topic<\/span> test<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

      Describe this topic<\/p>\n

      1
      <\/div><\/td>
      \/<\/span>opt\/<\/span>kafka\/<\/span>bin\/<\/span>kafka-topics.sh --zookeeper<\/span> localhost:2181<\/span> --describe<\/span> --topic<\/span> test<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

      Star a terminal producer<\/p>\n

      1
      <\/div><\/td>
      \/<\/span>opt\/<\/span>kafka\/<\/span>bin\/<\/span>kafka-console-producer.sh --broker-list<\/span> localhost:9092<\/span> --topic<\/span> test<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

      Start a terminal consumer<\/p>\n

      1
      <\/div><\/td>
      \/<\/span>opt\/<\/span>kafka\/<\/span>bin\/<\/span>kafka-console-consumer.sh  --bootstrap-server<\/span> localhost:9092<\/span> --topic<\/span> test<\/span> --from-beginning<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

      I put my Zookeeper and Kafka in the “\/opt<\/em>” folder, but you can put it in any path.<\/p>\n