This blog is devoted to the community Nerd or Geek, for those who like IT and coffee, and containing random thoughts and opinions on things that interest me.
If you’re not familiar with Kafka, I suggest you have a look at my previous post What is Kafka? before, and you can have a look at how I created the Kafka cluster here.
Idea
I found the idea of an OCI dashboard with smashing dashing in a colleague’s blog some time ago and I decided to pivot a little bit in a simple Raspberry Pi Kafka example where I can get OCI data and combine it with the smashing dashing dashboard idea.
For the producer part, you can use my Micronaut Sense hat example as a start point and just change to get the OCI data. You can get the full Micronaut Kafka Producer code on my GitHub.
For the consumer part, you can use my Quarkus example. You can get the full Quarkusl Kafka Consumer code on my GitHub.
clone the code and change the update.py file to read the value from the REST interface that Quarkus generates.
Results
Because this is a kind of Oracle Cloud solution you can use Oracle Streaming Service instead of hosting your Kafka.
Oracle Streaming Service (OSS) is a real-time, serverless, Apache Kafka-compatible event streaming platform for developers and data scientists, it provides a fully managed, scalable, and durable solution for ingesting and consuming high-volume data streams in real-time. To learn more about Oracle Streaming Service, see the documentation overview.
You can check my GitHub with some OSS examples and check the Micronaut one.
You just need to change the “application.yml” file from my Micronaut sense hat idea.
The Kafka consumer and the dashboard can be hosted in a cloud VM as well.
OCI Logging service is a highly scalable and fully managed single pane of glass for all the logs in your tenancy. Logging provides access to logs from Oracle Cloud Infrastructure resources. These logs include critical diagnostic information that describes how resources are performing and being accessed. To learn more about the Logging service, see the documentation overview.
OCI Service Connector Hub moves data, such as logs from Logging, to services, such as Object Storage, Streaming, and Monitoring. It triggers functions for custom data processing and sends notifications about changes to cloud resources. To learn more about the Service Connector Hub, see the documentation overview.
Here using Service Connector Hub I can get the logs from my tenancy and send them to OSS in a simple way.
This is another blog about Raspberry PI, and today I want to show how I did a simple Kafka cluster demo using the Inky pHAT. It’s kind of a continuation from my previous blog, Kafka at the edge with Raspberry PI.
If you’re not familiar with Kafka, I suggest you have a look at my previous post What is Kafka? before, and you can have a look at how I created the Kafka cluster here.
The Inky pHAT is an add-on board for Raspberry Pi, that has a low-energy, high-falutin, electronic paper (ePaper / eInk / EPD) display for your Pi, in three different color schemes: red/black/white, yellow/black/white, and black/white.
You can learn more about Inky pHAT here and check the API here.
Real-Time Locating System (RTLS) enables identifying and tracking the location of objects or people in real-time. It is used everywhere in transportation and logistics across industries.
Use Cases
Real-time alerting on a single event: Monitor assets and people and send an alert to a controller, mobile app, or any other interface if an issue happens.
Continuous real-time aggregation of multiple events: Correlation data while it is in motion. Calculate average, enforce business rules, apply an analytic model for predictions on new events, or any other business logic.
Batch analytics on all historical events: Take all historical data to find insights, e.g., for analyzing issues of the past, planning future location requirements, or training analytic models.
This is not an exhaustive list.
A postmodern RTLS requires an open architecture and high scalability and of course, the implementations can rely on Kafka.
Idea
A simple Raspberry Pi Kafka example where I can use one node to get the open Dublin Bus data and display it in real-time on the other node using Inky phat.
For the producer part, you can use my Micronaut Sense hat example as a start point and just change to use Dublin Bus data instead. You can get the full Micronaut Kafka Producer code on my GitHub.
Kafka can be deployed as a single broker in a vehicle and a global Kafka infrastructure can spread to multiple cloud providers, regions, countries, or even continents and integrate with tens or hundreds of factories or other edge locations.
Curiosity
1- Dublin bus is already using an e-ink display to show bus stop data.
2- I did a demo for real-time data from connected vehicles some time ago.
3- I won a prize at Europe’s First government-funded Blockchain Hackathon with the idea of an app to track medical devices combining Kafka and BlockChain.
This is another blog about Raspberry PI, and today I want to show how I did a simple Kafka cluster demo using Sense Hat & GFX Hat.
If you’re not familiar with Kafka, I suggest you have a look at my previous post What is Kafka? before, and you can have a look at how I created the Kafka cluster here.
The Sense HAT is an add-on board for Raspberry Pi, tha has an 8×8 RGB LED matrix, a five-button joystick and includes the following sensors: Gyroscope, Accelerometer, Magnetometer, Barometer, Temperature sensor and Relative Humidity sensor.
You can learn more about Sense hat in my previous blog.
The GFX HAT is an add-on board for Raspberry Pi, tha has a 128×64 pixel, 2.15″ LCD display with snazzy six-zone RGB backlight and six capacitive touch buttons. GFX HAT makes an ideal display and interface for your headless Pi projects.
You can learn more about GFX hat here and check the API here.
The idea here is to focus on scenarios where the Kafka clients and the Kafka brokers are running on the edge. This enables edge processing, integration, decoupling, low latency, and cost-efficient data processing.
Edge Kafka is not simply yet another IoT project using Kafka in a remote location. Edge Kafka is an essential component of a streaming nervous system that spans IoT (or OT in Industrial IoT) and non-IoT (traditional data-center / cloud infrastructures).
Multi-cluster and cross-data center deployments of Apache Kafka have become the norm rather than an exception. A Kafka deployment at the edge can be an independent project. However, in most cases, Kafka at the edge is part of an overall Kafka architecture.
Apache Kafka is the New Black at the Edge in Industrial IoT, Logistics, and Retailing
A Raspberry Pi 2 nodes Kafka cluster, with a Micronaut Kafka producer that gets sense hat data and a Quarkus Kafka consumer that puts the result in a REST that GFX Hat reads using python API.
The Micronaut producer gets the Sense Hat humidity, pressure, and temperature values and sends it to a Kafka topic.
The Quarkus consumer reads a Kafka topic and generates a REST interface with the last topic value. I used the GFX Hat to display the result.
You can get the full Micronaut Kafka Producer code on my GitHub.
You can get the full Quarkusl Kafka Consumer code on my GitHub.
Kafka is a great solution for the edge. It enables deploying the same open, scalable, and reliable technology at the edge, data center, and the cloud. This is relevant across industries. Kafka is used in more and more places where nobody has seen it before. Edge sites include retail stores, restaurants, cell towers, trains, and many others.
This is another blog about Raspberry PI, and today I want to show how I did a simple Kafka cluster.
If you’re not familiar with Kafka, I suggest you have a look at my previous post “What is Kafka?” before.
This is a really simple tutorial and you can find similar instructions over the internet.
Pre-Requisites
Raspbian installed
Components
3x Raspberry Pi3 B with power supply and micro SD card with Raspbian.
Despite every time that you look at a cluster tutorial they add a switch, router, and some internet cables I did use just simple wifi.
For the software part, we will download Zookeeper and Kafka, and there’s no installation. Is just about untar and changing some settings files.
Download latest Apache Zookeeper & Apache Kafka.
I put my Zookeeper and Kafka in the “/opt” folder, but you can put it in any path.
Final Thoughts
I did use a Raspberry PI 3 B, but Raspberry PI already released The Raspberry Pi4 with 8g.
I don’t think we are too far from seeing a new Raspberry version, make Pi4 16gb or PI5, who knows.
I did use the Raspbian that is 32bits OS, and Raspberry PI already released Raspberry Pi OS, (yes they changed the name), that is the new official operating system and there’s a 64bits version.
With a more powerful computer, memory, and a 64bits OS it’s open to a lot of more ideas and scenarios. Kubernetes, Strimzi, and keda are just some initial things.
Another thing is that Kafka is about to remove the Apache Zookeeper dependency, and this will make a lot of changes.
I found a lot of tutorials about Kubernetes and 64bits OS for Raspberry, but could not find any Kubernetes Kafka example or Strimzi. I think with this new OS and more powerful hardware is just a question of time.
MicroK8s – is a powerful, lightweight, reliable production-ready Kubernetes distribution. It is an enterprise-grade Kubernetes distribution that has a small disk and memory footprint while offering carefully selected add-ons out-the-box, such as Istio, Knative, Grafana, Cilium, and more.
Canonical has released Ubuntu 19.10 with a focus on accelerating developer productivity in AI/ML, new edge capabilities for MicroK8s, and delivering the fastest GNOME desktop performance.
The Raspberry Pi 4 Model B is supported by Ubuntu 19.10. The latest board from the Raspberry Pi Foundation offers a faster system-on-a-chip with a processor that uses the Cortex-A72 architecture (quad-core 64-bit ARMv8 at 1.5GHz) and offers up to 8GB of RAM. With the Raspberry Pi 4 Model B, developers get access to a low-cost board, powerful enough to orchestrate workloads at the edge with MicroK8s.
K3s – a flavor of Kubernetes that is highly optimized for the edge. Though K3s is a simplified, miniature version of Kubernetes, it doesn’t compromise the API conformance and functionality.
The primary role of a Kafka producer is to take producer properties & record as inputs and write it to an appropriate Kafka broker. Producers serialize, partitions, compresses and load balances data across brokers based on partitions.
In order to send the producer record to an appropriate broker, the producer first establishes a connection to one of the bootstrap servers. The bootstrap-server returns list of all the brokers available in the clusters and all the metadata details like topics, partitions, replication factor and so on. Based on the list of brokers and metadata details the producer identifies the leader broker that hosts the leader partition of the producer record and writes to the broker.
Important Producer Settings:
Acks
The acks setting specifies acknowledgements that the producer requires the leader to receive before considering a request complete. This setting defines the durability level for the producer.
Max.in.flight.requests.per.connection
The maximum number of unacknowledged requests the client will send on a single connection before blocking. If this setting is greater than 1, pipelining is used when the producer sends the grouped batch to the broker. This improves throughput, but if there are failed sends there is a risk of out-of-order delivery due to retries (if retries are enabled). Note also that excessive pipelining reduces throughput.
Compression.type
Compression is an important part of a producer’s work, and the speed of different compression types differs a lot.
To specify compression type, use the compression.type property. It accepts standard compression codecs (‘gzip’, ‘snappy’, ‘lz4’), as well as ‘uncompressed’ (the default, equivalent to no compression), and ‘producer’ (uses the compression codec set by the producer).
Compression is handled by the user thread. If compression is slow it can help to add more threads. In addition, batching efficiency impacts the compression ratio: more batching leads to more efficient compression.
Batch.size
Larger batches typically have better compression ratios and higher throughput, but they have higher latency.
Linger.ms
There is no simple guideline for setting linger.ms values; you should test settings on specific use cases. For small events (100 bytes or less), this setting does not appear to have much impact.
//prepare the record String recordValue ="Current time is "+ Instant.now().toString(); System.out.println("Sending message: "+ recordValue);
ProducerRecord<String, String> record =new ProducerRecord<>("java_topic", null, recordValue);
//produce the record
producer.send(record);
producer.flush();
//close the producer at the end
producer.close(); } }
High-level overview of Kafka producer components – Kafka the Definitive Guide Book
2. Consumer
The primary role of a Kafka consumer is to read data from an appropriate Kafka broker. In order to understand how to read data from Kafka, you first need to understand its consumers and consumer groups.
A consumer group is a group of consumers that share the same group id. When a topic is consumed by consumers in the same group, every record will be delivered to only one consumer. As the official documentation states: “If all the consumer instances have the same consumer group, then the records will effectively be load-balanced over the consumer instances.”
This way you can ensure parallel processing of records from a topic and be sure that your consumers won’t be stepping on each other toes.
Each topic consists of one or more partitions. When a new consumer is started it will join a consumer group and Kafka will then ensure that each partition is consumed by only one consumer from that group.
Kafka the Definitive Guide Book
Kafka the Definitive Guide Book
You can have many consumers reading the same records from the topic, as long as they all have different group ids.
Kafka the Definitive Guide Book
Consumer groups provide the following advantages:
Each instance receives messages from one or more partitions (which are “automatically” assigned to it), and the same messages won’t be received by the other instances (assigned to different partitions). In this way, we can scale the number of the instances up to the number of the partitions (having one instance reading only one partition). In this case, a new instance joining the group is in an idle state without being assigned to any partition.
Having instances as part of different consumer groups means providing a publish/subscribe pattern in which the messages from partitions are sent to all the instances across the different groups. Inside the same consumer group, the rules are as shown in the secondi image, but across different groups, the instances receive the same messages (as shown in the third image). This is useful when the messages inside a partition are of interest for different applications that will process them in different ways. We want all the interested applications to receive all the same messages from the partition.
Another advantage of consumer groups is the rebalancing feature. When an instance joins a group, if enough partitions are available (that is, the limit of one instance per partition hasn’t been reached), a rebalancing starts. The partitions are reassigned to the current instances, plus the new one. In the same way, if an instance leaves a group, the partitions are reassigned to the remaining instances.
This is not an exhaustive list, so if you know someone that you think should be here, please post a comment with. And of course, some here are Kafka related.
Starting with the basics, I add the official ones;
Feel free to send anyone that you think should be here.
I couldn’t find anyone for IBM and Google clouds.
I couldn’t find anyone for Python Faust as well.
This blog post is part of my series of posts regarding Kafka Connect.
If you’re not familiar with Kafka, I suggest you have a look at some of my previous post;
This post is a collection of links, videos, tutorials, blogs and books that I found mixed with my opinion.
TL;DR
This post is about code, I’ll show all steps to develop your Sink Kafka Connector.
Table of contents
1. Use Case
2. Code
3. Links
1. Use Case
One scenario that has become popular and I came across with some questions around in the last months is to use Kafka to trigger functions as service.
In simple words: Send a message in your kafka topic and your function(s) gets invoked with the payload which you sent to Kafka, you can now have a function trigger in response to messages in Kafka Topics.
My example is using OCI functions.
2. Code
The connector needs extends SinkConnector. Is almost the same from source and you need to implement six methods here.
@Override publicvoid put(Collection<SinkRecord> records){ // here is where the magic happenings. // just add a boolean triggerFn method that trigger the function. }
@Override publicvoid stop(){ // TODO Auto-generated method stub
This blog post is part of my series of posts regarding Kafka Connect.
If you’re not familiar with Kafka, I suggest you have a look at some of my previous post;
This post is a collection of links, videos, tutorials, blogs and books that I found mixed with my opinion.
TL;DR
This post is about code, I’ll show all steps to develop your own Source Kafka Connector, and in the end, I’ll show an easy way to deploy your code. Openweathermap source kafka Connector.
Table of contents
1. Get Started
2. Use Case
3. Shema
4. AbstractConfig
5. Connector
6. Task
7. Code
8. SMTs
9. Converter
10. Transformation
11. Deploy
12. Links
1. Get Started
Kafka Connect aims to reduce the burden of connecting Kafka with external systems such as databases, key-value stores, search indexes, and even file systems. Connectors are the high-level abstractions that manage the streaming of data from and to Kafka. Each connector speaks the language of the external system it connects to, hiding this complexity from the application developer who can focus on more valuable business logic.
The best place to start when implementing your Source Connector is the Confluent Connector Development Guide.
OpenWeatherMap real-time weather data website. This API is already used in a lot of examples around Development. I already used some time for IoT and Data analyse demos.
They have a free API tier, which is limited to 1 request per second, which is quite enough for tracking changes in weather. Their different API schemas return plenty of numeric and textual data, all interesting for analysis. This is a perfect example for us. I tried to choose an API that contains a more complex Json, as a result, to show more in a didactic way the Shema part.
A partition is a division of source records that usually depends on the source medium.
In our case, there is only one API URL and we are only ever requesting current data, so a very good partition would be query weather data for different cities and each partition would be processed by a separate task.
3. Schema
The Schema interface is one of the most important components of Kafka Connect. This allows you to define what the data looks like without having to worry about the way the data is stored.
Here if you have a look in the json you are going to see that the json has some structures inside, for example, you can see weather that contains, id, main, descriptions and so on.
AbstractConfig is an abstract class that defines an interface that our WeatherAPIConfig
needs to adhere to, and here is where we get the configuration for our connector.
We need to pass the openWeatherApiKey, a list of cities, that we want to get the weather data and the kafka topic. I add pollFrequency as a hardcoded value because the API has a limitation of the number of requests in the free tier.
@Override publicString version(){ // you can add your own version strategy returnnull; }
@Override publicvoid start(Map<String, String> props){ // here is were we implement the configuration class }
@Override public Class<?extends Task> taskClass(){ // just need pass the name of the class returnnull; }
@Override public List<Map<String, String>> taskConfigs(int maxTasks){ // here is a function partition strategy to split the list o cities to a different task returnnull; }
@Override publicvoid stop(){ // since we don't start anything on Star method we don't need do anything here }
@Override public ConfigDef config(){ // here we already defined in the weatherAPI class returnnull; }
6. Task
In a similar manner to the Source Connector class, we implement the Source Task abstract class. Here we have only 4 methods.
@Override publicString version(){ // you can add your own version strategy returnnull; }
@Override publicvoid start(Map<String, String> props){ // initialize the configuration and the weather API client }
@Override public List<SourceRecord> poll()throwsInterruptedException{ // this method is going to be called continuous and because of the API limitation, we need to add a sleep to decrease the poll frequency. returnnull; }
@Override publicvoid stop(){ // TODO Auto-generated method stub }
7. Code
I create one package model where I add my DTO to the fields that I used in my schema and package shema.
WeatherAPIClient this class just calls the API and passes the list of cities.
I use the WeatherAPIConfig config to get all configurations;
From an architectural point of view, SMT resides exactly between the connectors and the converters and they apply their data transformation just before storing the data into Kafka and right before extracting the data from it.
The Kafka Connect frameworks come with many built-in SMT to address the most common transformations use cases like for example:
mask sensitive message fields,
add identifiers,
tag events,
cast data types to comply to the destination,
convert data (e.g. timestamp conversion like time-based data conversion standardization),
As you can see SMT can be combined and chained pretty easily by simply configuring them next to the Kafka Connect sinks and sources.
But the most important thing that should always be kept in mind is the “one message at a time”.
This is particularly crucial because it strongly limits SMT’s usage. In particular, it doesn’t permit any kind of transformation that is characterized by any kind of aggregation logic. Processing one message at a time is mandatory due to how partitions work and how the offset semantic is implemented in Kafka.
9. Converter
Here I create a package converter where I add an example of a converter, Serialize, and Deserializer. Here we have only three methods to implement.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
@Override publicvoid configure(Map<String, ?> configs, boolean isKey){ // get the properties }
@Override publicbyte[] fromConnectData(String topic, Schema schema, Object value){ // convert to array of bytes returnnull; }
@Override public SchemaAndValue toConnectData(String topic, byte[] value){ // convert to schema value returnnull; }
GitHub
You can check the full code in my Github, and I did a Scala demo as well, where I just follow the same structure of folders, packages and classes.
This blog post is part of my series of posts regarding Kafka Connect.
If you’re not familiar with Kafka, I suggest you have a look at some of my previous post;
Standalone mode is the best way to get started with minimum config and infrastructure setup. In this section, we will see how to configure a connector in Standalone mode and transfer file content to Kafka topic.
In the same directory where we have created the yaml file execute the following command to start Kafka cluster. When the below command runs for the very first time it downloads the image. Once the image is downloaded it creates a Kafka cluster.
1
docker-compose up
For starting any Kafka connect cluster we require – workers config and connector (file-stream) config.
Create two files: workers-config.properties and file-stream-connector-properties.
Workers-config.properties:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
bootstrap.servers=127.0.0.1:9092
key.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=false # we always leave the internal key to JsonConverter
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter.schemas.enable=false # Rest API
rest.port=8086
rest.host.name=127.0.0.1 # this config is only for standalone workers
offset.storage.file.filename=standalone.offsets
offset.flush.interval.ms=10000
File-stream-connector-properties:
1 2 3 4 5 6 7
# These are standard kafka connect parameters, need for ALL connectors
name=file-stream-kafka-connect-standalone
connector.class=org.apache.kafka.connect.file.FileStreamSourceConnector
tasks.max=1 file/FileStreamSourceConnector.java file=source-input.txt
topic=kafka-connect-standalone
You should see three files in the folder; docker-compose.yml, file-stream-connector-properties and workers-config.properties
Create an input file source-input.txt, the content of the file is transferred to Kafka topic.
1
touch source-input.txt
Mount a host directory in a docker container: Make sure we are in the directory where we have created the files. After mount, we automatically switch to cp-kafka-connect.
1
docker run --rm -it -v "$(pwd)":/kafka-connect/ --net=host confluentinc/cp-kafka-connect
This docker image expect the connector in this folder;
/etc/kafka-connect/jars
Create Kafka topic “kafka-connect-standalone” with 3 partitions and replication factor 1.
Now you can test: Open file source-input.txt and type some message to it & save it. The message should have been transferred to Kafka topic.
What happened in the background: We wrote data to the source file and Kafka connect standalone pushed the data to topic. No programming, just configs.
Now stop Kafka Connect (press Ctrl/Command + C). Once Kafka connect gracefully stopped, list all files in the given directory. We have a new guest in the form of a file “standalone.offsets”.
This file is created by Kafka connect to keep track of from where it should resume reading messages from the source file on re-starts.
When Kafka connect starts again it should resume reading without publishing duplicate messages to the topic.
Try to execute the above command (connect-standalone workers-config.properties file-stream-connector-properties) again and validate if you do not have a duplicate message.
2. Source Distributed mode
Now let’s configure a connector in distributed mode.
Create Kafka topic “kafka-connect-distibuted” with 3 partitions and replication factor 1.
curl -d @<path-to-config-file>/connect-source.json -H "Content-Type: application/json" -X POST http://localhost:8083/connectors
Now write some message in the source file and once the file is saved, all messages are posted in topic “kafka-connect-distributed”.
check for the message copied from file to topic by FileSourceConnector. Note that messages are stored in JSON format as the connector topic created earlier with config value.converter.schemas.enable=true.
Strimzi simplifies the process of running Apache Kafka in a Kubernetes cluster. It provides container images and Operators for running Kafka on Kubernetes.
This blog post is part of my series of posts regarding “Kafka Connect Overview“.
If you’re not familiar with Kafka, I suggest you have a look at my previous post “What is Kafka?” before.
This post is a collection of links, videos, tutorials, blogs and books that I found mixed with my opinion.
Kafka Connects goal of copying data between systems has been tackled by a variety of frameworks, many of them still actively developed and maintained. This section explains the motivation behind Kafka Connect, where it fits in the design space, and its unique features and design decisions
Kafka Connect has three major models in its design:
Connector model
Worker model
Data model
The connector model addresses three key user requirements. First, Kafka Connect performs broad copying by default by having users define jobs at the level of Connectors which then break the job into smaller Tasks. This two level scheme strongly encourages connectors to use configurations that encourage copying broad swaths of data since they should have enough inputs to break the job into smaller tasks. It also provides one point of parallelism by requiring Connectors to immediately consider how their job can be broken down into subtasks, and select an appropriate granularity to do so. Finally, by specializing source and sink interfaces, Kafka Connect provides an accessible connector API that makes it very easy to implement connectors for a variety of systems.
The worker model allows Kafka Connect to scale to the application. It can run scaled down to a single worker process that also acts as its own coordinator, or in clustered mode where connectors and tasks are dynamically scheduled on workers. However, it assumes very little about the process management of the workers, so it can easily run on a variety of cluster managers or using traditional service supervision. This architecture allows scaling up and down, but Kafka Connect’s implementation also adds utilities to support both modes well. The REST interface for managing and monitoring jobs makes it easy to run Kafka Connect as an organization-wide service that runs jobs for many users. Command line utilities specialized for ad hoc jobs make it easy to get up and running in a development environment, for testing, or in production environments where an agent-based approach is required.
The data model addresses the remaining requirements. Many of the benefits come from coupling tightly with Kafka. Kafka serves as a natural buffer for both streaming and batch systems, removing much of the burden of managing data and ensuring delivery from connector developers. Additionally, by always requiring Kafka as one of the endpoints, the larger data pipeline can leverage the many tools that integrate well with Kafka. This allows Kafka Connect to focus only on copying data because a variety of stream processing tools are available to further process the data, which keeps Kafka Connect simple, both conceptually and in its implementation. This differs greatly from other systems where ETL must occur before hitting a sink. In contrast, Kafka Connect can bookend an ETL process, leaving any transformation to tools specifically designed for that purpose. Finally, Kafka includes partitions in its core abstraction, providing another point of parallelism
In simples words
Kafka Connect is a distributed, scale, fault-tolerant service designed to reliably stream data between Kafka and other data systems. Data is produced from a source and consumed to a sink.
Connect tracks the offset that was last consumed for a source, to restart task at the correct starting point after a failure. These offsets are different from Kafka offsets, they are based on the sourse system like database, file, etc.
In standalone mode, the source offset is tracked in a local file and in a distributed mode, the source offset is tracked in a Kafka topic.
2. Source & Sink connectors
Producers and Consumers provide complete flexibility to send any data to Kafka or process in any way. This flexibility means you do everything yourself.
Kafka Connect’s simple frameworks allows;
developers to create connectors that copy data to or from others systems.
Operators to use said connectors just by writing configuration files and submitting them to Connect. (No code)
Community and 3rd-party engineers to build reliable plugins for common data sources and sinks.
Deployments to deliver fault-tolerant and automated load balance out-of-the-box.
And the frameworks do the hard work;
Serialization and deserialization.
Schema registry integration.
Fault-tolerant and failover.
Partitioning and scale-out.
And let the developers focus on domain specific details.
3. Standalone & Distributed
In standalone mode we have a source or a Sink and a Kafka broker, when we deploy Kafka connect in the standalone we actually need to pass a configuration file containing all the connection properties that we need to run. So standalone’s main way of providing configuration to our connector is by using properties files and not the Rest API.
In the distributed mode we usually have more than one worker, since these workers can be in different machines or containers they can not share the same storage space, so a properties file is out of the question. Instead kafka connect in distributed mode leverage kafka topics in order to sink between themselves.
4. Converters & Transforms
Pluggable API to convert data between native formats and Kafka. Just like the name says. Converters are used to come for data from a format to another.
In the Source connectors converters are invoked after the data has been fetched from the source and before it is published to kafka.
In the Sink connectors converters are invoked after the data has been consumed from Kafka and before it is stored in the sink.
Apache Kafka ships with Json Converter.
Transform is a simple operation that can be applied in the message level.
There’s a nice blog post about Single message transforms here.
Single message transforms SMT modify events before storing in Kafka, mask sensitive information, add identifiers, tag events, remove unnecessary columns and more, modify events going out of Kafka, route high priority events to faster datastore, cast data types to match destination and more.
5. Life cycle
I build this animation using the slides from (Randall Hauch, Confluent) Kafka Summit SF 2018 here.
Sequence diagram
6. Code
Confluent makes available a code example that you can found here.
