![\"\"](\"http:\/\/www.igfasouza.com\/blog\/wp-content\/uploads\/2020\/04\/kafka-connect-image.png\")
<\/p>\n<\/p>\n
What\u2019s the story Rory?<\/b><\/p>\n
This blog post is part of my series of posts regarding “Kafka Connect Overview<\/a>“. Table of contents<\/b><\/p>\n 1. Kafka Connect Kafka Connects goal of copying data between systems has been tackled by a variety of frameworks, many of them still actively developed and maintained. This section explains the motivation behind Kafka Connect, where it fits in the design space, and its unique features and design decisions<\/p>\n Kafka Connect has three major models in its design:<\/p>\n The connector model addresses three key user requirements. First, Kafka Connect performs broad copying by default by having users define jobs at the level of Connectors which then break the job into smaller Tasks. This two level scheme strongly encourages connectors to use configurations that encourage copying broad swaths of data since they should have enough inputs to break the job into smaller tasks. It also provides one point of parallelism by requiring Connectors to immediately consider how their job can be broken down into subtasks, and select an appropriate granularity to do so. Finally, by specializing source and sink interfaces, Kafka Connect provides an accessible connector API that makes it very easy to implement connectors for a variety of systems.<\/p>\n The worker model allows Kafka Connect to scale to the application. It can run scaled down to a single worker process that also acts as its own coordinator, or in clustered mode where connectors and tasks are dynamically scheduled on workers. However, it assumes very little about the process management of the workers, so it can easily run on a variety of cluster managers or using traditional service supervision. This architecture allows scaling up and down, but Kafka Connect\u2019s implementation also adds utilities to support both modes well. The REST interface for managing and monitoring jobs makes it easy to run Kafka Connect as an organization-wide service that runs jobs for many users. Command line utilities specialized for ad hoc jobs make it easy to get up and running in a development environment, for testing, or in production environments where an agent-based approach is required.<\/p>\n The data model addresses the remaining requirements. Many of the benefits come from coupling tightly with Kafka. Kafka serves as a natural buffer for both streaming and batch systems, removing much of the burden of managing data and ensuring delivery from connector developers. Additionally, by always requiring Kafka as one of the endpoints, the larger data pipeline can leverage the many tools that integrate well with Kafka. This allows Kafka Connect to focus only on copying data because a variety of stream processing tools are available to further process the data, which keeps Kafka Connect simple, both conceptually and in its implementation. This differs greatly from other systems where ETL must occur before hitting a sink. In contrast, Kafka Connect can bookend an ETL process, leaving any transformation to tools specifically designed for that purpose. Finally, Kafka includes partitions in its core abstraction, providing another point of parallelism<\/p>\n
\nIf you’re not familiar with Kafka, I suggest you have a look at my previous post “What is Kafka?<\/a>” before.
\nThis post is a collection of links, videos, tutorials, blogs and books that I found mixed with my opinion. <\/p>\n
\n2. Source & Sink Connectors
\n3. Standalone & Distributed
\n4. Converters & Transforms
\n5. Life cycle
\n6. Code
\n7. Books
\n8. Link<\/p>\n1. Kafka Connect<\/h2>\n
\n
![\"\"](\"http:\/\/www.igfasouza.com\/blog\/wp-content\/uploads\/2020\/04\/workers.jpg\")
<\/p>\n