Apache Kafka

Apache Kafka is a wildly popular Unified Event Log.
It adopts a client-server architecture:

  • The server is called a message broker
  • Clients are of two types:
    1. Event Producers, and,
    2. Event Consumers

Kafka Connect ties the producers and the consumers to the broker, helping relay streams inward and outward.

Processing an event stream to create a new data stream, involves the continuous application of transformation operations such as filtering, projection, joining, and aggregation. Doing this in application code (Java) can be relatively inefficient due to cross-platform latencies (roundtripping between the message broker and Java backend server).

Native stream processing libraries that run directly on the streaming platform, while supporting high-level abstractions for expressing streaming operations, were created to serve this need.
Kafka Streams is a Kafka-native event streaming library for Java applications.

CNS/PNS: a biological corollary for Event streaming platforms

The human body has a central nervous system - CNS (the key nerve bundles of brain and spinal cord that process nerve signals), and a peripheral nervous system - PNS (all other nerves forming the outer network) that branches out from the brain and spinal cord and extends to the limbs, skin, and internal organs.

The PNS is a two-way carrier:

  1. It carries sensory information to the CNS, which then decides how the body should react.
  2. It delivers these commands from the CNS to the muscles and organs

It consists of a Somatic Division that controls voluntary actions (moving arms and legs) and an Autonomic Division (ANS) to control involuntary actions (heart beating, lungs breathing, and stomach digesting). The hypothalamus and brainstem of the brain control the ANS, and it’s two opposing divisions:

  • Sympathetic (Fight or Flight): Increase heart rate, dilate airways, and slow digestion during stress or danger.
  • Parasympathetic (Rest and Digest): Lower heart rate, constrict airways, and stimulate digestion during recovery.

Think of a neural data pipeline carrying event information to the brain.
Kafka Producers would be what would carry these events (generated by mobile devices, online activity, etc.)
to the CNS (the Kafka Broker serving as the persistent event store, supporting ordered replay, with Kafka Streams as the processing engine).
Then Kafka Consumers would carry the brain’s commands to the organs and they would attune their output accordingly.

graph TD
    subgraph Clients
        P[Event Producers]
        C[Event Consumers]
    end

    subgraph Kafka Ecosystem
        B[Kafka Message Broker <br> Server]
        KS[Kafka Streams Library <br> High-level Abstractions]
        KC[Kafka Connect <br> Import/Export]
    end

    Source[Outside World / Clickstreams] -->|Import via KC| B
    P -->|Produce Events| B
    B -->|Consume Streams| KS
    KS -->|Transform: Filter/Join/Aggregate| KS
    KS -->|Produce Streams| B
    B -->|Consume Events| C
    B -->|Export via KC| Destination[Outside World]

    style B fill:#33f,stroke:#333,stroke-width:2px
    style KS fill:#99c,stroke:#333,stroke-width:2px