Event Driven Architecture (EDA)

EDA adopts an ‘event-streaming’ paradigm that accords centrality to events.

An event stream is an undending flow of events (a stream of data or events) captured in real time,
where an event is anything of note that happens, such as an offline purchase, online activity, or a sensor reading.

Systems designed around event streams might consume data from multiple event streams.
Event streaming involves the continuous application of transformation operations (filtering, projection, joining, aggregation) on the stream, to produce a new stream of data. This makes events (streams) the primary input and output of the application.

Technologically EDA is realised with the help of a Unified Event Log.
Also known as an ‘event streaming platform’, this is a software component that can consume, process, store and produce event streams.
Such a data streaming platform provides three critical functions around events:

  1. moving events to/from the platform
  2. storing or persisting events
  3. processing an event-stream

The business case for EDA

Users interacting with applications create a steady stream of events.
In the digital economy, even offline events such as purchases, generate business events such as monetary transfers. Businesses need to handle event streams, or unending flows of events so they can meet customer expectations around responsiveness.

‘Clickstreams’ (a stream of user actions on an application) are often used for analysis.

While these events could be captured using an RDBMS, they need to be subject to analysis in order to extract business value.
EDA is an event architecture that supports this analysis while processing high-volume event streams with daily event counts in millions.

The problem with Message Queues

Event logs score over the previous standard in distributed systems – message queues (MQ) – in that the consumers do not need to be aware of the producer.

Message queue or more generically Message Oriented Middleware (MoM) infra is owned by each application that uses one.
So if system B wants to source data from system A, it (system B) needs to be aware of system A’s MQ infrastructure.
Scale that to multiple awareness for multiple data sources.

An event streaming platform being an org-wide shared platform, any consumer can access all the data, without requiring an awareness of each producer.

The event streaming platform –replacing the message queues – becomes a central nervous system. - storing, processing and generating event streams.


graph TD
    subgraph Sources [Application Silos]
        T[Online Ticketing]
        S[PoP Sales]
    end

    subgraph Storage [Data Stores]
        DB1[(Ticketing DB)]
        DB2[(Sales DB)]
    end

    subgraph Routing [Message Queues]
        MQ1[Queue A]
        MQ2[Queue B]
    end

    subgraph Subscriptions [Consumers]
        C1[Consumer Group 1]
        C2[Consumer Group 2]
    end

    %% Flow Connections
    T --> DB1
    S --> DB2
    
    DB1 --> MQ1
    DB2 --> MQ2

    %% Complex Point-to-Point Routing
    MQ1 --> C1
    MQ1 -.-> C2
    MQ2 --> C2
    MQ2 -.-> C1

    style Sources fill:#f5f5f5,stroke:#333
    style Storage fill:#e1f5fe,stroke:#0288d1
    style Routing fill:#fff9c4,stroke:#fbc02d
    style Subscriptions fill:#e8f5e9,stroke:#388e3c

What’s the big deal about events anyway?

Businesses derive value from EDA wherever event streams embody essential and actionable information. This includes time sensitive decisions (does a deviating sensor reading indicate machine failure?), transactional data for fraud, intrusion detection and trading, etc.

EDA is not the right fit if the entire data lies in a single database instance, or the volume of events is small enough to fit in a single database and not require distributed data stores.

Event Streaming vs Traditional Message Queues

Records (events) in a data streaming platform are stored durably – they are thought of as being business critical.

Whereas messages in traditional messaging systems are ephemeral – they are considered ‘tactical’ communication between two systems.

Event streaming platform components

Kafka brokers (server) for event storage

The Kafka broker handles the ‘store’ in the move, store, process trifecta of an event streaming platform. An event is conceptually composed of three parts:

i. Key (event identifier – used for routing and grouping events) ii. Value (the event payload itself – it could be a trigger for an action, or the result of an action) iii. Timestamp (when the event occurred)

Data is stored as key value pairs in byte format, serialized using Avro, Json Schema, or Protocol buffers.

Kafka brokers are deployed in a cluster – never standalone, always three or more servers. Each broker has a role (leader and follower).

Producers send record storage requests and consumers send record retrieval requests to the leader broker. Follower brokers fetch data from the leaders, which combined with sharding (replicating a portion of the data from another leader) supports resilience.

Schema Registry for data governance

Schema Registry stores schemas of event records, which provide the contract that binds record producers and consumers. The broker stores data in binary format (byte arrays). The producer places the object schema in the Schema Registry and uses it to serialize records to binary, and the consumer


graph LR
    subgraph Event Record Structure
        K[Key <br> Routing & Grouping]
        V[Value <br> Payload / Action Trigger]
        T[Timestamp <br> Occurrence Time]
    end
    
    K --- V --- T
    
    style Event Record Structure fill:#fafafa,stroke:#333

retrieves it from the Registry to deserialize the record from binary. Schema Registry also provides serializers and deserializers (Avro, JSON Schema and Protocol Buffers).

Producer and consumer clients

Producer clients send records into Kafka – this entails serializing records and sending the bytes to the broker. A producer client might be a microservice that captures a clickstream and produces (moves) records to the Kafka broker (store).

Consumer clients read records from Kafka – this entails consuming bytes from the broker and deserializing them back into objects. A consumer client be a Kafka Streams application that processes clickthrough on a campaign email and correlates it with user uptake of a new feature.

Kafka Connect

Kafka Connect provide an abstraction over (above) the foundational layer of producer and consume clients. It enables importing of data from external systems into a Kafka broker cluster using Kafka Connect source connectors (say from a document or relational database). It also enables the reverse pathway of exporting of data from a Kafka cluster into an external system using Kafka Connect sink connectors, while allowing lightweight data transformations with Simple Message transforms (SMTs).

Kafka Streams

A stream processing library written in Java, used by client applications at the periphery of – not within the Kafka broker cluster. A client streams application consumes records from a Kafka source topic and processes each record, and the stream terminal action writes records back to a Kafka sink topic.

ksqlDB

An event streaming database that uses Kafka Streams under the covers. It specifies streaming operations as SQL – which is its USP. That is, event stream processing via SQL.


graph LR
    subgraph External Systems
        DB[(Relational DB / Docs)]
        App[External App / Sink]
    end

    subgraph Kafka Cluster Periphery
        Prod[Producer Clients]
        KConnSrc[Kafka Connect Source]
        KConnSnk[Kafka Connect Sink]
        KStr[Kafka Streams / ksqlDB App]
        Cons[Consumer Clients]
    end

    subgraph Kafka Broker Cluster
        TopicSrc[Source Topic]
        TopicSnk[Sink Topic]
    end

    %% External Data Flow
    DB -->|Import| KConnSrc
    KConnSnk -->|Export| App

    %% Inside Cluster Pathways
    KConnSrc -->|Bytes| TopicSrc
    Prod -->|Serialize & Move| TopicSrc
    
    %% Processing and Consumption
    TopicSrc -->|Consume| KStr
    KStr -->|Transform| KStr
    KStr -->|Write Back| TopicSnk
    
    TopicSnk -->|Consume & Deserialize| Cons
    TopicSnk -->|Extract| KConnSnk

    style Kafka Broker Cluster fill:#e1f5fe,stroke:#0288d1,stroke-width:2px
    style External Systems fill:#f5f5f5,stroke:#333

Scenario: Applying Kafka

Let’s say a rural farmer, Dhanya Bhumi, applies for a farming subsidy through a Common Service Center (CSC) facilitated by the local self-governance machinery (Gram Sabha). Land ownership (7/12 extracts previously verified by a revenue official - Talati) is instantly verified via API-integrations with the state’s digital land records database (Mahabhulekh). The State Agriculture Department finalises the approved list of farmers in the state or central agrarian scheme Direct Benefit Portal (Krishi DBT). An officer digitally signs the Fund Transfer Order (FTO) carrying names, account numbers, amounts. API integration pushes the FTO into the Public Financial Management System (PFMS), managed by the Controller General of Accounts. PFMS is linked with RBI’s central banking system e-Kuber, which debits treasury funds from the government/department’s dedicated holding account. Thereafter the Aadhaar Payment Bridge System (APBS) via the National Payments Corporation of India (NPCI) or the NEFT clearing network (half-hourly batches with inter-bank netting) is used to transfer the direct benefit (subsidy) to the account holder.

Let’s look at how the Kafka event streaming platform might be used in this scenario.

Kafka Connect source connectors are used to pull new application records from the DBT portal. As data privacy is requirement, a simple transform (SMT) is applied to mask Aadhaar numbers and bank account details while ingesting the stream of application records into the Kafka unified event log. At this stage different applications begin to consume the ingested records.

The agrarian department uses a Kafka Streams application to read the subsidy request from the ‘subsidy-applied’ source topic, perform some processing and writes records to ‘notify-subsidies’ sink topic if the applicant is eligible for other subsidies. The department’s farmer outreach microservice consumes records from the sink topic and sends emails to the nearest CSC (Maha-e-Seva Kendra).

A messaging consumer client consumes the application submission record from the subsidy-applied source topic, to alert the identity owner via an SMS Gateway integration, to pre-empt identity theft:

“प्रिय धन्य भूमि, राज्य पुरस्कृत कृषक यंत्रीकरण योजना के लिए आपका आवेदन आपले सरकार सेवा केंद्र के माध्यम से प्राप्त हो गया है। यदि आपने इसके लिए आवेदन नहीं किया है, तो तुरंत 1800 120 8040 पर कॉल करें।”

“Dear Dhanya Bhumi, Your application for a State Agriculture Mechanisation Scheme has been received via CSC. If you did not apply for this, call 1800 120 8040 immediately.”

Similarly, when the payment order is generated, the messaging consumer client reads records from the ‘fto-signed’ topic, to inform the farmer of the amount approved, eliminating kickback demands:

“कृषि विभाग ने ₹1,500,000 की सब्सिडी के लिए आपके फंड ट्रांसफर ऑर्डर को डिजिटल रूप से मंजूरी दे दी है। PFMS के साथ प्रोसेसिंग चल रही है।”

“Your Fund Transfer Order for subsidy of Rs. 1,50,000 has been digitally approved by the Agriculture Department. Processing with PFMS.”

For its internal dashboard monitoring the state level agrarian economy, the DBT Portal has a Kafka Streams application that pulls records from the ‘fto-signed’ source topic. It processes subsidy data to build patterns of districts consuming subsidies most actively and sends the results to a ‘subsidy-disbursement-trends’ sink topic. A Kafka Connect sink connector pulls records from the sink topic and imports them into an Elastic stack dashboard.

graph TD
    subgraph Ingestion_Layer ["Ingestion Layer"]
        DBT[Krishi DBT Portal] -->|Kafka Connect Source <br> + SMT Masking| Topic1[Topic: subsidy-applied]
    end

    subgraph Processing_Mesh ["Processing & Event Mesh"]
        Topic1 -->|Consume| KStream1[Kafka Streams: <br> Eligibility Engine]
        Topic1 -->|Consume| SMSClient1[SMS Gateway Client]
        KStream1 -->|Write Eligible| Topic2[Topic: notify-subsidies]
        Topic2 -->|Consume| Outreach[Outreach Microservice]
    end

    subgraph Financial_Workflow ["Financial Workflow"]
        PFMS[PFMS / e-Kuber / NPCI] -->|Generate FTO| Topic3[Topic: fto-signed]
    end

    subgraph Downstream_Operations ["Downstream Operations"]
        Topic3 -->|Consume| SMSClient2[SMS Gateway Client]
        Topic3 -->|Consume| KStream2[Kafka Streams: <br> Analytics Engine]
        KStream2 -->|Write Aggregates| Topic4[Topic: subsidy-disbursement-trends]
        Topic4 -->|Kafka Connect Sink| Elastic[Elasticsearch / Kibana Dashboard]
    end

    %% Human Alerts
    SMSClient1 -.->|Security SMS| Farmer((Farmer: Dhanya Bhumi))
    SMSClient2 -.->|Approval SMS| Farmer
    Outreach -.->|Email Alert| CSC[Maha-e-Seva Kendra / CSC]

    style Ingestion_Layer fill:#f9f5ff,stroke:#7f56d9
    style Processing_Mesh fill:#f0f9ff,stroke:#026aa2
    style Financial_Workflow fill:#f0fdf4,stroke:#12b76a
    style Downstream_Operations fill:#fffbeb,stroke:#d97706

This decouples the critical path (e-Kuber/PFMS disbursement engine) from the notification services and the analytics dashboard. A failure in the SMS gateway layer will not lock up or crash the financial treasury system.

The unified event log acts as the Single Source of Truth – an immutable ledger of administrative state changes (subsidy-applied –> fto-signed), providing an audit-ready sequence of events for anti-fraud investigators.

Standardizing data masking at the edge (inside Kafka Connect via SMT) ensures that unencrypted personally identifiable information (PII) never reaches downstream topics, satisfying zero-trust architecture requirements out-of-the-box.


graph TD
    subgraph Operational Security & Resiliency
        A[Kafka Connect Edge SMT] -->|Enforces| B(Zero-Trust Architecture: No raw PII downstream)
        C[Unified Event Log] -->|Enforces| D(Single Source of Truth: Immutable Audit Ledger)
        E[Asynchronous Architecture] -->|Enforces| F(Fault Isolation: SMS Gateway crash cannot halt Treasury)
    end

    style B fill:#ecfdf5,stroke:#059669
    style D fill:#f0fdfa,stroke:#0d9488
    style F fill:#eff6ff,stroke:#2563eb

(Bejeck, 2024)

References

  1. Bejeck, B. (2024). Kafka Streams in Action: Event-driven applications and microservices, 2nd Edition. Manning.