Back to System Design Course

What are Events?

Events are actions. Whenever something happens inside or outside organization, an event has occured. These actions can be anything: a customer purchase, an inventory update in a warehouse, an attempted security breach, a change in state for an application, the list goes on.

Many events are time-sensitive or mission-critical. A business is event-driven when events trigger reactions within the organization. In the digital age, events are handled by systems that record, process and respond to events based on preset logic.

Event in system design share common characteristics:

• They are a record that something has happened.
• They are immutable - they can't be changed or deleted.
• They can be persisted indefinitely - events can be stored and accessed forever.
• They can be consumed an unlimited number of times - there is no limit to the number of times an event can be processed by a service.

Event Notification

Event Driven Architecture handle event notifications - an event notification is a message confirming an action has been taken. The notification may be a simple confirmation of the action or contain additional data about the event that downstream systems use to apply their business logic.

What is a event driven architecture?

Event Driven Architecture(EDA) is a system design practice built to record, transmit and process events through a decoupled architecture. This means that systems don't need to know about each other to share information and accomplish tasks.

How it differs from the APIs?

Decoupling is a central component of EDA, and it offers a major advantage over APIs. When sending an API call, the application needs to know the dependencies and parameters so it can send a properly formatted request. It then waits for the response before proceeding. This also means that there will always be a request and a response to transfer data.

In EDA,

• Event producer sends its notification without needing to know what happens after, leaving it to the Event router.
• Event router(Broker) - to decide where the message goes.
• Consumer(Sink) - receives the event to process without needing to request it.
• Event Channel - flow of event data between the producer, router and consumer.

Note: Every service that consumes the event works on it own i.e.. events are processed asynchronously.

Eg: Wifi Router - it receives the data from the internet and distributes it to different devices that need it in real time.

Design Patterns

Saga Pattern

• Problem Statement

Let's say a payment processing application needs to confirm and update the debiting account balance and confirm the outgoing payment.

The interactions in the workflow can be defined as follows

1. Account balance is checked to see if there is sufficient balance available in the account
2. If there is balance, the account is debited.
3. Payment is recorded as a confirmed payment or rejected based on the result from step 2.

Monolith Approach

In the monolithic world, the entire flow can be part of the same application, recorded in one DB.

If the tables for recording the payment and account data are in same schema in the same database server, a single transaction can update both the tables.

If the tables are in different schema present in different nodes of a distributed database server, a 2 PC commit protocol can be used to commit/rollback transactions. The payment is only confirmed once the account is debited. If the account balance update throws an error while updating, the payment update is also rolled back and vice versa.

Microservices Approach

In the moder world, the business workflow is divided into different services, where two completely different systems may be involved for maintaining account and payments related data. Each of these applications has its own database(per database per service pattern).

In such a scenario having a protocol like 2 Phase Commit, is very difficult to implement because of the varied nature of storage infrastructure involved.

The challenge with distributed transaction which needs to be resolved here is loose coupling of the services while still maintaining transactional consistency.

The above problem can be dealt with a EDA, by breaking the entire transaction into series of transactions which when executed locally in a certan order can help in maintaining transactional consistency across the individual domain boundaries of the services.

A saga is a series of local transactions which, when performed in order helps to maintain ACID rules across multiple data sources.

The saga pattern can help in maintaining transactional consistency by,

• Each app emitting events as they perform their local transactions(Choreographed Saga)
• Single app acting as the orchestrator for a distributed transaction that spans across different apps(Orchestrated Saga).

1. Choreographed Saga

Event Choreography is a architectural pattern in EDA in which each service performs its own business workflow by publishing and subscribing to events emitted by other services. There is no centrailzed coordination or logic.

When do you need?

• Abstraction is required between the services involved in a process
• When there is multiple teams and systems involved and there is no single owner of the processes.
• Useful for small/simple workflows.

Architectural Layout

In a choreographed saga, each participant application updates its own datasource and triggers an event in a message queue, announcing the completion of the update, other applications, subscribe to the event and act upon it.

How the pattern solves the problem statement

This pattern introduces loose coupling between the interacting applications and relies on eventual consistency.

In the above scenario, there are a series of local transactions in the Payments and the Account Service who interact by passing events to the respective message queues on successful completion

• Payment Service creates a record in Payments DB and marks the payments as PENDING.
• Payment Service then publishes a message to the Payments Queue, signalling the completion of the event of creating a payment in PENDING state.
• Account Service consumes the event and checks balance validity of the debting account, if valid updates the balance.
• Account Service publishes an event signalling the completion of the event in Accounts Queue.
• Payment Service subscribes to the event and updates the payment state to POSTED.
• Payment Service publishes a message to the Payments Queue, signalling the completion of the event.

The success of this pattern can be measured around the following failure scenarios:

• Account Service fails to get account balance from DB : There will be no messages generated in the Accounts queue. The payment will remain in PENDING state until the account service succeeds.
• Validation check fails in the account service : A compensating event will be generated in the Accounts queue. The payment service will rollback the transaction.

Trade-Offs

This pattern may not be good, if the dependency between the services chain beyond 3 or more and the client facing service needs to respond. For example, if the account service needs to call another service, like the customer service which is responsible for checking customer details in a similar fashion and that service fails to respond on time, the account service needs to take a call to update a compensating transaction to rollback its DB changes and also post a failure message in the accounts queue.

2. Orchestrated Saga

Event Choreography helps in creating a true decoupled distributed system but can have some downsides. They are

• It is not very easy to add a new step in a business transaction which is spanning across multiple business domains.
• Monitoring and handling error scenarios as a transaction flows through multiple business domains.

Event orchestration is an architectural pattern in which a single service acts as an orchestrator for facilitating event distribution.

Orchestration is a way to centalize the workflow of logic for a business process. It coordinates the workflow by sending commands to approproate service, consuming the resulting events. In contrasts to choreography, Orchestration tells other services what action/command to perform rather than those services being reactive to other events in the system.

When do you need?

• Complex workflows requiring full control over the process steps.
• When there may be a need to enforce the order event processing.
• When a single teams owns the process.
• When a workflow requires alternative path based on the result of an Nth event result.
• When Retries, Rollback are required to mitigate the failures.

Extending the problem statement

Expanding on the problem statement a little bit, let's say a new service(notes service) is introdueced which needs to be updated before each successful payment is recorded. This new service has its own domain services and DB and can interface with both API calls and Message streams.

Implementation of such rule constraint features is difficult in the choreographed saga, as we are not sure on the ordering for the event processing(the notes DB can never get updated even after the payment is recorded).

An orchestrator can be good fit in this architecture where a particular service(mediator) establishes the ordering of event processing.

Architectural Layout

In an orchestrated saga, the participants in the transaction are orchestrated by a coordinator service.

There are two ways to coordinate the orchestration.

One Participating Service Acts as the Orchestrator

In this flavour, one of the participating services in the saga, acts as the orchestrator. Generally, the servic which initiates the transaction acts as the orchestrator(in this case, the payments service).

If then issues commands(not events) to the other services in the saga to perform their local transactions.

Once the transactions are performed the pariticipating service responds with a success message. This triggers the next command in the sequence.

Third Party Service acts as Orchestrator

In this flavour, there is a third party service which acts as the mediator for the saga.

This service can be any other service which uses mediator frameworks(Apache Camel, Spring integration) or a mediator tool(Mule) to orchestrate the transaction which can interface on message queues and API Responses.

Message queues provide more decoupling compared to API responses.

In this case, all the services will be interfacing with mediator through request and response queues on agreed upon message contracts.

How the pattern solves the problem

With this pattern, servcies can be easily added or removed from a transaction, hassle free while maintaining transaction ordering.

In this case, the Note Service can be added to in the correct sequence or removed from the transaction, without the other services getting impacted by the change.

Trade-Offs

The business rules for the workflow are leaked to one particular service.

The orchestrator can be a single point of failure.

The orchestrator needs to persist the state of the transaction till completion. If the transaction fails in one of the participating services, the services updated before it needs to be sent a compensating transaction to perform a rollback.