Ness Delivers Order Management Platform at Top 20 Global Bank
The wealth management division of a top 20 global bank engaged Ness to re-architect its order management system using Kafka. Ness worked closely with Confluent to deliver a re-architected streaming application that efficiently, reliably, and securely handles orders for the client’s entire line of products.
About The Client
Client is the wealth management division of a top 20 global bank. They provide investment advice to help preserve and grow wealth, customizing their strategy with each client using a combination of equities, options, and mutual funds.
The client’s proprietary trade order management system was in need of a technology refresh in order to provide greater throughput, resiliency, and fault tolerance. The new platform had to meet the following requirements:
- Process a minimum of 250 orders per second with the ability to scale linearly
- Real-time replication across multiple data centers
- Recovery of messages from a specific point in time (in case of unavailability of external systems)
- Message sequence guarantees for all events related to the same trade order
- Elimination of single points of failure
Why The Client Chose Ness
Ness is a Premier Confluent Systems Integrator and the only partner in the Confluent ecosystem focused exclusively on financial services.
Ness was selected because of its unique combination of business domain knowledge, technical expertise, and Agile delivery process.
The new architecture was based on a data processing platform with Kafka at its core. An initial Proof of Concept (POC) allowed the client to assess the effectiveness of the new system for handling equities orders. Ness incorporated client input on the performance and reliability of the re-architected application and expanded the system to handle the client’s complete line of products.
The Kafka cluster was based on reference architecture provided by Confluent, which included VM sizing, kernel parameters, file handles, and additional specifications
- A multi-region cluster was built across three states, providing synchronous, real-time replication in an Active-Active-Active environment.
- Ansible playbooks were written to automate deployment of the cluster, based on the cp-ansible playbook provided by Confluent.
- Confluent Control Center was setup to monitor system health and end-to-end message delivery.
- SSL with SASL/SCRAM was used for security, with the intention of future replacement with Kerberos.
- Monitoring was setup using the customer’s AppDynamics deployment.
A microservices architecture consisting of the following services was built out:
- A bulk order entry service, whose input is a Kafka topic. This simulated a delay and published messages to the output topic.
- A gateway component that reads messages from the output topic and converts them to FIX protocol using a simulated FIX engine.
- A HTTP order entry service, which received single order at a time over REST and published them to the output topic.
Performance and Fault Testing
Ness worked closely with Confluent to develop an extensive fault-testing plan for the client’s application while the Confluent team worked to help build out the extensive platform deployment.
- KSQL Datagen was used to produce test data in bulk, based on a provided schema. Ness setup an AVRO schema for “Order” messages.
- The high availability and fault tolerance of Confluent’s multi-region cluster enabled the team to prove that the system could withstand a single data center failure, albeit with reduced capacity, and without message loss.
Ness provided mentoring to client teams throughout the software development lifecycle, enabling them to maximize the benefits of their new Kafka-based system.
- Training team members in the use of Spring for Apache Kafka
- Performing code reviews
- Implementing efficient Git flows
Results and Benefits
Ness worked closely with Confluent to deliver a high-performance, resilient order management system for the client. The Ness team successfully delivered the entire system on budget and according to the client’s specifications. As a result, the client now has an application that can handle orders for its entire line of products in a highly available, fault-tolerant system. “Ness’s expert knowledge of trading systems complemented Confluent’s expertise with Kafka-based architectures to successfully redesign this mission-critical system,” said Tom Arnemann, Confluent Professional Services. “Ness’s attention to detail, combined with its iterative approach to software development, ensured that the client’s business needs would be met by the new streaming platform.”