Global
Case Study
The “Original Fintech” Takes to the Cloud: Migrating Instinet’s Data to AWS
Executive Summary
Instinet engaged Ness to migrate its core database to AWS, a major step in the Digital Transformation of “the Original Fintech.” Instinet, a major player in agency trading and a subsidiary of Nomura, has tough and uncompromising security requirements. Ness built a secure AWS Landing Zone to enforce these stringent controls. Ness then leveraged its business domain knowledge, technology expertise, and an Agile delivery process to adopt a “data-first” migration strategy. A new pipeline services model using AWS MSK and AWS KDA has re-orchestrated Instinet’s existing data capture and distribution logic by providing near real-time access to data. Extending Instinet’s legacy on-premises environment to the cloud has enabled the legendary electronic broker to achieve record scale—deftly handling exponentially higher market volumes amid the unexpectedly disruptive global landscape of the last 18 months.
About Instinet
Instinet provides technologically advanced, agency-model brokerage services to institutional clients worldwide and is the wholly-owned execution services arm of Nomura Group. They help institutional investors create, protect and capture alpha, reduce complexity, and lower overall trading costs to ultimately improve investment performance.
Instinet seeks to add value across the investment life cycle, providing:
- Agency Execution Services
- Crossing and Liquidity Venues
- Instinet Technology Solutions
The Challenge
Instinet’s traditional on-prem infrastructure faced projected scaling issues amid continued, exponential increases in market trading volume. The equities broker regularly ingests and processes 1TB of trade data daily. Additionally, it has accumulated petabytes of data throughout its 50+ years, creating a heavy operational and capital-expenditure load. By maintaining its core database (“CoreDB”) in a physical data center, Instinet faced challenges and logistical inefficiencies that impacted the development of new products and created potential risks to the client experience.
Any proposed solution required that the data platform could seamlessly integrate into existing systems, so as not to disrupt access to multiple data consumers.
Why AWS
AWS’s depth in the financial services industry and its ability to customize Instinet’s workflows made a compelling case to move select infrastructure the cloud. Choosing AWS would not only keep Instinet compliant with a host of evolving global regulatory requirements, but would also deliver the ongoing low-latency integration, scalability, and information security that the firm and its clients require.
Why Instinet Chose Ness
Instinet engaged Ness because of the consultancy’s extensive experience working with AWS and its expertise within the financial services industry. Ness is a Premier Confluent Systems Integrator and an AWS Advanced Consulting Partner with Financial Services, Migration, and DevOps Competencies. Ness has a long track record of leveraging its business domain knowledge, technology expertise, and Agile process to deliver on mission-critical projects in capital markets.
The Solution
Ness took a data-first migration strategy that re-orchestrated Instinet’s existing business logic in readiness for optimal cloud deployment. Ness teams created a data ingestion pipeline that utilizes AWS-managed services for horizontal scaling, high availability, and ease of operations.
- Amazon Managed Streaming for Apache Kafka (MSK) holds Kafka topics, allowing for replay and re-ingestion of data. Amazon MSK allows for parallelizable processing, and Kafka topics are consumable by different systems at Instinet.
- Amazon Kinesis Data Analytics (Amazon KDA) consumes data from Kafka topics and persists data to Amazon Aurora.
- Amazon Elastic Container Service (Amazon ECS) contains the producers to the Kafka topic that securely pulls data from on-prem. It also contains applications for loading market and reference data and for providing summary and statistical analysis on the data.
- Amazon Aurora, a MySQL database service, provides an alternative solution to Instinet’s on-prem MySQL database.
- Amazon Redshift as a multi-year data warehouse of historical trading and reference data provides long-term queryable storage alongside Aurora.
- AWS Lambda and AWS Step Functions automate all weekly operational processes that were normally handled by database administrators and operational support.
Additionally, trade-related data was moved off EMC Storage into tiered storage in Amazon Simple Cloud Storage (Amazon S3).
Results and Benefits
Instinet’s investment in AWS has dramatically increased its data-ingestion rate while enabling other applications to process real-time data in the cloud. Moving CoreDB to Amazon Aurora has increased insert and query rates eightfold, allowing the broker to sustain increased business in the face of historic market volumes., During market peaks, these increases in processing capability mean that the technology applications accessing and consuming data have seen exponential improvements in speed – by a factor of five.
The introduction of the Amazon Redshift ecosystem has also provided a cost-effective ability to increase the amount of data that can be retained in online queryable storage by years, and flexible scaling that accommodates Instinet’s projected market volume growth into the foreseeable future.
Additionally, Instinet has experienced four important business advances:
- It has reduced the release process from days to mere hours;
- The operational maintenance window has shrunk from being an overnight process to now taking about an hour;
- By moving trade-related data to Amazon S3, the storage footprint has been cut by 50%;
- And using on-demand resources has aligned with the goal of slashing the company’s carbon use.
Finally, Instinet has gained greater, more nuanced visibility and can access its data assets via multiple parallel applications, and is able to more efficiently harness AI/ML deep learning to develop an array of dynamic future product offerings.
“Migrating our core database to AWS is another example of how Instinet’s tradition of innovation continues to evolve and keep us at the forefront,” said Minor Huffman, CTO of Instinet. “Ness has helped us move to the cloud in a way that not only understands our business goals and challenges but also the long-range opportunities that can be unlocked by leveraging their expertise with cloud and streaming technologies.”
