M1 Finance realized that its current infrastructure wasn’t equipped to handle its current rate of growth. It needed an efficient and scalable data pipeline on Amazon Web Services (AWS) to capture and process login data, allow rapid querying and accelerate fraud detection.Solutions Cloud, Public Cloud, Data, Data Management, Data Modernization Platforms Cloud, Amazon Web Services (AWS), Data, AWS Data
Intelligent financial management
M1 Finance is a groundbreaking online financial services company headquartered in Chicago. The company offers an innovative money management application for intelligent investors who want to access features such as automation, leveraged investing and banking services, all wrapped into one integrated solution. Its mission is to empower people with tools and automation to improve their financial well-being.
M1 Finance has been growing rapidly, doubling its employee count from 30 in 2019 to more than 65 employees in 2020. The company’s platform has also experienced significant growth, with over 200,000 accounts totaling approximately $2 billion in assets. To ensure a secure user experience, M1 Finance aims to provide a seamless interface with robust security and fraud detection.
Legacy infrastructure inhibits growth
M1 Finance’s platform was built on AWS prior to its engagement with Onica, a Rackspace Technology® company. The company had a data warehouse comprising an Amazon Redshift cluster managing just under 400GB of data. The cluster facilitates its internal analytics needs and serves as a clearinghouse for data from backend services as well as third-party vendors and other sources.
In addition to the data warehouse, M1 Finance has a data lake using Amazon S3 that serves as a staging ground for loading data into Amazon Redshift and as an archive store. However, the data lake wasn’t equipped with capabilities for ad hoc queries — a functionality M1 Finance was looking for to improve its fraud detection capabilities.
With rapid user growth, the company found it was collecting significantly higher amounts of data from internal systems than what could fit in its existing data warehouse. Handling such high data volume could get very expensive with traditional data warehousing.
Additionally, the company also had Java applications, which had accumulated data over many years, that could be highly useful for fraud analytics. This data was trapped in silos of legacy databases and it wasn’t able to find a solution to query a high volume of rapidly moving data with its existing system.
As an organization in the financial industry, M1 Finance takes user data security extremely seriously, and it’s imperative that its team has the ability to track system access and usage to detect and prevent fraud. The company’s existing process for reviewing system logs was manual and not scalable. As the volume of data and logs continued to increase, the company required a more efficient process to execute a task that is highly time-sensitive.
The M1 Finance team wanted real-time visibility into suspicious activity so it could respond as soon as possible. It learned about Onica from the AWS Jumpstart program and was impressed with the expertise and capabilities available at Onica. It found the team had insight and knowledge that could be valuable to M1 Finance’s needs.
“The Onica team builds trust and confidence that they care about our problems and about producing custom solutions that are highly effective and cost-efficient.”
A clear goal leads to a strong solution
Onica worked closely with the M1 Finance team to architect and build a solution leveraging AWS services for enhanced automation, visualization, analysis and data reporting. The Onica and M1 Finance team also worked collaboratively to establish clarity on project scope.
The Onica team designed an efficient and scalable data pipeline to capture small AWS Database Migration Service files dropped into M1 Finance’s Amazon S3 bucket from its authentication database. Login event data is captured, organized and partitioned, and data partitions are registered within an AWS Glue data catalog. The files are compacted into fewer larger files for optimal I/O performance, which can then be queried through Amazon Redshift Spectrum or Amazon Athena.
Onica extended the capabilities of its database by creating a separate data path. This facilitates a variety of cost controls, access controls and governance that can be applied at the Amazon Redshift Spectrum or Amazon Athena query interfaces. Together, this allows M1’s engineering team and frontline fraud analysts with SQL knowledge to query the data dynamically and in real-time without requiring an engineer’s expertise.
The solution implements Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift Spectrum, and AWS Lambda. Amazon Redshift Spectrum was used because the data was also going to be combined with other data already within an existing Amazon Redshift cluster. Amazon Redshift Spectrum uniquely provides an additional layer where data can live outside of the data warehouse on Amazon S3, but still be queried like it’s within the cluster. This allows M1 Finance to avoid paying large costs for querying a large scale of data, and still maintain optimal functionality.
“Onica’s team really went above and beyond in ideating and implementing the solution,” said Richard Whaling, Lead Data Engineer at M1 Finance. “A great degree of passion and craftsmanship was visible in the code style and design of the implementation, which is rare to find in these kinds of engagements and really awesome to work with.”
Automation reduced process times from several hours to near real-time in the new solution.
Real-time scalability and efficiency
The new solution streamlined the process significantly, automating what was previously done manually over multiple hours to a practically real-time solution. The design of the solution provides scalability, low cost operation and integration with existing data sets.
Furthermore, the solution offers great flexibility, replacing the process of needing to queue tasks for an engineer with one where data analysts can perform ad hoc dynamic queries. M1 Finance business users now have the ability to build robust queries at scale and are able to address their multi-petabyte size data sources and return meaningful timely results. This provides a significant advantage in fraud detection where every minute reduced in processing time can mean proportionately greater success and improved security.
To help mitigate cost risk for M1 Finance, Onica built alerts for Amazon Redshift Spectrum and Amazon Athena. M1 Finance’s team was also pleased with the delivery of the solution that was carefully tailored to its existing practice and unique use case. The company felt confident about the way the team at Onica handed-off the solution, and educated the M1 Finance team in effective implementation and management.
About Rackspace Technology
Rackspace Technology is a leading end-to-end multi-cloud technology services company. We can design, build and operate our customers’ cloud environments across all major technology platforms, irrespective of technology stack or deployment model. We partner with our customers at every stage of their cloud journey, enabling them to modernize applications, build new products and adopt innovative technologies.
Let’s Talk Strategy
Tell us a little about your challenges and we’ll contact you.