Optimizing Data Analytics and Backend Processes for a Financial Institution

By /

Finance

Optimizing Data Analytics and Backend Processes for a Financial Institution

Illustration of a financial trading engine powered by cloud infrastructure and real-time data integration

Focus Areas

IT Infrastructure

Digital Transformation

Data Analytics

Illustration of system modernization through Oracle to SQL Server data warehouse migration for improved performance and scalability.

Business Problem

A large financial institution group was facing problems with generating timely financial reports. The group used Elasticsearch databases and MySQL to extract data and generate financial reports. However, this report generation process was slow and prone to errors, and delayed the delivery of accurate financial insights to the internal leadership team as well as external stakeholders such as regulators and investors.

Key challenges:

  • Slow Report processing: The client’s existing system – reliant on MySQL and ElasticSearch databases – took too long to extract data to generate reports. The delay frustrated external stakeholders who needed these insights to make timely financial decisions and also slowed down internal decision-makers.
  • Data extraction prone to error: The reports generated from the existing system were prone to errors due to the inefficiencies of the data extraction process. This often led to the need for manual corrections which caused further delays. 
  • Out-of-date backend infrastructure: While the institution and the volume of data had been scaling, the backend system architecture had not been. Due to this, as more data flowed in, the performance of the backend system reduced, and led to operational challenges.
  • Increased operating expenses: The ineffectiveness of the backend system and the effort required to manually correct errors in the generated reports increased operational expenses and decreased overall profitability.
  1.  

The Approach

Through close collaboration with the client’s internal IT, finance, and data analytics teams, Curate’s consultants decided to focus on implementing backend optimization techniques, upgrading data workflows, and using serverless architecture to redesign and streamline the entire reporting process.

Key components of the solution:

  1. Requirements gathering and assessment: Following a thorough assessment of the client’s existing infrastructure and workflows, which included interviewing key stakeholders across departments, reviewing the current report generation processes, and performing a detailed analysis of the performance bottlenecks in the system, Curate’s consultants defined the following key objectives for the solution:

    • Optimize data extraction and processing to reduce the time required to generate financial reports 

    • Enhance the accuracy of the reports by automating data validation and minimizing manual interference

    • Implement a solution that could be scaled to accommodate the institution’s growing data volume

    • Reduce operational costs by improving backend efficiency and automating key processes

  2. Optimizing the backend with Python and FastAPI: Several key changes were implemented to resolve the inefficiencies present in the backend:

    • Curate’s consultants improved the efficiency of data queries – significantly reducing the time required to pull data from both databases – by rewriting the extraction logic in Python. The data extraction workflows were refactored from MySQL and Elasticsearch to Python to optimize performance.

    • FastAPI for REST API Development: FastAPI, a high-performance Python web framework that enabled quicker and asynchronous data processing, was used to develop a new REST API for handling data requests. This enabled the system to handle multiple data extraction requests simultaneously without compromising performance and sped up the report generation process by 40%. 

    • Caching Frequently Accessed Data: For the most frequently accessed data, a caching layer was implemented using Redis. This further improved the efficiency, reduced the load on the MySQL and Elasticsearch databases, and sped up the reporting process.

  3. Utilizing AWS Serverless Architecture: AWS Lambda serverless functions were introduced in addition to optimizing the backend to streamline data processing and automating key tasks in the report generation workflow.

    • Automating workflows with AWS Lambda: AWS Lambda functions were used to automatically trigger data extraction, transformation, and report generation tasks. The automation ensured that reports were generated quickly, consistently, and did not always require manual intervention.

    • Serverless computing for scalable infrastructure: AWS Lambda could automatically scale the financial institution’s infrastructure based on demand. In periods of peak demand, the system could handle a large volume of requests for report generation without compromising performance. Since the client only paid for the compute resources used during report generation, it also lowered operating costs.

    • Real-time data processing: To enable real-time data streaming, Curate’s consultants integrated AWS Kinesis into the system to process new financial data as it became available. This improved the accuracy of the reports and the relevance of the insights.

  4. Improving Data Validation and Report Accuracy: To reduce errors, the Curate team created automated data validation processes:

    • Automated Data Validation: To ensure the extracted data met the required accuracy and completeness conditions, Curate’s consultants developed validation scripts that would run before the report was generated. This removed the need for manual checks and corrections and reduced the time spent on revisions.

  5. Ongoing monitoring and continuous improvement: A continuous monitoring and optimization framework was set up to ensure the continued efficiency of the system.
    • Alerts for issues in report generation: Performance monitoring of the system and alerts were set up with AWS CloudWatch for issues such as failed data extraction or incomplete reports. The IT team would be alerted to any such issues, allowing them to quickly resolve problems before they had any significant impact on reporting timelines. 
    • Performance monitoring: By utilizing AWS CloudWatch and ElasticSearch Monitoring, the system performance could be tracked. Metrics such as data query times, report generation times, and resource utilization allowed Curate’s team to further optimize and ensure the effectiveness of the system.
    • Regular Audits and Optimizations: To ensure that the system remained agile and scalable with the growing data volume, Curate’s consultants performed regular audits of the backend processes, identified potential issues, and made the necessary improvements.
  6. Training and Change Management: The Curate team provided thorough documentation and training to the client’s internal teams to ensure they could manage and maintain the new system and workflows. The training included:

    • Using the new Python-based data extraction workflows and FastAPI endpoints.

    • Best practices for leveraging AWS Lambda and serverless architecture.

    • Techniques for monitoring system performance and handling error alerts.

Business Outcomes

The transition from Waterfall to Agile, led by Curate Consulting, resulted in transformative improvements for the healthcare provider:

Decrease in Operational Costs


AWS Lambda and serverless architecture implementation reduced operational costs by 25% since resources were now scaled based on demand.

Improved Report Accuracy


Data validation automation processes decreased errors by 30% in financial reports, ensuring accuracy, consistency, and minimal manual intervention.

Scalable Infrastructure


The serverless architecture could handle growing data volumes without performance deterioration, allowing the client to generate reports efficiently even though their financial data was growing exponentially.

Sample KPIs

Here’s a quick summary of the kinds of KPI’s and goals teams were working towards**:

Metric Before After Improvement
Time to generate financial reports 10 hours 6 hours 40% reduction
Operational costs (backend infrastructure) $500,000/year $375,000/year 25% reduction
Report accuracy (errors per report) 20 errors/report 14 errors/report 30% improvement
Resource scaling time 3 hours 5 minutes 95% improvement
Financial report generation capacity 100 reports/day 200 reports/day 100% increase
**Disclaimer: The set of KPI’s are for illustration only and do not reference any specific client data or actual results – they have been modified and anonymized to protect confidentiality and avoid disclosing client data.

Customer Value

Curate Consulting’s expertise in Agile methodologies not only improved operational efficiency but also enhanced the healthcare provider’s ability to serve their patients more effectively:

Faster Financial Reporting


The client could now deliver more timely and accurate insights to internal and external stakeholders. The optimized data extraction workflows and automated backend processes reduced the time required to generate financial reports by 40%.

Improved Decision Making


The client’s leadership team, especially, could make quicker data-driven decisions due to the enhanced financial reports. This also strengthened the institution’s reputation with external stakeholders.

Enhanced security and system performance monitoring for financial services.

Conclusion

Curate collaborated with a large financial institution group to improve their data analytics and backend operations for quicker and more precise financial reporting. By utilising Python, FastAPI, AWS Lambda, and serverless architecture, Curate’s consultants assisted the customer in lowering operating expenses, speeding up report generation, and scaling their infrastructure to accommodate growing data volumes. This resulted in improved accuracy of financial insights, more business analytics value, improved customer trust, and better decision-making capabilities.

All Case Studies

View recent studies below or our entire library of work