Summary: This article delves deeper into the reasoning as well as the decision-making process in choosing the infrastructure and architecture of the iClosS platform.
Since the inception of iCloss Project we were careful to create a pure cloud based Infrastructure life-cycle, with all the resources required to support development, testing and implementation of the project in a public cloud, specifically Microsoft Azure. Here’s in this article we explore a bit more about our high-level Infrastructure and Architecture. We have based these decisions on years of expertise on building scalable and secure mission critical products with a focus on being true to the business and technical stakeholders. The goal of this article is to provide a common understanding of the proposed infrastructure and help our future clients and prospects get a below the hood take on how we have built iClosS.
The iClosS project is a big data project that will have a significant impact on the way it integrates with our clients businesses processes and systems. To support this project, we will be provisioning Infrastructure resources that include servers, storage, and networking equipment. iClosS will be making using of Microservice, which are gaining a lot of popularity these days, as it allows businesses to break their systems down into smaller, more manageable pieces. This makes them easier to scale and faster to develop. The following are infrastructure resources that will be provisioned to file transfer, database, file processing and reports.
File Transfer
With iClosS we provide a Secure File Transfer service for clients to upload their CSV format files for processing to intake data needed for Aggregate Spend and Transparency Reporting process. As this will be an Internet-facing service and iClosS has ensured that the state-of-the-art Security Practices are in place to secure this transaction. The Secure File Transfer Protocol (SFTP), has advantages to setting up and maintaining an on-site server as there are no hardware requirements resulting in reduced business costs, as well as enhanced data security and convenient file management. Kubernetes is used as it’s flexible and scalable, open source tool orchestrator that streamlines the task of managing containers. Azure Kubernetes Service (AKS), uses only the resources you need, with security and compliance done right, supports agile project management and accelerated app development. We at iClosS have used Atomz SFTP, as the technology stack for this service.
Database
Within iClosS we need to be able to intake, transact, refine, and report on large volumes of data depending on the client in question. Based on that we have gone ahead and made sure that the service we provide will have the ability to store large volumes of data, for both raw and processed data, as well as to generate reports by end-users. iClosS databases are not accessible externally, for example, without a VPN connection. Comparing Microsoft SQL Server and Azure SQL, we have based it on SQL Server, as they share many similarities in functionality and compatibility.
File Processing
The service processes files uploaded by clients through file transferring service and then stores the resulting normalized data using the database service within the iClosS tables. This procedure is called ETL and here we have used a combination of Microsoft Data Factory and Azure Data Factory to fully manage serverless data integration service.
Reports
In this service, Power BI will be used to allow end-users to generate reports based on pre-defined templates or create their own custom ad Hoc reports to satisfy their short-term requirements. This is also an Internet-facing service and we have secured the same using HTTP(S) protocol for all client-server connectivity, as well as the connections can be proxied through reverse proxy services like HA Proxy or Cloud Flare for better security.
The iCloss project is a significant integration for our clients and will require the deployment of various infrastructure resources. The project will have 4 microservices: file transfer, database, file processing, and reports. Each of these services will require specific infrastructure resources that needs to be carefully implemented.
Summary
At iClosS we understand when a client chooses us for their Aggregate Spend and Transparency reporting, they have placed a huge amount of trust on this team. We are confident that we will stand true to our promise of treating the customer data and infrastructure as our own and make sure we follow the best industry practices in engaging and handling the systems, workflows, and data entities. We have used our vast experience made the Platform as secure and scalable and we hope to showcase our best-in-class processes with the best in industry technology platform to our clients.