Data aggregation is a type of data and information mining process where data is searched, gathered, and presented in a report-based, summarized format to achieve specific business objectives or processes and/or conduct human analysis. Data aggregation may be performed manually or through specialized software. An aggregation platform, takes care of the collection, processing, and sometimes even the presentation of data. It’s an essential part of data integration. Data aggregation helps summarize data from different, disparate and multiple sources. It increases the value of information.

The best data integration platforms can track the origin of the data and establish an audit trail. You can trace back to where the data was aggregated from. It’s important to understand that aggregate data is not limited to numbers. Data aggregation may be performed manually or through software expressly written for such purpose.

For that matter and in order to boostup some of our R&D ideas, we decided to develop a one of a kind platform for data aggregation using the best technologies in the market. It has to be pretty fast and pretty performant, should be cloud native and modular and should be able to scale to billions of records automatically. The system is currently in beta and we have used Rust as the programming language plus we heavily used DAPR for our seamless microservice architecture implementation. The whole setup is on Google’s GCP and Also we have used Clourflare’s workers platform for our serverless hooks.

Being stable, this platform is going to do our future projects a solid by providing them the best way possible to access the data scattered on the internet with the maximum flexibility on the structure and API design.