Data collection platform
The Company is operated in North America and has been the leading provider of strategic data and analytical solutions for the agriculture, crop protection and animal health industry for almost 30 years. They were using and maintaining multiple legacy systems and custom applications to collect, cleanse and analyse sales, product, and customer data from thousands of sources across the US and Canada. These systems consisted of hundreds of databases, over 20 web portals and countless SSIS packages, stored procedures and custom workflows which were maintained and operated by multiple teams in 3 different geographic locations.
Those systems were storing and processing redundant data and contained duplicated inefficient manual processes for transforming and cleansing incoming data resulting in costly data quality issues. An enormous amount of time had to be spent to integrate, modify and support those systems.
Onboarding new customers required creation of new databases, configuration changes in multiple systems and applications and the process was taking weeks. Initial historical data loads could take days.
Our solution begins with clarifying business requirements, advising in software architecture and day to day development, on both, backend and frontend technologies. The key objective is keeping consistent agile processes. The solution consists of a single data integration platform replacing all existing tools, websites and processes. It is based on 10+ scalable Microservices hosted on Azure Service Fabric cluster which allows asynchronous processing with multiple service instances making processes faster and efficient. Thanks to Azure Service Bus and message based communication through parallel queues managed by priorities we gain significant performance in the data ingestion process. To make it easy to scale, the system was implemented based on CQRS (Command query responsibility segregation) and Event Sourcing principles. SignalR is used to update the user’s view, and inform them about changes in systems state.
The implemented solution improved file processing time, which decreased from days to minutes. The geo independent tool aided the expansion of the business to multiple locations and industries much faster and improved the organization’s ability to make more accurate and efficient decisions.Onboarding of new customers was reduced from weeks to days.
It reduced costs of maintenance and support, resources required for data extraction and the trouble of tying multiple systems together.
The vastly improved data quality allows our customers clients to make sound business decisions and gives an opportunity to find new markets and distribution channels.