Nonprofit Data Engineering Work & Support

Technology Stack

Java
Spring Boot
MongoDB
Oracle
Angular

The Client

This is a product review website of a leading nonprofit organization. Their website publishes thousands of reviews for products like cars, bikes, refrigerators, cooktops, laptops, printers, etc. The platform owners publish these online reviews after testing these products in-house, gathering customer feedback, and putting together other statistical reports. These reviews, posted through data-based scientific methods, help customers make informed buying decisions.

The Challenge

Since this product review website contains massive amounts of data, the client needed an experienced tech partner who could help them with data capture and engineering. The client was already facing problems extracting high volumes of data from disparate sources. So, they were looking for a knowledgeable partner who could consolidate data from varied sources and bring them together into a single solution or a data warehouse and then ultimately present it to the end user.

After trying to manage the workflow in-house, the client found it too overwhelming to handle it entirely alone. So, they turned to Capital Numbers. The client chose us because we have a solid grasp of data handling solutions.

Capital Numbers delved deep into the criticalities of the client's data engineering needs and jotted down the key areas to focus on. Some of these areas included:

Mapping data sources to the target database
Data cleansing
Data ingestion
Data synchronization
Data transformation
Visualizations at scale

The Solution

We started by establishing a robust ETL path that would help us extract, transform, and load data strategically.

We worked towards moving data from varied sources, like CRMs, third-party lists, mobile apps, etc., to the target database systematically using custom logic.

We didn’t move data from these sources to our target database in bulk because that would cause latencies. Instead, we moved data in batches, which ensured a streamlined migration.

Once we extracted data from different sources and put them in a central repository, we looked for inconsistencies or incorrect values (if any). We discarded duplicate entries. We got rid of all anomalies. We also applied additional custom rules to improve data quality.

After sorting and improving the data quality, we worked towards loading data into our data warehouse. Here, we gave special attention to verify all data belonged to appropriate tables when loaded in the data warehouse. We used custom scripts to ensure the data loading process took place smoothly.

Currently, it’s an ongoing process where our data engineers regularly populate the data warehouse with new data. We also configure components when data is added or changed in the data warehouse.

Every time our data engineers feed new data, our data visualization experts use custom logic and HTML5/CSS3 to visually present that data on the frontend through graphical reports for end users.

Our backend developers used Java for this data engineering project because Java is ideal for ETL environments that handle huge amounts of data volume, data velocity, and data veracity. Java allows flawless extraction, transformation, and loading of big datasets.

Our engineers leveraged Oracle database and MongoDB to sort, structure, and segment massive datasets. Both these database solutions ensure excellent data querying, availability, and security.

For the frontend, we used Angular because it’s lightweight, cross-browser compatible, and allows data to move from Javascript code to the view without manually writing code.

All in all, from sourcing data to presenting it to the user, we handled it all.

Results

Because of Capital Numbers’ end-to-end solutions, the client can efficiently take charge of an enormous data flow.

The resulting outcomes are the following:

Batch-driven ETLs

We help the client perform batch-driven data extraction, transformation, and loading, ensuring the system doesn’t slow down.

Data Ingestion

We closely work with the client’s team to gather and ingest data from complex formats.

Data Syncs

We regularly test ETL paths to ensure data is well-synchronized across sources and formats.

Spring Boot Upgrade

We also upgraded the existing Spring Boot v1.4 to v2.3.12 to accommodate more data formats and variety.

Data Variety

We regularly handle massive data variety related to products like:

Cars
Vacuum Cleaners
Dishwashers
Refrigerators
Televisions
Printers
Laptops
Bikes

We also work with thousands of data points, such as:

Brand Names
Franchise Names
Attributes
Prices
Usage Rates
Energy Efficiencies
Model Names
Model Availabilities
Product Descriptions

We source and extract information related to the following:

Sub-category IDs
Sub-franchise Names
Sub franchise URLs
Version Names (42-inch TVs, 55-inch TVs, etc.)
Attribute Values (Polyester Pillows, Memory Foam Pillows, etc.)

Visual Reports

We turn such vast data sets into analytical reports using scientific algorithmic functions. For example:

Reports we generate for cars include the following:

Road Test Results
Comfort Reports
Reliability Scores
Owner Satisfaction Scores

Reports we generate for refrigerators include the following:

Energy Use
Thermostat Control
Temperature Uniformity
Predicted Reliability

Data Intelligence

All the data intelligence reports we display visually are in the form of colorful charts.

Comparative Analysis

By looking at these reports, buyers can compare products before purchasing one.

End-to-end Data Engineering Workflow

Our end-to-end data engineering workflow - from data collection, preparation, and transformation, to analytical solutions, has benefitted the client immensely.

Empowering the Client’s Team

Today, even if the data volume grows tenfold, our services help the client ensure faster time-to-market at lower costs - saving their bandwidths and efforts significantly.