Data Science App Empowering Venture Capitalists to Find High-Value Startups

Solutions We Offered

Our team began the work by studying the existing analytical app. We found the data sets badly scattered in the existing setup. For example, S3 buckets contained raw data, PostgreSQL contained SQL data, and DynamoDB contained NoSQL data. So, we prioritized organizing all that data in one place using Snowflake.

We chose Snowflake because this cloud data warehousing solution is excellent for storing large workloads of near-unlimited scale. It allows firms to store overwhelming amounts of data economically while dynamically scaling compute resources to meet the demands of active querying processes.

We ran CRON jobs and custom APIs within Snowflake to automate data extraction from external sources. Additionally, we implemented ETL processes to centralize all data in desirable formats for structured analysis.

Next, we had to show the collected data on the frontend. So, we implemented Sigma because it reflects data on a spreadsheet-like analytics interface. Although Sigma resembles a traditional spreadsheet, unlike traditional spreadsheets, it pushes far more complex queries to the data warehouse to return accurate results against billions of data.

Moreover, traditional spreadsheets have a hard limit on the number of rows. However, Sigma can visualize billions of rows of datasets in seconds, putting no limits on analytics. Sigma also allows calculations to be performed on columns instead of rows, allowing users to work on large datasets without worry.

In addition, once built, Sigma’s reports directly query live data sets to the data warehouse and update them in real-time. So, we didn’t have to manually ingest data pipelines to connect the backend and frontend, which saved us time. We also developed custom React plugins and JavaScript APIs in the Sigma package to power up the frontend.

Once we fully prepared the application for data collection, storage, monitoring, analysis, and reporting, we hosted it on the AWS server. The collaboration required us to push multiple statistical formulas to present the most granular data out of billions of data values.