Data Scraping Services for UK-based Client

Technology Stack

Python
Django
Beautiful Soup
Scrapy
Celery
PostgreSQL
Selenium

The Client

Established in 2000 and headquartered in Chester, UK, RealBuzz is an online publishing company known for creating content-rich social networking sites, online communities, and race registration systems. Since its inception, it has forged partnerships with the biggest global events, marathons, and fundraising programs, and reached millions of users worldwide.

The Challenge

RealBuzz wanted to enhance its website by adding two search options, namely:

Race Finder - This would help users search for global summer events like marathons, races, trial runs, cycling, swimming competitions, etc.
Find My Nearest - This would help users look for local gyms, physiotherapists, chiropractors, and podiatrists, in and around the UK.

To accomplish these tasks, RealBuzz sought Capital Numbers’s help.

We got our technology experts together and engaged in a few meetings with the RealBuzz team to discuss the scope.

Our goal was to extract relevant data points to create high-quality searches that would add value to the RealBuzz website, enhance the user journey, and increase conversions. Our biggest challenges were to:

Understand the target customers (e.g., runners, cyclists, swimmers, etc.)
Scrape huge data volumes from the web (e.g., half marathons, full marathons, swims, local gyms, physios, etc.)
Filter key insights from the enormous data pool
Modify the data to improve engagement

This apart, we also had to accomplish these tasks and deliver the project within two weeks, in these COVID times.

The Solution

To address our client’s needs, we chose a tech stack that would create smart searches and produce the best results.

For starters, our dev team chose the high-level programming language Python. Not only did we select Python for its code readability, but also because of its extensive set of libraries that adds to its extensibility. Plus, it comes with third-party modules, built-in data structures, and process control capabilities that improve the overall productivity.

Next, to speed up the development process, our experts used Django. It comes with features that help with user authentication, content management, and traffic handling. Django also has a robust ORM layer that simplifies data management and boosts processes. Plus, it keeps security issues like clickjacking and cross-site scripting at bay. Therefore, we went ahead with Django.

Since this project required extensive data scraping, we used Beautiful Soup. This tool helps pull data out of HTML and XML files, as per the requirements. It helps parse some aspects of web pages without the hassle and makes the entire process of fetching web data a cakewalk.

To add efficiency to the data collection process, we chose the Scrapy framework. This web crawler scrapes thousands of web pages in a simple, fast, and extensible way. We also integrated Celery into this Django project to help create and queue multiple tasks seamlessly in the background.

As for the database management system (DBMS), PostgreSQL was our pick. This highly powerful DBMS helps protect data integrity, build fault-tolerant environments, and manage data, no matter how big or small the dataset is. Moreover, it supports audio, video, and image storage. It even comes with full encryption, which makes it a highly secure DBMS.

To automate the testing process and reduce development time, we picked Selenium. This tool helps find bugs at early stages, makes test scripts reusable, enables testing in volumes, and minimizes human intervention. Besides, it supports cross-browser and cross-device testing. Naturally, Selenium was a preferred choice.

Results

By using this tech stack, Capital Numbers’ could successfully scrape actionable web data to expedite growth for RealBuzz. We deftly extracted authentic data from limitless sources to help our client improve their on-site search capabilities.

Our tech talents offered unparalleled services in areas like:

Find My Nearest

We successfully added the ‘Find My Nearest’ search option in the RealBuzz website to help users get information about the physios, gyms, chiropractors, and podiatrists, based in and around the UK.

To develop this search functionality, our team conducted in-depth research and created an authentic database by extracting hundreds of UK-based postal codes at high speed with smart tools.

Race Finder

We efficiently incorporated the ‘Race Finder’ search feature to help athletes/runners look for races, trial runs, marathons, cycling competitions, etc., held in the UK, the USA, Asia, Australia, and various European countries. We used Google APIs to gather data about the start and endpoints of events/races.

Massive Dataset Management

Our experts collected and managed data from 1000+ sources in a cost-effective way while ensuring absolute control, 100% accuracy, and 0% risk.

Accurate location-based Searches

Our team gathered, filtered, and optimized datasets to create valuable local searches that would help users find services close by, and drive local leads for the client.

ETL (Extract, Transform and Load) Process

Our specialists did an excellent job of synthesizing data from multiple DBMS through the ETL process for advanced profiling, analysis, and decision-making.

Solution Integration Under a Single Admin Panel

We integrated the ‘Find My Nearest’ and ‘Race Finder’ searches into the existing client site to help RealBuzz get the maximum out of a single admin panel, in terms of engagement and conversions.

False Positives and False Negatives

Our data professionals executed tests for false positives and false negatives in a skillful way to create error-free databases, with zero data duplication and repetition.

Search Algorithms

We carefully engineered advanced search algorithms that would create unique searches, trigger quick results, and cater to all user queries with perfection.

Database Indexing

Since we had to pull out data from huge tables, we leveraged the power of database indexing to tap onto only the most relevant insights, eliminate the rest, and produce accurate results.

On-time Deployment

By following a structured project planning process, Capital Numbers could deploy best-in-class solutions for RealBuzz, within a turnaround time of two weeks, despite the current COVID crisis.

Capital Numbers’ Technical Expertise

We took a systematic approach, focusing first on understanding our client journey and then leveraging our technical expertise to source meaningful data that perfectly meet the client’s scraping needs.

Capital Numbers’ fully-managed, enterprise-grade data scraping services have resulted in high client satisfaction and proved its ability to offer refined solutions in niche segments, yet again.