Sample Case Study

Automated Data Collection and Change Monitoring


Partner: A provider of market intelligence, data, and technology solutions for global capital markets, serving over tens of thousands of clients and generating billions of annual revenue.

Employs hundreds of analysts to perform manual research and data entry, spanning multiple industries with hundreds of thousands of websites and data points.

While thorough, this manual research effort was expensive, monotonous, rate-limited to the productivity of each employee, and subject to human error.

Typical automation efforts could not make a big enough impact considering the variety of websites, data points, and standardizations required.


A suite of generic and flexible web crawling solutions capable of handling a wide variety of web and document formats.

Each crawler had one key function (ex. scraping employee profiles from team pages or checking websites for new documents posted) but were built to handle thousands of websites with differing patterns and varieties.

A point and shoot process for the remainder of websites that couldn’t be captured by generic crawlers. This system allows the research teams to “extract once, monitor forever”, without requiring any developer input or technical training, and can scale limitlessly with new projects and additional websites.

dark circle

Real-Time Monitoring

Data point monitoring frequency increased from monthly or quarterly to daily, and even more frequently for mission critical data.

dark circle

Focus On Value

Entire teams were able to be repurposed to higher-level work and act on real-time changes.

dark circle

Customer Satisfaction

Client complaints regarding incorrect or outdated data dropped to nearly zero.

dark circle

Cost Reduction

Ability to reduce or eliminate large contracts with external data vendors.

Service Highlights

Crawl websites to collect data on organizations, products, prospects, people, events, news, and more.

Build datasets on the fly with up-to-date information while monitoring web pages for any relevant changes.

With years of experience crawling thousands of websites, we’ve pushed the envelope beyond “site-specific” crawlers to intelligent and flexible crawlers, capable of combing through diverse websites without the need for site-by-site logic.

Our crawlers are particularly good at handling websites that are notoriously difficult due to CAPTCHA’s, logins, anti-crawling measures, and other limitations.


What Clients Say