Discuss a project

DL-012 · Case study

Data extraction for a global drivetrain supplier

Dana wanted strategic decisions backed by millions of competitor data points. We built 25 custom scrapers and an ETL pipeline that loads standardised data into their cloud databases.

Client
Dana Inc.
Location
USA
Industry
Automotive
Capabilities
03Build
01

Challenge

  • Strategic decisions needed reports built on millions of competitor data points.
  • Competitor sites used complex forms with around 1,000 option permutations.
  • Collection had to avoid access restrictions while respecting robots.txt.
02

Environment

  • One of the world's largest suppliers of integrated drivetrain and propulsion systems.
  • Competitor websites across the automotive market.
  • Internal and cloud databases as the destination for clean data.
03

Intervention

  • Built 25 custom web scrapers for competitor websites.
  • IP rotation and proxies, with per-scraper rate limits following robots.txt.
  • Ten machines running scrapers asynchronously, about 250 crawlers at once.
  • ETL script standardising, aggregating and loading data into cloud databases.
04

Outcome

  • Average competitor prices for specific product types.
  • New features identified in competitors' similar products.
  • Insight into global electric car demand trends, refreshed on demand.
Around 250 crawlers on ten machines, behind rotating proxies, collect competitor product data; an ETL step standardises it into cloud databases that feed pricing, product and market insights.COMPETITOR SITES10+ websitesPROXIESIP rotationSCRAPERS25 / ~250 crawlersETLClean / standardiseCLOUD DBAggregated dataINSIGHTSPrice / product / market
  1. 01PROXIESIP rotationSCRAPERS25 / ~250 crawlers
  2. 02COMPETITOR SITES10+ websitesSCRAPERS25 / ~250 crawlers
  3. 03SCRAPERS25 / ~250 crawlersETLClean / standardise
  4. 04ETLClean / standardiseCLOUD DBAggregated data
  5. 05CLOUD DBAggregated dataINSIGHTSPrice / product / market
Fig. 13Competitor data pipelineAround 250 crawlers on ten machines, behind rotating proxies, collect competitor product data; an ETL step standardises it into cloud databases that feed pricing, product and market insights.Scale NTS

Results, as published.

11M+
Data pieces scraped
10+
Websites
25
Custom web scrapers
~250
Crawlers running simultaneously

Client notes.

Client notes · DL-012Dana Inc.
Even though we don’t see each other on a daily basis, they feel like part of our internal team. They are phenomenal engineers, fully engaged in what we do, and their dedication is evident in every implemented project.
SignedJohn C. MillerPricing Manager, Dana, Inc.

Stack and services.

Stack04

  • Custom web scrapers
  • Proxies / IP rotation
  • ETL pipeline
  • Cloud databases

Services used01

Tell us what has to work.

A senior engineer reads every brief. Describe the system, what it runs on today and what happens when it fails.

Discuss a project