How Data Engineering Transformations Boosted Analytics Performance by 2700%
01. Overview
The DataBridge Team successfully reduced the stack of technologies and significantly improved the performance of report generation.
3
AWS, Snowflake Computing, Fivetran, and MicroStrategy.
Performance Increase 2700%
02. Background
Our client is the largest independent native advertising platform, trusted by major global publishers such as Time Inc., Wenner Media, Scripps Networks Interactive, Thought Catalog, Warner Bros., and USA Today Sports. Their platform delivers over three billion native ad impressions monthly, seamlessly integrating with editorial content to maximize engagement.
A core aspect of their business is understanding user behavior and audience interaction with their ads. Their analytics focus on website performance, content rendering, and continuous improvements in user experience.
03. Problem
The client had outgrown their existing data analytics environment, which relied on a custom-built MySQL-based system. This system lacked a semantic layer, had poor performance, and was reaching its row capacity limit. Retrieving data for analysis had become a slow, manual process, leading to wasted resources on inefficient tasks.
04. Challenge
The client lacked in-house expertise to carry out the necessary improvements to their analytics pipeline. Hiring a full-time specialist was not a viable option, so they sought a partner who could redesign their data infrastructure to enhance performance and scalability.
05. The Approach
The DataBridge Team transitioned the client’s analytics stack from 12 different technologies to just four: AWS, Snowflake Computing, Fivetran, and MicroStrategy.
- Migration to Snowflake: The client eliminated MySQL dependency and adopted Snowflake as their core data warehouse. Snowflake allowed direct access to raw JSON data streams and facilitated seamless ingestion of logs from Amazon S3 without preprocessing.
- Fivetran for Data Integration: Fivetran improved data extraction and loading, making it easy to transfer information from multiple sources, including SaaS applications, marketing platforms, and Salesforce, into Snowflake. This enabled rapid transformation into analyst-ready views.
- Optimized Data Processing: With Snowflake, raw JSON logs could be processed efficiently, eliminating the need for Spark. Now, four different data sources are synchronized within two hours, handling 40 terabytes of mixed data while reducing storage costs.
06. Outcome
The client expressed high satisfaction with the improved performance, particularly for complex tasks involving granular data, aggregation, deduplication, and frequency-level analysis.
- Massive Performance Boost: Queries that previously took an entire weekend to run now complete in just 30 seconds with only four lines of SQL.
- Enhanced Data Processing Efficiency: Reports that previously required 45 minutes in MySQL now execute in one second in Snowflake.
- Cost and Resource Savings: Eliminating unnecessary technologies reduced operational costs and streamlined data management.
