Back to Success Stories

Infocepts Helps Retailer Optimize Reporting & Cut Costs with Databricks Serverless

Databricks Flash Migrate Retail & CPG

Infocepts partnered with a global retailer to migrate from Databricks SQL Warehouse Pro clusters to Databricks Serverless, resolving critical performance bottlenecks in Strategy reporting. The solution led to a 40% reduction in compute costs and ensured 100% of reports were generated on time.

The Challenge

A leading retailer faced critical performance issues with its business-critical Strategy report subscriptions, which ran on traditional Databricks SQL Warehouse Pro clusters. The system’s inefficiencies became particularly severe during the holiday season, leading to frequent escalations.

The retailer encountered multiple challenges:

  • Resource overutilization, with usage consistently exceeding 80-90% of allocated capacity, resulting in performance degradation and report failures.
  • Frequent subscription failures or delays, requiring 2-3 manual interventions per week to scale up resources, causing delays in report generation.
  • High startup times of 5-8 minutes for SQL Warehouse Pro clusters, delaying critical reports and triggering 3-4 complaints per week.
  • Cron job failures due to long cluster startup times, with 40-50% of jobs going into a pending state, leading to delays of 1-2 hours for high-priority reports.
  • Rising operational costs due to manual scaling and inefficient resource allocation.

The Solution

To address these challenges, Infocepts helped the retailer migrate to Databricks Serverless SQL Warehouses. This shift enabled automatic scaling based on workload demands, eliminating the need for manual cluster provisioning and significantly enhancing performance, cost efficiency, and operational reliability. The solution involved the following steps:

  • Analysis of 2,000+ Strategy report subscriptions and categorizing them based on business criticality, delivery timelines, and compute requirements.
  • Implementation of two serverless clusters: a large instance for mission-critical reports and a medium instance for less critical workloads.
  • Realigning report subscriptions from existing clusters to the new serverless clusters, ensuring continuity and performance improvements.
  • Deactivation of unused subscriptions to optimize compute resources and reduce system load.
  • End-to-end testing to ensure seamless transition and optimal performance.

Infocepts had previously enabled the retailer’s migration from Teradata to Databricks using its Flash Databricks Migrate solution, setting the foundation for this latest transformation.

The Outcome

The migration to Databricks Serverless delivered substantial improvements:

  • 40% reduction in compute costs by eliminating the need for over-provisioning and only paying for resources when in use.
  • Instantaneous cluster startup times, enabling faster query execution and improved report responsiveness.
  • 100% uptime for SQL warehouse & timely cron job execution, ensuring that reports were generated on time, reducing business disruptions.
  • Eliminated manual resource scaling, reducing operational risks and freeing up IT teams to focus on analytics and insights rather than infrastructure management.
  • Seamless handling of fluctuating workloads, ensuring consistent performance during peak business periods, especially during the holiday season.
Talk to us to learn how we can accelerate your Databricks Serverless Adoption.

About the Client

Our client is a prominent North American retailer with over 600 stores across the U.S. and Canada renowned for its wide range of pop culture merchandise, novelty items, and unique gifts.

Recent Customer Stories

When Launch Decisions Needed Real-Time Clarity
One Audience, Many Currencies: How a Big 3 TV Network Unified Its Audience Measurement
How Pharma Commercial Operations Analytics Linked Marketing to Sales?
One Launch, 10,000 Different Buyers: How Pharma Key Account Profiling Won Them Over
Media & Entertainment
How AdSentinel Helped a Media Organization

We'd Love to Hear from You!

Open Popup