Challenge

The previous pipeline posed challenges in computing revenue across various metrics at a specified level. Add new Business Logics to the current table and excluding deprecated products, locations and unused downstream consumptions.

Approach

Rebuild the Data Pipeline to compile all products , and campaign details for each contract on the most possible granular level by ensuring the query performance & information is easily accessible for downstream tables and dashboards.

Outcome

Reduced runtime of the daily pipeline from 2 hours to 20 minutes by optimizing the tables, eliminating irrelevant functionalities thus enhancing pipeline performance.