Failure to Launch: From Big Data to Big Decisions

“Big data is bullshit” – Harper Reed, President Obama Campaign CTO

Not long ago big data was all the rage, the trendiest concept in the world of business intelligence. Now, however, the hype appears to be fading fast, with few companies having realized the bottom-line benefits promised to them by various vendors from the undertaking of big data initiatives.

You can download PDF version of this whitepaper here.

One of the key drivers of this gap between promise and the reality has been the shortage of effective and adequate human resources in the organizations to undertake such efforts.  This vacuum has resulted in minimal actions being taken on the wealth of information big data has made available for the organizations, a core driver behind the failure of tangible impact being realized. Recent research conducted by IDG Enterprise suggests that organizations are facing numerous challenges due to this limited availability of skilled resources, with 65% indicating they are overwhelmed by the influx of data.

The key to making big data initiatives a success lies within making the produced data  more digestible and usable in decision making,  rather than making it just ‘more,’ resulting in  the creation of an environment wherein  information is used to generate real impact.  Put another way, the survival of Big Data is  more about making the right data (not just higher volume) available to the right people (not just higher variety) at the right time (not just higher velocity).

Pro-active Reporting

Pro-active reporting is a next generation business intelligence domain, where, instead of users reviewing standard sets of reports, solutions automatically identify and build custom reports for the users, notifying them of critical trends as well as the root causes behind such trends.

Data Unavailability: Challenger Disaster

January 28, 1986 marks a dark day in space history, with the Space Shuttle Challenger disintegrating 73 seconds into its flight, taking the lives of 7 crew members.

Years after the disaster, The Rogers Commission discovered that the vehicle was not cleared to operate at the low temperatures of that morning, and this fact was not adequately communicated to decision makers on time.

This tragic incident highlights the vital importance of pro-active reporting everywhere, as not always the right data reaches the right decision makers at the right time.

Three main pillars exist within the pro-active reporting realm:

1. An Intelligent Alert Mechanism

‘Intelligent’ triggers, notification messages, and alerts are vital components of the pro-active reporting architecture. In such  architecture, rather than end users setting up  alerts for re-defined thresholds on key  metrics, intelligent agents continuously  analyze them and discover significant changes  automatically, be it foreseen or unforeseen.

The difference between the two approaches  can make or break a business – while in the  traditional reporting approach, a decision  maker would be unknowingly at ease when  the bottom line has not changed much, the  decision maker following the pro-active  approach would know that although the  bottom line is stable, it is, for example, losing  significant share in a specific market segment  (temporarily balanced by an increase in share  in another) and address this risk.

2. An Apt for Causal Intelligence

Another key characteristic of the pro-active reporting approach is the heavy focus on change, and the root causes behind the change. Instead of providing a traditional look at snapshot figures and charts, pro-active reports highlight delta in time, and analyze key contributors to this change. The output is usually a chain of mathematical explanations of changes – e.g. “our sales decreased by 3% this month. 90% of the decrease is  attributable to the sales decrease in the youth  segment for product A in region 1, which in  turn was triggered due to the 10% decrease in  regional advertising spend.”

Apt for Causal Intelligence is not only a matter of reporting, it is a philosophy that requires the discipline to gather and analyze data on all possible root causes of change.

3. A Readiness for What-ifs

The third important feature of the pro-active reporting approach is the focus on analyzing the impact of alternative business scenarios on the outcomes. Such scenarios can answer questions ranging from – “What if we had increased our marketing spend by 10%,” to “What if we had closed our store in downtown Manhattan,” along with sensitivity analysis on potential outcomes.

The what-if analyses should follow the same principles as causal intelligence, correlating various factors with each other, based on historical learnings.

Impact on Decision Making

Utilization of the pro-active reporting approach provides three main benefits to an enterprise:

1. Speed of Decisions

An average analysis request for a question as simple as “why did our sales drop?” can take from hours to weeks in most enterprises to answer. In addition, the question is often asked too late, after the sales has dropped and has continued to do so.

Pro-active reporting picks up the early signs of such gaps, and makes them known immediately to the right people, along with the possible root causes. This, not only decreases time to uncover the truth from weeks to seconds, but also makes it possible to address possible risks and opportunities long before it is too late.

Late Reaction: Flight 370

One of the most mysterious events in recent air travel history is the disappearance of Malaysia Airlines Flight 370. On March 8, 2014, the plane disappeared from civilian radar, with its 227 passengers and 12 crew.

One of the most striking findings about the event was the fact that air trac controllers did not realize that the plane was missing until 17 minutes after the fact.

This case highlights the gap between speed of data collection and their interpretation by people even in most drastic events. Pro-active reporting minimizes the need for such interpretation and avoids such delays.

2. Accuracy of Decisions

The decision-making process in companies today is less than ideal in many cases, even in those that can be considered best-in-class, for a number of reasons:

The fatigue of analysts looking at same numbers over and over again results in incorrect information being reported

The lack of time to perform comprehensive analysis possibly results in key factors being overlooked

The lack of capabilities in analysts to truly uncover root causes

The interpretation of data with a bias due to various internal pressures resulting in misleading information being reported

Pro-active reporting solutions ensure the  above factors that drive accuracy-related  issues are bypassed, and done so in a rapid  and efficient manner, analyzing all key  measures as well as possible scenarios, all  while ensuring zero bias is in play.

Human Oversight: The BP Oil Spill

April 20, 2010 marks one of the black pages in marine history, with 4.9 million barrels of oil spilled into the Gulf of Mexico, claiming 11 lives.

When the presidential committee reported its findings, one thing was clear – timely recognition of the risks and first signs were inadequate.

A test of the seal that resulted in the blow-out was incorrectly assessed to be in proper working order by a technical team, and the workers failed to recognize the first signs of the impending blow-out.

The committee writes “BP did not have adequate controls in place to ensure that key decisions…were safe or sound.”

3. Cost of Decisions

Big Data analysts and scientists are some of the scarcest resources in most enterprises.  Using pro-active reporting platforms frees up the valuable time of such resources, making sure that they can be used in value-added, more complex tasks that cannot be automated or done by technology in a more efficient and accurate manner.

Risk Detection: Financial Melt-Down

The 2008-2009 global financial crisis was one of the worst in financial markets history. Even the Federal Reserve was late to realize the risk, with one of the Fed’s senior forecasters recorded saying at the time:  “we’re still expecting a very gradual pickup in GDP growth over the next year.”

The relations between trends in volumes & risks of sub-prime loans, asset-backed securities, and credit default swaps and what it meant for the market went undetected by most, with reliance instead on agency ratings to identify sound investments.

As the variety and complexity of indicators to follow increases, it becomes humanly impossible to track them over time, which is better left to systems.

Pro-active Reporting Use Cases

Application areas of pro-active reporting are as wide as regular reporting – it can be used anytime where a decision needs to be made.

Most importantly, pro-active reporting facilitates decisions that need to pick up early trends or understand the dynamics of change.

Some of the most common use cases across industries are as follows:

Corporate Performance Management

The most common use of pro-active reporting is around performance management, bringing in more intelligence to traditional scorecards, automatically linking and reporting the changes in key performance indicators across an organization – ranging from financial to customer, organization, and process indicators.

Channel Management

With thousands of stores, partners, and employees, large B2C organizations need to monitor sales and service quality / quantity trends. Pro-active reporting facilitates automated tracking of fluctuations at the most micro level, and can provide comprehensive feedback on what to fix and where to fix it – e.g. sales in our store A for product 1 has decreased 5%, because of a new competitor store in the area and out-of-stock issues this month.

Customer Relations Management

From key account management to customer value management, pro-active reporting  answers various questions, such as which  macro customer segments have been  churning recently, which micro segments are  becoming best targets for cross-sales, which  specific corporate accounts are in early stages  of a downward trend, etc.

Product Portfolio Management

Especially in sectors where product variety is high – such as retail and e-commerce -monitoring sales performance of each and every product closely becomes an almost impossible task. With pro-active reporting, not  only is all kept under control, but also the  explanation of sales fluctuations for each  product is available – e.g. Product A sales  dropped 10% because our youth segment  started purchasing less in region 1, with  increased number of returns and complaints.

Service Level Management

Although timeliness and quality of processes are regularly monitored for service level assessment in most leading organizations, very few identify the root cause behind fluctuations automatically. With pro-active  reporting, all is identified – from the list of  steps causing delays in a specific time frame,  specific employees causing the delays, types  of applications (e.g. with incomplete  documents) causing the delays, etc. In other terms, all findings that are usually obtained only during process redesign efforts become a daily routine with pro-active reporting.

Getting Started

Pro-active reporting solution implementations follow a cyclical and continuous process, as follows:

Step 1. Define the list of KPIs to follow. This list of KPIs depends on the application area and business priorities, and can range from a list of 3-4 indicators to hundreds in complexity.

Step 2. Define the possible relations between the KPIs. To be defined by the business (with people who work in and around the KPIs as part of their day to day work), the linkages various KPIs have with each other have to be explored and determined. Three main types of causality relations may be defined:

I – Selective Relations: These relations explain the interaction between KPIs and their sub-sets across various dimensions (such as breakdown of customer acquisition across market segments).

II – Calculative Relations: These relations represent the mathematical interactions between different KPIs (such as revenue being related to the number of active customers and revenue per active customer).

III – Deductive Relations: These relations explain the interaction between KPIs and direct / indirect factors correlated to these KPIs (such as sales being related to marketing spend).

These first 2 steps require detailed  brainstorming and discovery sessions, usually involving most (if not all)  organizational units, as relations and  dependencies usually span across the whole value chain (e.g. supply chain  inventory affecting sales as much as  marketing).

Step 3. Build up KPI and relation data.  Pro-active reporting mind-set usually brings in additional data requirements to an organization, especially regarding factors affecting each KPI. Building a complete pro-active reporting platform requires the collection of more data, at a finer granularity. Establishing deductive relations on the other hand, usually require detailed regression analysis, in order to quantify the level of correlation in between.

Step 4. Establish the notification system. Once everything is in place, the next thing is to define which alerts should be delivered to whom within the organization, based on which analysis and with what frequency (e.g. daily, weekly, monthly and even hourly, if used for operational purposes).

Repeat. Finally, as with all systems, pro-active reporting should be monitored and fine-tuned over time, based on changing business priorities, market conditions and learnings.

About CIWare

CIWare stands for Causal Intelligence and is a pro-active reporting solution developed by Forte Wares.  Designed with the adaptability to suit various business problems, CIWare enables businesses to make faster and more precise decisions. Automating the whole discovery process, CIWare also saves valuable time and resources, and eliminates the human-error risk in business analysis.

Unlike traditional BI solutions, CIWare focuses on change in the business – rather than looking at a snapshot of the business indicators, it analyzes their change over time, as well as contribution of various factors to this change.

For more information please visit or email us at


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s