Agile Data Analytics and Business Intelligence

Twenty-First Century organizations rely on data. It is the fuel that powers continuous improvement and success. Still most organizations struggle when it comes to harnessing their data to deliver actionable intelligence leading to competitive advantage.

In this white paper, we will discuss how to apply an Agile approach to deliver successful, cost-effective Data Analytics and Business intelligence (BI) solutions.

Winning Formula

Being successful with Data Analytics and BI requires a holistic approach to understanding how data and analytics is used in your organization and how to transform that into actionable intelligence that is used by the business units to drive performance.

We can think of Data Analytics and BI as a formula with five dimensions, as seen in the following diagram.

These five dimensions can be seen as layers in a Data Analytics and BI ecosystem. Each dimension feeds into and enables the next layer above it. At the bottom layer is the business processes that capture, process and store information. At the top layer are the goals producing success for the organization. In the middle are data, tools and business capabilities.

Being successful using data to drive business performance requires a holistic approach that includes all five dimensions.

Data Analytics & BI Roadmap

An important first step is to create a Data Analytics and BI roadmap.

The roadmapping project should include three key elements:

  1. A current state assessment
  2. Interviews with key stakeholders to determine current and future needs for data and analytics
  3. A future state roadmap

The current state assessment will define your existing Data Analytics and BI efforts. It is important to maintain focus on the entire data ecosystem. Both roadblocks and opportunities can be uncovered at any of the five dimensions.

The next part is to identify key stakeholders involved in the Data Analytics and BI at your organization and get their honest and candid feedback. Key stakeholders typically include: executive team, business unit managers and the analysts who actually work with the data from day-to-day. Each of these stakeholder groups will have their own goals and challenges.

Finally, a roadmap will tell how to get from here to there.

The Data Doctor Is In

The Current State Assessment phase can be seen as a diagnostic of your existing solutions. It is very common for an organization to face challenges in particular areas within the ecosystem that can be addressed to gain quick wins.

Without going through the diagnostic phase of a Current State Assessment, it is easy to reach the conclusion that, “Our whole system is broken. We need to scrap the whole thing and start over.” Another dysfunctional conclusion is the resignation that, “We have a solution that benefits one or two departments. Everyone else will just have to make due.” The status quo, all too often is frustrated analysts struggling to pull the data they need into Excel and wasting hours on end manually manipulating data to provide management with the reports they need.

Neither of these conclusions makes anyone happy. If we asked our stakeholders to communicate in emoji language, there may not be many smiley faces. We might be more likely to see pile of turds.

The Actions listed on the right side of our Ecosystem diagram shows some typical Data Analytics and BI tasks performed in an effective organization. They are also typical actions needed to get an ineffective program back on track.

Two brief examples demonstrate the value of diagnostics leading to quick wins:


Unreliable data requires analysts to manually pull data from source systems and compile reports in complex Excel spreadsheets


A Data Quality assessment uncovers a Business Process improvement in how data is entered into the source systems. Staff is re-trained in the new process providing accurate data going forward.


Data is not available across business units resulting in each being reported in separate silos


A Data Modeling project maps data for each business unit, from source to consolidated reporting. A consolidated dashboard is created in the BI tool avoiding costly re-engineering of the database and enterprise reporting system.

Agile Approach

An Agile approach to Data Analytics and Business Intelligence starts with the Current State Assessment and Roadmap. The big picture view is divided up into pieces that can stand alone as quick-win solutions. The quick-win solutions are then aligned in the roadmap to add up to an overall solution. In this way, business users get usable deliverables while the larger needs of the enterprise are being met as the individual pieces are fit together into a whole.

A next key step is business process modeling and data modeling. This is the beginning of the actual solution phase. This is where we figure out how all the pieces fit together into a solution that meets our needs.

As long as the initial Assessment, Roadmap and Data and Business Process Modeling stages are done well, the pieces will fit together as they are rolled out according to the roadmap. Additionally, each part will be improved along the way by real-world use.

A fear sometimes expressed is that, “We will have to re-engineer the partial components to fit into the larger solution.” In an Agile world, this statement is turned on its head. This fear is addressed through due diligence in the roadmapping and modeling phases as we’ve shown. On the other hand, an Agile approach allows for re-engineering of the component parts based on real-world feedback. This ensures that business users at the end of the rainbow get something that works for them. It meets their needs, even if those needs have changed during the project.

In a traditional waterfall approach, the business users often will not find out if what is being built will actually meet their needs until it is too late to change it. Then they are stuck with what they get.

Agile Applied to the Enterprise

There are cases where the right answer is to build an entire enterprise-wide Data Analytics and BI solution as a single project. Some conditions that may require an enterprise solution include:

  • Sensitive financial information is subject to regulatory requirements for auditability and data governance
  • Sensitive customer information (PII) must be protected through controls built into the system
  • Organizationally expedient to outsource to a large firm specializing in enterprise solutions

However, even in these cases, the enterprise approach often leaves business users waiting months or years for basic reporting. Often when the solution finally is completed, key business users are still left not able to get the reporting and analysis they need.

The Agile approach can be applied to Enterprise Data Analytics and BI projects to ensure business users have their needs met while the larger project is being developed.

In some cases, an Enterprise approach must be taken to lay the foundation. For example, auditable data governance and security measures may be required when integrating data into an analytical data warehouse. An Agile approach can be built into the later phases by empowering business units with flexibility in the reporting and analysis tools sitting on top of the data foundation.

But what happens when even that first part takes a year or longer? Business users are still left frustrated. The Agile approach can come to the rescue. Cost-effective, interim solutions can be developed for the business units in parallel to the larger solution. The Assessment, Roadmap and Modeling phases will identify quick-win projects that can be delivered which still support the larger requirements for an Enterprise-level solution. The befits of an Agile approach are still realized. Business users get tools they need in a timely manner and feedback from live use of the tools in earlier phases can feedback into an improved solution in the later phases.

An Agile approach can deliver urgently needed analytical solutions to business units in a timely and cost-effective manner, even if a large-scale enterprise solution is being developed at the same time.

The following two use cases will demonstrate how an Agile Data Analytics and BI approach works.

Use Case #1
Unified Reporting across Finance, Operations and Marketing

A goal at many companies is to integrate financial data with other company metrics. For example, sales pipeline or product information. But often the GL hierarchies are not aligned with customer segments, sales territories or product taxonomy. Furthermore, GL hierarchies typically do not map cleanly into BI drill-down paths.

We start with a Data Modeling and Roadmap phase. Data from each of the business silos is modeled to define common dimensions and hierarchies that can be created to map data silos to one another. A Data Analytics and BI Roadmap is created defining how each silo will be addressed independently and how they will be integrated.

Use Case #2
Measuring the Impact of Online Media on Offline Sales

How do we measure the impact of online media spend on offline sales? This is a question being asked by many organizations. It is not an easy question to answer. Especially when it comes to social media, the goal is not just to drive clicks but to generate a rising tide of positive awareness that translates into increased sales.

The impact is indirect. So how do we measure it? The first step is to align measurement of online media performance with offline sales performance. Then measure the correlation between spikes in media performance to spikes (or lack of) in offline sales. This can be very complex with machine learning algorithms or it can be as simple as plotting the two trends together on the same graph.

We start with an Assessment, Data Modeling and Roadmap phase. The goal is to report each digital media channel and offline sales by common dimensions: Data/Time, Product and Geography. This will involve a Data Modeling project. It will also involve a Business Process Assessment to ensure all digital media are coded consistently with product identifiers.

Concluding Thoughts

An Agile approach to Data Analytics and BI can be summed up in the following five points:

  • Current state assessment and roadmap tailored to the goals and realities of the organization
  • Data and business process modeling
  • Incremental deliverables each completed in a timely manner and delivering value
  • Agile “sprints” delivering quick wins to specific business users that are aligned with the overall roadmap
  • User experience from live deliverables feeding into improved future deliverables

There is no one-size-fits-all approach to Data Analytics and Business Intelligence. Every solution is custom because every organization has its own challenges and goals. By following an Agile approach, customized solutions can be delivered in a timely and cost-effective manner that delivers value to business users at each step in the process.