Over the last couple of years Agile BI has been talked about quite a bit, and I wanted to throw my hat in to the debate.
There are growing numbers of people to whom an Agile approach appeals; people who have seen software projects take too long, delivering things the customer doesn’t want, at too much cost. People try it and they can see the benefits immediately: users and the business are involved every step of the way, and projects make real progress.
On the other hand, ‘Agile’ can be quite polarizing for some. There are those who baulk at the fervent views of some Agile practitioners, and experience tells them that although Agile techniques may include many useful ideas, it isn’t necessarily the panacea it is sold as. Others are more comfortable with traditional approaches to software, and are unsure that Agile can deliver quality systems. Some apply Agile principles without real thought, and end up failing to deliver value to the customer, and as a result, give Agile a bad name.
My view has been that success is driven by more than just methodologies- good people are equally important, if not more so. So I approached the concept of Agile BI with a healthy degree of caution…
What is Agile BI?
The proponents of Agile BI take some of the Agile principles and apply them to BI projects. In many cases this will include adoption of the techniques applicable to general software development, such as 2 week sprints and standup meetings, but in other cases it has been used more as branding to highlight the low time-to-value of a particular approach.
What are people’s concerns?
- BI projects tend to be iterative, embrace change, involve the business and users, and deliver increasing amounts of useful functionality over time. Experienced BI practitioners know that their projects are different to other software projects they may have worked on. In BI, the waterfall approach is rarely used unless part of a larger, wider project. So is the application of the term ‘Agile’ purely for marketing? Everyone would prefer to be seen as agile rather than rigid or inflexible!
- Are Agile techniques a good fit for BI projects? Despite being iterative, some of the challenges in producing good BI take time- longer than can fit in a 2 week sprint. Is Agile being used as an excuse for providing things faster, with less quality and reduced thought?
- Once you have reports in place with people relying on them, it becomes more costly to change them. Agile methods sometimes have you avoid considerations of future requirements during the current sprint, and you are encouraged to embrace changes and refactoring in the next sprint. This avoidance of ‘big design up front’ could lead to costly and confusing changes for users of reports.
Agile BI in practice
On most BI projects we have worked on, we recognise many of the underlying ideas of Agile are the best fit. We broadly follow a Kimball approach, which has inherent iteration, and furthermore we recognise that requirements change as we go along, and we have to embrace that.
We often produce a prototype or proof of concept that is intended to demonstrate the use of specific technologies, but also to bring users into the discussion about what it is they want to see, enthusing them about what could be achieved. During requirements gathering, areas of data to be reported on are prioritised, and thereafter several short phases are planned which are often longer than 2 weeks, but short enough to ensure that progressively more functionality is delivered and real business value is gained early on. If priorities change, or if requirements change, that is fine; we don’t rigidly deliver each phase in isolation.
I think this is typical of most approaches to BI, and has some agility.
Learning from Agile- applying the practices and methodology
Given that BI projects can be categorised as an agile approach without the name Agile necessarily being applied, is there anything to be learned from Agile (with a capital A)?
Definitely! BI projects will benefit from more thought on methodology that is to be applied, rather than considering themselves as a different type of project that is somehow exempt from setting down processes and methods.
By carefully examining the principles of Agile and more importantly, the techniques and practices used on Agile projects, we can apply the best of them to our BI projects and benefit from similar benefits enjoyed on Agile software projects. We need to pick and choose the Agile practices that can help, whilst avoiding those that don’t fit with our situation.
A degree of caution can be healthy for many things labelled as Agile with a capital A, but much can be learned from the Agile movement. BI is naturally business focussed and iterative, and by bringing in the best practices from Agile software projects we can improve our processes and effectiveness.
A footnote- true Agile BI
True, fully-fledged Agile can work on some types of BI projects, such as a short BI engagement that is fixed in duration. This kicks off with a short, fixed period of understanding the business, followed by a small number of 2 week sprints. Each sprint provides a prioritised set of data. Crucially to the success of this approach, no reports are developed on the data. In effect you are interactively designing a data warehouse model with the user in these sprints, surfacing the data in a tool such as Excel or PowerPivot so it can be demonstrated and discussed with the business users. The Agile approach works extremely well for this scope of project. However, the real work in producing the reports still needs to be done.