Agile software development has become the preferred methodology of countless teams worldwide. The principles behind the agile manifesto serve as the rule book for those of us who claim to be part of this paradigm shift. A quick read through these 12 principles will make it very obvious that agile is about much more than tools. Agile is about developing a product in the most efficient way possible while also considering that human beings are involved in the process.
Agile principles apply to data warehousing just as well as application development. There really is no difference in the rule book. Data warehousing, however, does have some domain specific challenges that impact how agile a delivery team can be. Below are the minimum thresholds that I believe must be met for a team to consider themselves fully compliant with agile principles.
Rapid Delivery = User story to DW in 3 weeks
“Building a data warehouse” is not a phrase typically used to define the word rapid. Data warehousing is often complex work requiring a variety of skillsets traditionally resulting in lengthy projects. Slow data warehousing is not effective in today’s rapidly changing environment. At a minimum, an agile data warehousing team should be able to deliver a single user story into production in 3 weeks or less.
Responsive to Change = Loosely coupled dependents & little to no rework
Change as it relates to data warehousing can mean many things. We may need to change how an attribute’s history is tracked from SCD1 to SCD2. A new source system may need to load an existing dimension. There are countless opportunities for change. Agile data warehousing teams must be able to respond to routine change request without needing to rebuild what often amounts to many dependents. These dependents must be loosely coupled allowing one component to change with minimal domino effect.
Business Involvement = Communicate with sponsors (typically not IT) 2 or more times per week
Data warehouses are built to support business processes and decision making. Requirements are best described by the sponsor directly. During each agile iteration requirements must be collected, sourcing & modeling decisions made, data must be validated, and user testing completed. It is critical that business sponsors be involved throughout each iteration. Communication should not require the non-technical business sponsor to learn data warehousing lingo.
Continuous Improvement = Team reflects on good & bad at completion of each iteration
One of the most effective ways to learn is by making mistakes, but that is only if you realize you made a mistake and plan to change. Agile teams must take time to reflect on what went well and what did not go so well. This simple routine is imperative to ensuring steady improvement is made.
There are many techniques that make achieving these goals possible. The right skillsets and resources (including tools) can transform traditional data warehousing processes into something that would please the seventeen agile manifesto authors.
