I’m happy to be attending the TDWI Agile BI conference in San Diego this week. Today, Sunday, kicks off a series of great learning sessions. I chose to start the conference by attending Ralph Hughes’s Agile DW 101 all day session. In this session Mr. Hughes starts out describing the very basics and moves though a number of topics that focus on how to make agile work for data warehousing projects. Here are some of my most significant takeaways.
‘What’s the point of agile if we can’t modify [assets] after moving to production?’This short quote says so much. Traditional data integration development methods typically includes many tightly coupled dependencies. This is the nature of data integration. We are completing a series of tasks each depending on the prior task’s output. Making a change in such a process often requires downstream logic to be re-developed which leads to very costly waist. In my opinion, trying to take an agile approach to data warehousing while using traditional development methods (custom scripting or generic ETL tools) is a recipe for failure. You may have limited success in the first few iterations, but eventually you will spend more time doing rework than new work.
‘Things like coding an SCD2 dimension follow a pattern… Let machines handle that stuff & focus on things that require human intervention.”Of course! The concept of working at the highest level possible has driven innovation and progress in all areas of human existence. We don’t cut down a tree, mill it, dry it, etc… when we want to build a house. Instead we use dimensional lumber or modular components which makes the process so much more efficient. There is no need to build a car from raw materials. Doing so would be more costly and would result in lower quality than simply buying a car. The same is true for data warehousing. There are well established design patterns and best practices for nearly all development tasks. It is so inefficient and risky to develop data integration processes at a low level. Let’s use computers for what they are good at, following rules, and allow human resources to focus on critical thinking tasks.
‘How do you make sure that the next person [developer] is going to be able to maintain that code?’This question was asked by a fellow session attendee, and I think she made an important point. Again, if we are going to work in an agile manner we can’t be spending valuable units of work fixing prior iteration bugs. We also can’t spend valuable time trying to figure out complex custom code when our star developer decides to accept a new opportunity. We can go a long way toward preventing these situations by using data warehousing productivity tools to guide the development process using proven industry standard best practices. Doing so leverages the knowledge & experience of a much larger pool while hedging against common risks associated with cowboy coding. Thank you TDWI & Ralph Hughes for a full day of agile data warehousing content. Good stuff!
