Posts tagged ‘BI’

New Path, Same Focus

I’ve worked with Mariner for almost 12 years. It’s been a very good journey, with many great experiences. I’ve worked with a lot of great people, and delivered some really interesting BI solutions to clients in a number of industries. One aspect of my job that I always particularly enjoyed was helping developers be more productive when creating BI solutions, and reducing the repetitive (read: “boring”) aspects of developing solutions on the Microsoft stack.

Recently, a new opportunity to focus more heavily on that came along. As a result, after a long and enjoyable career with Mariner doing business intelligence consulting, I am taking a new position with Varigence, a company that is producing tools that will make implementing BI solutions faster and easier, as well as introduce new capabilities and better integration into the Microsoft BI stack.

I’m really looking forward to the new role and the new experiences it will offer. I will continue to be heavily involved in Microsoft BI, so I plan to maintain this blog and continue speaking and writing on it as often as often as possible.

PASS Summit 2008

Last post on presentations for a while, I think / hope.

I’ll be presenting on Unit Testing in SSIS at the PASS Summit 2008, occurring in Seattle, WA from Nov. 18-21. There’s lots of great content planned at PASS this year, and it’s definitely worth attending. If you are there, drop by the session and say hello.

The Microsoft BI Conference

I’m doing a couple of chalk talks at the Microsoft BI Conference, occurring in Seattle, WA on October 6-8. The chalk talks are a different format than the typical presentation: smaller audience, no (or very few) slides, and more conversational in format. I’m doing one on unit testing for SSIS, and another on building custom components for SSIS. I really enjoy the chalk talk format, and I’m looking forward to some good conversations. If you are planning on being at the conference, please stop by.

SQL Saturday in Greenville, SC

There’s a SQL Saturday coming up in Greenville, SC on October 11th. It’s a free, day long event with a number of great speakers doing sessions. I don’t include myself in the great speaker category, but I was asked to present a couple of sessions on SSIS, including one on data profiling in the ETL process, one of my favorite topics.

Again, it’s free and there’s some great content, so if you are in the area or close by, you should check it out.

The SSWUG BI Virtual Conference Is Moving…

but not very far, just to Nov. 5-7. In order to provide more value to attendees, the BI conference is being run at the same time as the SharePoint, SQL, and .NET conferences. So now, instead of just having access to the BI content, you’ll be able to pick and choose session from all four conferences. Personally, I love the idea of 3 days of solid BI content, but if you like a little more variety, this should be great.

Check in at to see the updated information.

Iterative Development In BI

One of the staples in agile (and many other methodologies) development is the idea of iterative development. That’s the idea that, rather than doing an entire project in one big pass, it is broken up into smaller iterations. Each iteration ideally consists of a specific set of deliverables with value to the end user (ideally working code), and is a small enough slice of the project that the deliverables can be met in a short time frame*. So in a traditional application development project, the first iteration might deliver a working data entry form to the end user. The second iteration might add an additional form and a background validation process, and so on until all the application requirements are met.

Another feature of iterative development is that each iteration involves some aspects of the full software development lifecycle. For each iteration, you expect to go though some requirements, design, development, and testing. But these activities are focused on just what is required for the current iteration. There usually is a small amount of up front time spent mapping out the entire project, but not in an extreme amount of detail.

Most of the BI projects I have been involved in, however, follow much more of a waterfall approach. There usually are intermediate milestones defined, but it’s usually not working code, it’s a requirement document, or an architecture diagram. What’s really depressing about this is the look of incomprehension I get when I suggest breaking things into smaller iterations.

Here’s some of the common objections I hear to doing iterative development, and my responses to them.

  • “We have to gather all the requirements up front so we can make sure we don’t miss anything.”
    This is one of the most common, and one of the more absurd, objectives. The chances that you can gather all or even most of the requirements up front on a project of any complexity are small. There’s even less chance that the requirements won’t change over the course of the project. I much prefer to do detailed requirements in a just-in-time fashion.
  • “If we don’t know all the requirements at the time we begin design, we might build something that can’t be extended.”
    I agree that designing a BI solution is easier if you have all the pieces in front of you when you begin. However, it is certainly possible to create a flexible design without knowing every detail about how it will be used. In fact, since BI systems encourage emergent behavior, you can count on new requirements arising over the course of the project, and after it is completed. That means a flexible design is necessary, even if you think you have all the requirements up front.
  • “It takes too long to produce working code to deliver iterations that quickly.”
    This is one argument that holds some weight with me. If you are starting from scratch, and you need to build a star schema, load it with data, and create reports on that to deliver to end users, it can be difficult to accomplish that in four to eight weeks. However, that’s where I feel that you may have to narrow the focus of the iteration – instead of building all 8 needed dimensions, you build 4 instead. Then you build the remaining 4 in the next iteration. This does require some creativity to make sure you are building something of use to the end user, while still being able to complete it in the iteration’s timeframe.

That covers some of the common objections. I have done short iterations on BI projects, with a high degree of success, so I know that not only is it possible, it works well. In a future post, I will list some of the benefits I find in doing iterative development on BI projects.

*What is a short time frame? This is somewhat subjective, and you will find opinions ranging from several days to several months. My personal feeling is that an iteration shouldn’t go past 8 weeks, and I think 4 weeks is a much more manageable size.