Tuesday, February 13, 2007

How release management could have gone better

One of the challanges I find in bringing a new application online is having the new application support compared with the old applicaiton support. We may be replacing a 20 year old system where the Service Desk has no visibility into what the AS/400 is actually doing, but they have had 20 years of experience to figure out things by intuition at this point. In contrast I'm bringing online a brand new shiny (shiny is key for executive buy in) high performance monster with over 600 CIs (Configuration Items). This new monster uses the latest and greatest development methods, methodology, and we have performance tested the heck out of it. Through observing users activity during testing we have determined that they will be able to accomplish their tasks twice as fast as on the previous system. In fact there is such a buzz around the product that instead of resisting the change, people are beginning to stop IT people and ask them when this software will be available for them to use. The system rolls out and we have one of those mid-morning parties with pastries and congratulate the project team.

Meanwhile the operations folks are scrambling to figure out why some users are experiencing slow response times, and others can't access the application at all. The problems seem remote and not even connected, then the APB comes out, all IT staff must put down thier pastries and go to the big conference room to figure out why this rollout is going so poorly. Apparently there is widespread panic among the user base that we shoudl rollback to the old system and spend another few years until this system is right. We scramble for a few days and detemrine that there were four users affected. One was a VP who had installed something on his machine that was conflicting with the new software. One was a user who's hard drive was full of family photos and MP3s and the software didn't fully install properly. Two users had reset their passwords earlier that day and needed to reboot their computer. It was some cached credentials thing with workstations and Active Directory.

As it turns out the problems were all very managable, but the application suffered some significant loss of face due to circumstances we never tried in test. Because the release plan didn't have a solid Service Desk behind it with a single point of contact for users with issues, we were unable to bring in calls in a uniform manner and deal with them in a meaningful way.

Bottom line here is that if we had a CMDB with auto discovery we would have know that the VP had something on his desktop that conflicted with the application. We would have known about the hard drive running out of space. Then we could work on the reboot your machine and all will be fine issues. Not only that but we would look like we know what we're doing and provide customers with increased uptime.

No comments: