Documenting architecture

How do we document the decisions we use to flush out our architecture?

One of my goals is to be able to leave my current employer and have my work continued by those that follow. I do not intend for the work to be carried out by fiat, but that the principles used to construct our enterprise architecture be understood, challenged, and when enough new information is learned, the architecture evolves.

In our current architecture we have three main platforms and I’m about to introduce a fourth. The first platform is our AS/400 system followed by our .NET 1.1 environment, our .NET 2.0 environment and our new 2008 environment.

Our AS/400 environment was the hub of our universe. It had just about every application we needed as an organization, but as we started to pull pieces out of the system, the system began to lose its position as the hub. We pulled out six major functions and we decided to pull out the last four functions and move to a Windows based infrastructure.

Our .NET 1.1 environment was grounded in quicksand with an enormously expensive, a classic clunky web application. The rumor was that it cost around one million dollars per web page developed, has a backlog of defects, and keeps 2-3 people busy full time with workarounds and fixing data that the application either can’t access or it just corrupts outright. The good news is that we were able to move the largest remaining piece from the AS/400 system. We introduced VMWare into the organization and virtualized just about everything. This caused significant overhead for our SQL Server 2000 infrastructure.

Our .NET 2.0 environment introduced publish and subscribe messaging with BizTalk 2006 at its core. We introduced a common tracing and logging strategy with the introduction of Paul Bunyan, we rolled out a new website based on a web content management system named Sitecore introduced the Microsoft Enterprise Library June 2005 and moved what we could to a new Physical x64 SQL 2005 infrastructure. Most importantly we did it for less than one million dollars per web page and without introducing a bunch of new workarounds and data corruption. Executive confidence in the IS organization improved. The last three pieces are almost implemented and the AS/400 is slated for shutoff at the end of June. We learned that as the technology improved we were now discovering process opportunities that needed to be addressed.

Right now we are in an ackward middle place. We are in the process of improving all sorts of processes. We are learning about ITIL, having some meetings and beginning to discuss what it means to us, and proceeding with a deliberate pace. We hired an Enterprise Business Analyst, who happens to be leading our implementation of the Rational Unified Process (RUP). We have a number of internally named processes specific to the update of software in production Program Change Request (PCR) and we have processes to fix data in production called Program Update Request (PUR). These processes are being evaluated, and re-implemented to meet the current organizational needs. This should keep us busy for the next few years. Maybe I’ll write more on this process stuff. It’s much harder than enterprise architecture. I get to herd cats (easy). Enterprise process improvement is more like building an ark (harder).

Office 2007, Exchange 2007, Communicator, MOSS 2007, System Center 2007, VMWare 3.x promise to be very disruptive technologies for us as we move into 2008.

Now on to our 2008 architecture. Developers really enjoy working with the Paul Bunyan product. It gives us the ability to really trace messages between servers in our organization in real time, but doesn’t really give us an efficient way to aggregate all of the messages on all servers in our environment in a single place. Windows Server 2008 will be out soon and we are beginning to make plans to migrate all of our code on to that platform as soon as it becomes available. This really gives us a solid automated deployment strategy using the web.config file to configure the web server portions of our ASP.NET configuration files. We are moving to the VMWare 3.x platform which gives us a real advantage in the types of devices we can have in our VMWare infrastructure. We are looking at ESX server’s ability to restart machines during hardware failure as an alternative to a server farm for uptime reliability. We are looking into hosting a virtual Google Mini search appliance for DR purposes and hosting a Virtual Network Load Balancers such as the ServerIron XL in a VMWare platform. VMWare has made significant increases in the performance story. With 2.5.x we were really limited to one processor, 4GB ram and 32bit processing. With 3.x the ceiling is much higher. We are now looking at moving Microsoft SQL Server 2005 back into the VMWare infrastructure and improving our Business Continuity Story. Visual Studio Code Named “Orcas” is slated for us some time in 2008. We’ve got some upgrading to do on our Team System Infrastructure.

Then comes the really fun stuff. The giant initiatives in 2008 are:
- Implement Office 2007, MOSS 2007, Communicator, VOIP and CRM
- Create composite applications using a SOA backend
- Implement System Center 2007 and integrate it with domestic and commercial applications
- Integrate TFS with Quality Center and Requisite Pro
- Leverage VMWare
- Integrate RUP, StageGate and Agile into a single view of project governance, management and execution.
- All the other stuff the business wants

Like many things this post is becomming way to long, I'll be back with more later.

Software Engineering as a Profession

At the recent ICSE 2007 in Minneapolis, Deborah G. Johnson described a framework which professionals have used to define themselves as professionals and differentiate themselves from hired guns. A few in the audience commented on how the UK and Canada had begun to organize their practitioners of software engineering into professional organization. I started to think about what this means for me. I asked myself how would a professional organization help me enjoy work more than I already do?

One of the things I really like about computers is watching people in accounting use spreadsheets. They add macros, complex formulas and really configure the computer to do what they need in order to answer some question. The really fill the need of pseudo programmers.

I really enjoy the speed at which I can build complex systems inside of our SOA ecosystem. I call a bunch of APIs, slap together some code, run it through some compiler and really configure the computer to do what someone has asked me to do with the computer.

Yeah I understand how to talk to the computer at a lower level, but am I practicing software engineering? Did I exercise any sort of social responsibility when putting together that SOA service? Did I really understand the engineering aspects of what I was doing, or was it just a factor more complex than putting together my kid’s Barbie house?

A manager I work with recently told me that “You are not qualified to know or even pretend to know what it’s like for a real person to edit HTML. It is such a part of you that it’s not worth trying to pretend.” I began to wonder….

During the dialogue after the keynote, I heard terms like practitioner and academic. I realize the distinction that the academics are making, but I’m not sure how a bunch of academics, even if they were unified would successfully mount a campaign to turn software engineering into a profession, especially when 90% of the software development community would find it hard to associate with a group of people who claim the developers aren’t really qualified to do what they love.

I see a movement in usability that will really make it easier for regular people who would not dare edit raw HTML to configure computers to do what they want. When I learn about the usability studies that Microsoft is doing specifically in regards to developer productivity I wonder if there aren’t really different types of software developers. I’ve yet to see it work, but I’m sensing that it’s getting easier and easier for non-programmers to program.

At the keynote on Wednesday we saw a presentation that demonstrated how you could configure a system to do what you want. Sure you needed to be tech savvy, but you didn’t need to be a programmer. It made me think that my days as a programmer were coming to a close.

Indeed that might be exactly what is happening. It is entirely possible that Software Engineers will design systems for regular people and the corporate developer falls by the way side.

I look forward to being able to abstract the process to make it cost effective for my co-workers not in IT.

ICSE - Day 1

I’m at the ICSE 2007 this week and attempting to discover how to put more science into what I consider to be a mostly artistic and political process called leveraging technology to solve business problems.
For the record this is the first conference since PDC 2005 where I’ve felt like I could really geek out. I’m attending this conference to see if I can learn from the software engineering discipline. What I’m discovering is that the discipline is present in things like aerospace, and there are some truly innovative DSL tools around factory automation, but in general and too my delight I’m finding an artistic side to this discipline. I’m also running into a lot of that “It depends” nonsense. I’m seeking answers not ideas!

Here are a few statements I heard and my reaction:
1. Software is like paint – It’s not about the paint, it’s about what you do with it. Some comments were around educational systems trying to face the challenge of educating software engineers in the art of painting compliers but when they go out into the word and look for a job, the industry is asking for a couple of complier painters, but a bunch of business application painters, etc.

2. “We lose 40% of our software engineering freshman because we try force they way they program” I was in a presentation today that focused around whether or not you should require parameters in the constructor or use the create-set-call pattern. I hope I got that right. They discussed three of the personas that Microsoft uses. I’m not sure what the percentage of each persona is, but I do know that of the programmers I’ve worked with, I can say that programmers work with their own habits, and in their own way. Having only one way of teaching didn’t work for me and I dropped out of my first Fortran class as soon as I found how to register.

3. “How did you become a good architect” – I worked with a few good architects.

4. SaaS is here to deliver the same ease of use functionality you get from Yahoo or Google and bring it to the enterprise space. – As I think about this I’m considering our current decisions around SalesForce.com and Microsoft CRM. I’m torn. On one hand I see a team of developers who know .NET technology, a BA group set on gathering requirements, and a QA group set on testing the changes. It’s how we are growing up and it’s for all the right reasons. I think that once we will go with Microsoft CRM for now but our next round might be with SalesForce.com. I’m thinking that the amount of control our organization would need to give up would not fly with all of the other organizational changes we have going on right now. From a software engineering perspective we would need to expose our internal data externally over secure web services and call to them from SFA.com. This is all very possible, but first we might have to implement something locally like Dynamics CRM, get our web services setup internally, and then migrate over to SFA.com once they get a little better and we get a little smarter. I really like the idea of running on the SFA platform, but I’m not sure I want to introduce SFA programming into our organization right now.

5. Barry Boehem is the reason it took so long for Agile to come about. He published a study that said it was very expensive to fix something in production and very cheap to fix it in design. – Barry commented on this and agreed. He also commented that it depends. The research he did and the data gathered applied to systems where the data could be collected. As soon as you introduce people into the process, how do you accurately gather requirements? Perhaps the principles behind Agile methodologies apply to more people intensive processes. Barry seems like a great guy.

6. Figure out when you should catch defects, classify your testing types and when you find a defect determine whether or not you should have caught it earlier or if you caught it in the right spot. – This is an interesting one. I’m curious to see how we tackle this. When a defect happens in production, we ask the question “Should we have caught this earlier, or is this the right place to catch them?”