This article focus' on evaluating CIS application's technical architecture. This is absolutely one of the most overlooked aspects of the selection process. I've spoken with several organizations who have made the mistake of only taking a cursory look at the technical architecture an application is built on. Not going through a thorough evaluation of an applications technical architecture is like buying a car without inquiring about the specs of the engine. Could you imagine buying large Sport Utility Vehicle only to find out on the way home it is running on a 4-cyclinder? You would probably have a good idea when you reached the first hill if you didn't notice a lack of power when you were pulling out of the dealership. The problem on the CIS side is you usually don't find out about the implications of a weak or poor architecture until you go-live if you don't allow for a detailed technical evaluation. I'll go into more detail later in this article about some of the problems incurred when trying to operate on a poorly architected application.

Due to the importance of an application's technical architecture TMG Consulting has designed a comprehensive methodology to evaluate and compare different architectures. From the highest level TMG Consulting focuses on the following technical areas in the technical evaluation. While the evaluation varies slightly from site to site below are some of the areas I focused on in my last evaluation. The next evaluation will be more extensive due to changes in technology and the market place:

The above list is just a portion of what I review during the TMG Consulting selection review. I think the above list will give you at least a flavor of some of the questions you should be asking during your technical evaluation.

The categories above are then broken out into individual subsections with questions that fall under them. It is similar to the TMG Consulting functional checklist which has become the industry standard for evaluating CIS systems from a functional perspective. The advanced technical review TMG Consulting uses has well over 100 technical questions that are then graded in an effort to rank order each proposing vendor. Below is a format to rank order the proposing vendors from a technical perspective.

Area	Possible Points	Vendor A	Vendor B	Vendor C	Vendor D	Vendor E
Deployment	5	4.000	4.909	4.818	3.091	5.00
Design	10	8.182	9.636	9.818	7.091	9.818
Duration of Support	10	7.33	8.000	8.000	7.33	6.67
Extensibility	15	11.800	14.600	14.600	9.000	14.600
Functionality (Operational)	5	3.583	4.917	4.33	1.417	4.677
Modifiablity	10	7.00	8.00	8.333	7.333	8.333
Organization (Technical)	5	4.133	4.533	4.533	3.600	4.800
Scalability	10	7.566	10.00	10.00	4.667	10.00
Staffing Impact	10	9.111	8.889	9.778	8.000	9.778
Support (Infrastructure)	5	4.667	5.00	5.00	4.667	5.00
Technical Architecture	10	8.889	9.778	9.778	8.000	9.333
Technical Session	5	3.500	4.000	5.000	3.500	3.00
Total	100	79.754	92.262	93.992	67.698	90.996
Based on a 30 Point Score		23.926	27.679	28.198	20.310	27.299
Rank Order	--	4	2	1	5	3

As you can see the vendors technical responses are broken out into the above categories that were defined earlier in the article. In a nut shell the methodology I just reviewed is what I use to help TMG Consulting clients eliminate risk for a technical perspective.

You may be wondering, why is this important?
Over the last few years I found that this is not only important but it is critical in selecting a CIS. You can usually work around functional problems through work arounds or even system modifications. You can not work around significant technical architecture issues! I've seen organization brought to their knees due to technical problems around their CIS. Some of the problems I see when technical architecture is ignored are as follows:

At the end of the day it is critical to check references closely after a comprehensive internal review is performed. If the vendor has not implemented at a similar site in terms of size and scope it should send out red flags. Consider asking a reference the following questions:

I can't tell you how many utility organizations have told me about how they made the mistake I just defined. In the last year alone I have spoken to several utility executives that are back out to market for a new CIS because they made the mistake of not digging deep enough into the technical aspects of the application. Most vendors know the weaknesses of their application and avoid going into those details if they are not forced to. Unfortunately, most utility organizations do not have the depth of knowledge to ask the right questions. In the end, the utility takes the vendor's word on the applications technical merits. Once the client gets into the implementation they begin to see the technical limitations of the application they selected as they are forced to modify daily processes to deal with the technical limitations of the application. In some cases I've seen organizations forced to split batches in order to complete a batch run within acceptable timeframes thus crippling the utility's business and customer service.

The question to ask your self is, how did I evaluate technology? Did my organization make these same mistakes? If you have concerns with those questions you would fall into the majority.

Conclusion
I'd like to point out that this is just one facet of the selection process I feel is critical. While I have covered this topic at an extremely high level in this article I believe you should have a good feel for what you should be looking for from a technical perspective. The main point in this article is to point out how critical the technical evaluation is. In this article I've described a couple of highly effective ways to do this.

This is one of the most over looked areas in the selection process. While this appears to be an obvious evaluation point and should be weighted accordingly I continue to be amazed how often technology is overlooked. The fact of the matter is more time and effort should be spent looking under the hood of some of these systems.

Mario Bauer is Senior Consultant for TMG Consulting. Mr. Bauer has over 12 years experience with utilities as: an analysts for a major utility, Senior Associate for a major utility consulting firm, and a Senior Consultant for TMG Consulting. Mr. Bauer has worked with approximately 100 utilities (electric, gas, water, wastewater, solid waste) around the world. Mario can be reached at Mariob@TMGConsulting.com.