|
|
|
Mitigating CIS Selection
Risks
By Mario Bauer; Vice President (October 2003)
Over the past few years CIS selections have become more and more
scrutinized due in part to the high number of high profile failures.
During this period utility organizations looking to replace their
CIS would host a few vendor demos and make a critical decision based
upon a half day demo. The vendors would come in with a great demo
put together by a very well prepared demo team. As a part of the
demo process the vendors would "talk" a little bit about
their technology and maybe spend a few minutes discussing their
proposed project plan.
Well, as we all know this process led to several high profile CIS
failures. Many embarrassed utility execs had to step back and look
at their selection process trying to figure out what went wrong.
After talking with several of these execs and reviewing their situations
there was a consistency in the processes they went through that
led to their failures. The following topics are the most commonly
overlooked areas according to people I've talked with:
- Functionality
- Technical Architecture
- Project Management
- Organizations IT Strategy
Functionality
Functionality has always played a huge role in organizations
selecting a CIS. The problem is so many times functionality get
confused with ease of use or look and feel. Most of these selection
teams in the selection process had/have a strong representation
from the customer service side. Typically there are several customer
service representatives (CSRs) on the CIS selection team. The focus
of this group typically is on the look and feel rather than functionality.
While both functionality and ease of use are critical they need
to be separated to properly evaluate a CIS.
After speaking with several of these organizations during their
implementation or right after go-live I typically hear the same
thing. "I didn't realize we didn't have this functionality".
"I didn't realize that the functionality they said they had
out of the box required extensive configuration and was not a part
of their quote". "They were not clear on the type of skills
that were necessary to set up these workflows".
At the end of the day the consensus was that if they could do it
over there they would have focused more on functionality rather
than just look and feel. Below is a chart that demonstrates how
the TMG Consulting selection process guards against not accounting
for functionality in the selection process.
As you can see each vendor has a functional score for look and
feel and a pure functional score. The look and feel and functional
scores role up to an overall functional score as shown in the chart
above.
One of the keys in the functional evaluation is to have several
organization subject matter experts or more commonly known as SMEs
on the selection team.
Technical Architecture
Technology has been so overlooked in the selection process
in many of these failed implementations. Most vendors are not really
questioned on their technology because most of the time the utility
does not have the expertise in house to address the issues and most
CIS selection consultants do not have the expertise to address it
either.
I've been to several sites and spoke to several organizations that
experienced several problems that were directly linked to their
lack of attention to technology in the selection process. Issues
around performance, costs and resources to support, are just a few
of the many issues I've seen at sites when technology is not adequately
addressed. Below is a list of topics that should be addressed in
a CIS technology evaluation.
| CIS Selection Technical
Review (Grading Criteria) |
Weight |
Explanation |
| 5% |
Deployment
Deployment considers the complexity and degree of difficulty
in installing, upgrading and supporting releases of the application
software, along with required fixes and upgrades. The database,
application, and client layers are all considered. |
| 10% |
Design
Design of the system pertains to the logical presentation of
both the current and future states of the system. Normalization
of the database, as witnessed in the ERD is critical, along
with well defined current and future state functional models,
standards, development methodology, flexibility in how forms
can be modified or tailored at the user and group level, along
with overall flow of the system are considered. |
| 10% |
Duration of Support
Duration of support will be judged on how long a specific release
of software will be supported subsequent to becoming a production
release. |
| 15% |
Extensibility
Extensibility has to do with two basic areas. First the availability
of true “bolt on” application software that is available
to use with the base application. Second, the availability of
tools to be able to access information. The ability to integrate
with easily accessed interfaces and APIs. Ideally, specific
interfaces are available that have been pre-programmed and,
if not, tools that are relatively intuitive are available to
develop then required interfaces. True B2B/Portal functionality
is also being evaluated in this area. |
| 5% |
Functionality (Operational)
Functionality pertains to the technical aspect of meeting the
user’s needs. Batch vs. online operations, real-time access,
etc. |
| 10% |
Modifiability
Modifiability addresses the degree of difficulty in modifying
the system to meet specific business needs outside of the core
application that will require development. Development environment,
availability of resources, mandatory SDLC, etc. |
| 5% |
Organization (Technical)
The organization aspect of the review pertains to the existence
of a stable company with a defined R&D budget, support mechanism,
adequate distribution channel, et al. aspect of a true on going
concern. Bundled offerings are also reviewed. |
| 10% |
Scalability
Scalability pertains to the proven ability for the evaluated
system to be able to support the designated customer base and
transactions volumes. This not only pertains to adequately sized
hardware in the proposed solution, but also considers the data
and process models and their ability to perform adequately at
the specified customer and transaction volumes. |
| 10% |
Staffing Impact
Staffing impact will be graded based on the ongoing technical
support requirement necessary to facilitate operations once
the new application goes live. |
|
5%
|
Support (Infrastructure)
The support assessment of this evaluation addresses the vendor’s
help desk organization. Levels of support, escalation mechanisms,
web-based tools and repositories, along with experience of and
access to qualified support staff are all considered. |
| 10% |
Technical Architecture
The technical architecture takes into consideration the physical
architecture of the system and how contemporary it is. The physical
server/application server/desktop (hardware), disaster recovery
and failover, security, application and database architecture
along with how the product is delivered to the end user are
all major considerations. |
| 5% |
Technical Session
The technical session is graded mostly on the ability of the
vendor resources to accurately and concisely address the concerns
of the technical team. Ability to discuss and demonstrate system
functionality, architecture, backup and recovery, tools, security,
user security, form changes and modifications, and administration
are all considered. |
The example above displays weighting associated with each area
that were a part of the evaluation process for this particular selection.
The bottom line is if you're not properly evaluating technology
during the selection it will definitely come up at some point during
implementation or soon after go-live.
Project Management
Project management in this industry has been severely overlooked
in the past several years. While there are several solid products
in the market place most of these vendors have sites that they would
prefer you not know about. The problem was not the product per se.
The problem was directly related to the project management during
the implementation of the product. Part of the problem is this is
the only industry I of know where people with absolutely no implementation
experience get away with running large implementations. These installations
are so complicated and require years of experience if you are going
to reduce the risk of failure. I've seen too many pumped up resumes
of former bill print salesmen trying to pitch themselves as Project
Managers.
In order to expose weak project management TMG has utilized the
following review to evaluate the project management proposed. The
criteria below has several sub-sections that roll up to sections
addressed in the chart below.
In the chart above are the areas I feel are most critical
and often the most overlooked during the implementation process.
Your CIS evaluation process must address these areas in one way
or another to reduce the overall risk. If the project management
evaluation addresses the above areas you can be sure to expose some
of the weaker project management plans proposed in this industry.
Several CIS vendors have begun to partner up with system integrators
to bring stronger project management tools to the table. Up until
a couple of years ago a system integrator in the proposal often
meant much higher implementation costs. In the last year the overall
cost of an SI has come down but you will pay for it. I would estimate
you pay at least 20-30 percent extra for an SI but many times I
feel the money is well spent in some cases.
Organizations IT Strategy
Most of the time when I'm on site during a CIS selection
I always try to find out what their IT strategy is. Is the organization
committed to an integrated solution or are they looking more at
best of breed solution which would require some investment dollars
in middleware? This is a question that should have been answered
in advance of the CIS selection but usually is not addressed until
we are well into the selection process. The chart below illustrates
this point.
There are several other variables to consider when selecting a
CIS such as references and vendor viability but I believe the areas
I mentioned are key to mitigating risk in the CIS selection process.
If you follow the areas I mentioned above I can assure you your
risk will be reduced. There is no one way to address to the risks
I mentioned above but if you choose not to thoroughly investigate
everything mention above I can assure you will ask yourself how
you could have overlooked it at some point down the road. My next
few articles will go into each one of the above areas in more detail.
Mario Bauer is Vice President and Senior Consultant
for TMG Consulting. He has over 12 years experience with utilities
as: an analyst for a major utility, Senior Associate for a major
utility consulting firm, and a Senior Consultant for TMG Consulting.
He has worked with approximately 100 utilities (electric, gas, water,
wastewater, solid waste) around the world. Mario can be reached
at mariob@tmgconsulting.com.
|
|
