You are here

Evaluate Vendor Proposals

28 August, 2015 - 14:19

Using vendor responses to the RFP, the logical specification, and the physical requirements, analysts must decide which, if any, proposal best meets the organization’s needs. The process of evaluating the vendor proposals includes three steps:

  1. Validate vendor proposals.
  2. Consider other data and criteria.
  3. Suggest resources.

Many organizations assign a team to evaluate the proposals. The team could consist of personnel with IT technical expertise, business process owners, system users, external consultants, lawyers, and accountants. The evaluation team completes these three steps to suggest the software, hardware, and services that best meet the organization’s requirements.

Validate Vendor Proposals

The first evaluation step is to validate the vendor proposal to assess whether the system (software or hardware) does what the organization requires by studying a proposed system’s specifications and performance.Specifications are straightforward descriptions of the hardware and software—such as a software package’s maximum table sizes or a printer’s speed—that can be examined to determine whether the system has the ability to perform as required. Performancefeatures can be determined only through testing, measurement, or evaluation and often include items such as user friendliness, vendor support, 1 quality of documentation, reliability, and ability of system to produce complete, accurate, and timely output.

Review Question

What is the difference between a specification and a performance measure?

 

One commonly used method for measuring system performance involves measuring the system’s throughput, which is the quantity of work performed in a period of time. For instance, the number of invoices that a system processes in one hour is a measure of throughput. Other performance measures, such as ease of use, are more subjective and may be more difficult to determine. Technology Excerpt 7.2 describes some performance factors to be considered when choosing an ASP.

After eliminating those proposals that do not meet minimum requirements, the evaluation team tests 2 the remaining systems to determine the accuracy of the vendors’ specifications and how well the equipment will work for the organization. Having determined what a system is, we test to determine what that system can do.

An evaluation team can test a system by:

  • Varying input (workload) parameters, such as quantity, timing, and type of input.
  • Varying system characteristics (parameters), such as quantity and size of data storage devices.
  • Varying the factors being measured, such as CPU cycle time (a system parameter) or execution time (a performance measure).
  • Testing an actual workload, such as a weekly payroll, or testing a workload model that is representative of the workload.
  • Testing the actual system or a model of the system.

Technology Excerpt 7.2

Issues to Consider When Selecting an ASP

  • Information obtained from present users about the ASP’s technical and service capabilities.
  • Level of systems availability. How is this measured? Do availability guarantees extend to services employed by the ASP (i.e., secondary ASPs)? What are the penalties for failure to meet targets?
  • Provisions for security, data backup, and disaster recovery.
  • Capabilities and certifications of technical support and operations personnel. What is the staffing at the hosting site? When are they there? Is technical support provided directly or by a third party?
  • How much can the application be tailored?
  • Software ease of use, reliability, and quality (particularly important when the system must be used with minimal training on a more occasional basis).
  • What is covered by the cost?
  • The ASP’s partners, including their software and hardware vendors and ASPs from who they obtain services.

Sources: Excerpted from Gary Anthes, “Avoiding ASP Angst,” Computerworld (October 16, 2000): 80–81; “Seven Issues to Consider When Using an ASP,” Viking Software Solutions (http://www.vikingsoft.com), November 27, 2000.

media/image1.png The Internet has made it possible for vendors to demonstrate their software on their Web sites. For example, at mySAP.com (http://www.mySAP.com) you can see demonstrations of and “test drive” SAP’s R/3 enterprise system. This site allows prospective buyers to evaluate the system’s capabilities and to identify the modules that will best meet their needs. Other vendors providing similar services include Microsoft’s Great Plains (http://www.greatplains.com), where you can test their enterprise system, and SBT (http://www.sbtcorp.com), where you can test their ACCPAC eTransact system. 3
Consider Other Data and Criteria
Rather than estimate vendor and system performance internally, the evaluation team can interview users of the vendors’ products and visit those sites to witness the system in action. Quite often, vendor presentations are made at the site of an existing user. External interviews—interviews conducted with personnel outside the organization—can provide valuable insights into vendor performance. Where appropriate, questionnaires can also be used to gather information from users. The following information might be collected from users:
  • Were there delays in obtaining the software or hardware?
  • Did the system have bugs?
  • How responsive is the vendor to requests for service?
  • Was the training the vendor provided adequate?
 

As mentioned in Technology Insight 7.2, there are several services that publish technical reviews, user surveys, and expert commentary on computer equipment, software, and a variety of related topics. The reviews and user surveys can be helpful when evaluating proposals.

A cost/benefit analysis is often used to determine the economic viability of the remaining vendor proposals. Quantifiable costs and benefits are summarized to determine whether vendor proposals can be justified economically. Ranking vendor proposals using the economic criteria is useful in the next step in systems selection, in which the evaluation team suggests which vendor proposal should be chosen. The identification and quantification of Information Systems costs and benefits, however, is a difficult process requiring as much art as science. Still, we must have some data with which to make a decision.

Suggest Resources

At this point, the study team recommends one vendor proposal. Management then chooses the software and hardware resources. To recommend one vendor, the evaluation team compares the proposals that have not been eliminated. The evaluation team might list the relevant criteria and indicate the performance of each vendor on each criterion.