Empirical verification of effectiveness for a knowledge-based system

In the last decade, information centers (ICs) have been proven to be a successful strategy for managing software resources of organizations. The initial success of ICs has increased user expectations and demand for the services offered but, because ICs are considered cost centers in most organizations, there is growing pressure for them to accomplish more with fewer resources. A knowledge-based system ICE, (Information Center Expert), has been developed to assist users with software selection. The study reported here focuses on the evaluation of ICE to determine users' perception of its effectiveness. This experimental evaluation of ICE was conducted at the University of Arizona's Center for the Management of Information (CMI), which operates as an Information Center supporting faculty and students in the College of Business. The use of student subjects in the experiment was deemed appropriate because they are, in fact, the end users of the CMI information center. The verification of the effectiveness of ICE was attempted by conducting a laboratory experiment to test the comparative advantages of using ICE or CMI consultants to obtain assistance with software selection. The experiment was designed as a 2 × 2 factorial. The independent variables were users (beginner or advanced) and type of consultation process (ICE or CMI consultant). The dependent variable was a measure of consultation effectiveness. Instruments for classifying users and measuring effectiveness of a consultation process were developed and validated.