You are here

Concepts of the expert evaluation methodology

21 January, 2016 - 14:46

The method can be divided into three stages (Valcheva & Todorova, 2005b):

1. Framing of the questionnaire, which must consists of the following very important parts:

  • List of the indicators for evaluating the effectiveness of e-learning, which rate of importance have to be assessed by the experts;
  • A cell, where every interviewee can put his mark (the evaluation scale is preliminarily determined by the questioner)
  • Information for the competence and the resource of the argumentation of the different experts, participating in the interview

2. Defining the circle of the experts that will be interviewed, and implementing the interview.

3. Defining the rate of competence of the experts, eliminating the inadequate opinions and processing the results. The rate of the experts’ competence is defined by:

(1)media/image2.png

where b1, b2, b3 are defined according to respectively - the official position and rank of every expert; the time, spend on working at the problem; and the resource of his argumentation. The coefficient varies in the interval from 0 to 1 and the experts’ opinions, which rate of competence is less than the preliminarily determined value has to be eliminated from the later processing of the results.

The meaning of b1, b2, b3 can be defined in the following ways:

Table 4.5 Meaning of b1

Official position and rank

Without degree

Doctor

Doctor of science

Assistant

2

4

-

Associate professor

4

6

8

Professor

8

10

12

 
Table 4.6 Meaning of b2

Time, spend on working at the problem (in years)

0

1

2

3

4

5

>5

 

0

2

4

6

8

10

12

 
Table 4.7 Meaning of b3

Resources of the argumentation

 

By intuition

2

Acquaintance with the problem state

8

Practical experience

10

Implementation of theoretical analysis on the problem

12

 

The next stage of the method includes the procession of the results, obtained by the provided interview. On the basis of the received experts’ assessments the weight coefficients of the suggested groups of indicators and actually the indicators themselves are processed, and the agreement rate of the experts is determined.

1. The procession of the results consists of the following stages:

Defining the weight coefficients of the suggested groups of indicators:

Processing the average assessment of the experts Ft for the rate of importance (weight coefficients) of each group of indicators by the formula:

(2) media/image3.png

where htj – the assessment of the j expert for the weight of the t group, m- the number of the experts and t obtains value from 1 to 5, according to the defined number of groups.

Procession of the normalized assessment kt for each of the groups:

(3) media/image4.png

Procession of the weight coefficients of the indicators within the groups:

Defining the average assessment Si of the groups of experts for the rate of importance of each indicator within the group:

(4) media/image5.png

where rij – the assessment of j expert for the weight of the i indicator, m- number of the experts.

Processing the normalized assessment for the weight of each indicator:

where n- number of the indicators.

(5) media/image6.png

Formation of the weight coefficients, according to the group weight, to which they belongs:

(6) media/image7.png

In that way the sum of the weights coefficients of the indicators within a given group is equal to the weight coefficient of the whole group:

(7) media/image8.png

2. For determination of the agreement rate of the experts, the average quadratic diversionis calculated:

(8)media/image9.png

Where

(9)media/image10.png

(10)media/image11.png

Based on (8) and (10) the variation coefficient Vi, is calculated, which characterized the agreement rate of the experts, participating in the research:

(11) media/image12.png

The smaller is the value of Vi, the higher is the rate of the experts’ agreement.