The method can be divided into three stages (Valcheva & Todorova, 2005b):
1. Framing of the questionnaire, which must consists of the following very important parts:
- List of the indicators for evaluating the effectiveness of e-learning, which rate of importance have to be assessed by the experts;
- A cell, where every interviewee can put his mark (the evaluation scale is preliminarily determined by the questioner)
- Information for the competence and the resource of the argumentation of the different experts, participating in the interview
2. Defining the circle of the experts that will be interviewed, and implementing the interview.
3. Defining the rate of competence of the experts, eliminating the inadequate opinions and processing the results. The rate of the experts’ competence is defined by:
(1)
where b1, b2, b3 are defined according to respectively - the official position and rank of every expert; the time, spend on working at the problem; and the resource of his argumentation. The coefficient varies in the interval from 0 to 1 and the experts’ opinions, which rate of competence is less than the preliminarily determined value has to be eliminated from the later processing of the results.
The meaning of b1, b2, b3 can be defined in the following ways:
|
Official position and rank |
Without degree |
Doctor |
Doctor of science |
|
Assistant |
2 |
4 |
- |
|
Associate professor |
4 |
6 |
8 |
|
Professor |
8 |
10 |
12 |
|
Time, spend on working at the problem (in years) |
0 |
1 |
2 |
3 |
4 |
5 |
>5 |
|
0 |
2 |
4 |
6 |
8 |
10 |
12 |
|
Resources of the argumentation |
|
|
By intuition |
2 |
|
Acquaintance with the problem state |
8 |
|
Practical experience |
10 |
|
Implementation of theoretical analysis on the problem |
12 |
The next stage of the method includes the procession of the results, obtained by the provided interview. On the basis of the received experts’ assessments the weight coefficients of the suggested groups of indicators and actually the indicators themselves are processed, and the agreement rate of the experts is determined.
1. The procession of the results consists of the following stages:
Defining the weight coefficients of the suggested groups of indicators:
Processing the average assessment of the experts Ft for the rate of importance (weight coefficients) of each group of indicators by the formula:
(2) 
where htj – the assessment of the j expert for the weight of the t group, m- the number of the experts and t obtains value from 1 to 5, according to the defined number of groups.
Procession of the normalized assessment kt for each of the groups:
(3) 
Procession of the weight coefficients of the indicators within the groups:
Defining the average assessment Si of the groups of experts for the rate of importance of each indicator within the group:
(4) 
where rij – the assessment of j expert for the weight of the i indicator, m- number of the experts.
Processing the normalized assessment for the weight of each indicator:
where n- number of the indicators.
(5) 
Formation of the weight coefficients, according to the group weight, to which they belongs:
(6) 
In that way the sum of the weights coefficients of the indicators within a given group is equal to the weight coefficient of the whole group:
(7) 
2. For determination of the agreement rate of the experts, the average quadratic diversionis calculated:
(8)
Where
(9)
(10)
Based on (8) and (10) the variation coefficient Vi, is calculated, which characterized the agreement rate of the experts, participating in the research:
(11) 
The smaller is the value of Vi, the higher is the rate of the experts’ agreement.
- 1940 reads
