ECVET - Application to mobility projects

Implementation of the the H.E.LE.N. referencing strategy
Country-to-country comparison has been carried out by using information corresponding to homogeneous ICT-related study cases belonging to five national qualification structures (namely France, Italy, the Netherlands, Sweden and England), used to populate a semantic knowledge base.

In order to characterize each national qualification element, the models of qualification systems described above were exploited to define suitable meta-data that were used for annotation purposes. Annotation was carried out manually, although a prototype of an automatic annotation tool was developed, by referring to a common glossary of terms (keywords) defined by the experts. Moreover, weights were assigned to meta-data to point out the relevance of the specific keyword.

For testing purposes, annotation was performed on lower level elements of the models, in some way corresponding to learning outputs/outcomes (e.g. the Contenuto, Descrittore, and Indicatore concepts in the Italian model, or the Compétence professionnelle, Critère d'évaluation and Niveaux de performance concepts in the French model, among others).

Table 1 shows an excerpt of meta-data and keywords linked to several lower level elements of the Italian model. As a matter of example, the competence element "Individuare le caratteristiche peculiari dell'oggetto da installare" (or "To identify distinguishing features regarding the installation of a given object", in English), marked with the Contenuto meta-data, could be characterized three keywords as "(to) install" (with weight 50%), "features" (25%) and "identify" (25%). It is worth remarking that, in a working scenario, weights would have to be jointly assigned by the experts, according to the meaning of the specific element. An excerpt of a possible annotation for the French qualification considered in the analysis is illustrated in Table 2. In this case, learning elements are described in terms of three meta-data, i.e. Compétence professionnelle, Critère d'évaluation, and Niveaux de performance. Relations between elements of the Italian and French models can be found in Figure 1.
Once the knowledge base was populated with information concerning selected study cases, the semantic engine logic for the calculation of the match between learning elements belonging to different qualifications was developed.
It is worth remarking that:

• referencing rules in UML meta-models were first described through a semantic formalism and included in the knowledge base together with weights related to values in analysis matrices;
• the semantic logic was configured to determine the matching, or degree of similarity between qualifications, at two levels of details:

- a lower one, that compares each lower level element (e.g. Contenuto, Descrittore, Indicatore, Compétence professionnelle, Limite de connaissance, Niveaux de performance, etc.) with the remaining lower level elements of the meta-ontology;
- a higher one, that compares a whole/aggregate learning element (e.g. a container of Contenuto, Descrittore, Indicatore and elements in the Italian system, i.e. a Competenza, or a container of Competenza elements, i.e. a Unità Capitalizzabile) like "Installare software e sistemi hardware" (or "To install software and hardware systems", in English) with a target element;

• comparing lower level elements could mean selecting a row from the annotation in Table 1 and comparing it with every row of the target annotation in Table 2;
• calculating a higher level matching would mean using all the keywords and weights of the source annotation and comparing them with all the elements of the target annotation, by exploiting lower level information;
• such a two level comparison strategy, that splits the matching calculation in two phases, was defined in order to better study how changes in referencing rules between lower level concepts could improve or worsen higher level matching. According to experimental observations, the above behaviors occur when there exists a marked difference between the expected referencing weight between two lower level items, and the value provided by concrete study cases (and most of the times, this is due to information that is put in the wrong container in the original document describing the qualification);
• by changing the weights that regulate relationships between lower level elements it becomes possible to overcome this limitation, and to provide matching values which are more accurate.

Table 1. Possible annotation of the "Installare software e sistemi hardware" aggregate element in the Italian model.



Table 2. Possible annotation of the “Installer une application logicielle” aggregate element in the French model.

The reasoning behind the two-way strategy to compute the matching degree is as follows:

• lower level matching: if a couple of lower level concepts has the same keywords, the system computes the matching degree as the sum of the weights of the keywords found in both elements (divided by the total number of keywords associated with each element), divided by the sum of the inverse of the total number of keywords linked to each lower level element;
• higher level matching (two ways): a) without using the referencing rules, i.e. the system computes the mean of the values found in the lower level matching (the weights of the linking rules between lower level concepts are not considered); b) using referencing rules, the result of the lower level matching is multiplied by the weight of the referencing rule between lower level element, and it is then divided by the number of possible combinations (source element - target element).

This two-way calculation is important for the achievement of the final goal, that is the identification of the level of similarity between qualifications; moreover, it could also be exploited to check whether, by properly varying referencing weights, similarity degree could be improved.

The identification of the correct weights (now done manually by the experts, but potentially manageable in an automatic way) will help national actors to better align qualifications in a transnational scenario, characterized by the application of referencing rules through the use of the EQF.

Partial results of an experimental test over the selected Italian and French study cases showed that, by using suitable referencing weights, the matching degree could be optimized. As a matter of example, Figure 3 shows the results of the higher level matching calculation between two ICT-related qualifications: the Italian higher level element "Installare software e sistemi hardware" is compared with the French learning elements included the considered profile. Without using the referencing weights the best matching degree is 38.04% whereas, when proper weights inserted by the experts (Figure 4) are considered, this value increases to 46.04%. On the other hand, the worst matching degree (between "Installare software e sistemi hardware" and "Dépanner un système informatique") decreases, from 6.46% to 4.17%.

Figure 3. Higher level matching between ICT-related Italian and French qualifications.




Figure 4. Parameters configuration for the relation between Italian and French systems.

Results of a lower level matching are presented in Figure 5, where a Descrittore element belonging to the Italian model ("Saper effettuare l'installazione dei principali prodotti di office automation"), is compared to all the lower level French elements. Values in the before to the last column are calculated without considering referencing weights, while values in the last column take into account also the above weights. It is worth remarking that, when comparing with a Critère d'évaluation or Niveaux de performance element of the French model, the lower level matching value is equal to zero (because of the weights assigned to the rules).

Figure 5. Lower level matching between ICT-related Italian and French qualifications.