Methodology

The aim of the SMART-QUAL project is not the creation of an exhaustive and extensive compilation of all possible Quality Indicators already in use or able for use, but of a SMART set of Indicators that could be defined as follows:

a)     Short: focused on the efficiency and effectiveness of IQAS and avoiding oversizing.

b)     Meaningful: useful for the stakeholders’ needs, mainly from IQAS from HEIs, Quality Agencies, and the Higher Education community.

c)     Appropriate: meeting the common and shared quality standards, which, in an European context, are specified in the ESG, supported by ENQA and other relevant stakeholders.

d)     Reunified: harmonized set and compilated good practices already in use.

e)     Transversal: suitable for different countries, contexts, and types of HEIs.

According to this aim, the QIS construction methodology has been divided into four main phases, as shown in the figure below.

Phase 1

First, it was analyzed the current situation of QMS in 36 HEIs spread over 5 European countries (Portugal, Spain, Belgium, Lithuania and Italy), with particular emphasis being given to the quality indicators used in these systems. This work included desk research (analysis of different institutional documents, such as quality manuals, strategic plans and activities plans, as well as the institutions’ websites), combined with formal and informal contacts with the institutions included in the sample.

The sample was made of 21 universities, 4 polytechnic institutes, 2 universities of applied sciences, 7 schools and 2 colleges. While 27 are public HEIs, 9 are private ones. Regarding their size, the sample comprises rather small institutions, with less than 5,000 students (12 HEIs), medium sized ones, with a number of students ranging from 5,000 to 15,000 (12HEIs), as well as large institutions, with a number of students that goes well beyond the 20,000 students (12 HEIs). 12 of the institutions have only one campus, while 24 of them are located at multi campuses. Furthermore, most of the HEIs are comprehensive (24) with only 8 having a specific character. Finally, while 20 are located in metropolitan cities (Milan, Barcelona, Lisbon, Vilnius and Brussels), 16 are placed in regional cities.

The reasons for the selection of these institutions/QMS are varied, ranging from the characteristics of the institutions themselves (e.g. history, dimension, representativeness and relevance within the national higher education systems, good positions in international rankings, reputation), to the easiness of contact with relevant people within the institution (augmenting the possibility of collecting reliable and valid information on the QMS), the availability of public information on the QMS, including when searching the institutional website (e.g. on the process and the role played by each body, in a transparent way), the existence of well-structured and integrated governance and management systems, interested in promoting the quality of the nuclear processes and their results, ensuring the involvement of all stakeholders, or the maturity level of the QMS.

Overall, 223 quality indicators have been identified in the 36 QMS analyzed. The main characteristics of these quality indicators can be summarized as follows:

• 201 are quantitative indicators while 22 are qualitative indicators.

•  Processes covered:

o  85 - teaching & learning | 63 - research | 60 - relations with society.

o  2 - teaching & learning and research | 1 - research and relations with society.

o  3 - teaching & learning and relations with society.

o  2 - teaching & learning and research and relations with society.

• Decision-making level.

o  117 - strategic | 31 - tactical | 30 - operational

o  8 - strategic & operational | 1 - strategic & tactical | 6 - tactical & operational

o 17 - strategic & operational & tactical

Phase 2

The second main source to feed the instrument was relevant literature on Quality Indicators for QMS, and particularly in the HEI setting. Up to 39 unique and valid resources have been analyzed, which are of different types: scientific articles, project and institutional reports, books and other scholar publications and management documents. These documents provided up to 302 indicators. The publication of the selected documentation was recent (46% published in 2018 and afterwards), whereas 26% were published before 2011. Most of these documents were scientific articles peer reviewed (58%).

These indicators were also classified in the three categories according to its scope: 46 % were on “Teaching & Learning”, 25 % on “Research” and 14 % on “Relations with Society”. The remaining 15% were documents combining two categories or the three of them at the same time. According to the decision making level attained, 13% were strategic, 36 % tactical, 20 % operational and 31 % were combinations of the previous levels.

Phase 3

Combining the results of the first two phases, a corpus of more than 500 Quality Indicators was compiled and became the initial input for QIS’s creation.

Phase 4

The authors expertise was key for grouping, prioritization, and harmonization of the set of collected Quality Indicators.

The creation of the QIS was divided in three steps:

a) Grouping of similar indicators from the collected ones.

b) Prioritization of the most relevant considering the collection results and the expertise of partners.

c) The harmonization of their metadata.

A list of 94 groups or labels were set in order to distribute the indicators. An iterative process in which all individual assignation of each author were merged. Next step was assigning these 94 groups into the 15 standards previously agreed. At this point, some close indicators in terms of content, or very high overlap were discarded. Other indicators that did not fit in any standard were also dropped. Other indicators that did not fulfill the requirements agreed also were discarded.

The result is that from more than 500 collected indicators the QIS is finally summarized in 27 basic indicators and 29 as recommended.