Cluster of indicators
The Activity 2 of SMART-QUAL project (Literature Review on Quality Indicators for Quality Management Systems) aims to collect quality indicators from relevant literature sources. The activity, together with Activity 1, contribute to collect and cluster a relevant corpus of quality indicators used in QMS and/or highlighted in specialized literature. This corpus will be the framework to build our final Quality Indicators Scoreboard (QIS). The methodology followed in this activity consists in documental analysis of relevant sources recommended by the project’ experienced partners, trying to cover all the scope of SMARTQUAL project, namely, all the three missions of university. Up to 39 unique and valid resources have been analyzed, a mean of 4,3 resources per partner. These resources are of different types: scientific articles, project and institutional reports, books and other scholar publications and management documents, and partners collected up to 302 indicators from them, that will be added to the 223 quality indicators collected in the previous activity. In this report, we are going to describe the results of the Activity 2 and cluster the Quality Indicators found, analyzing its coverage and identifying the relevant learnings we must consider in final QIS. The codification and integration of indicators will be a task of the next activity. This document is supplemented with a spreadsheet where all the indicators are gathered.
Results
Sources analyzed
Among the total of 49 resources analyzed, 39 are unique and valid resources. In the table below some metadata are described.
| Type of resource | % |
|---|---|
| Scientific article | 58% |
| Report | 23% |
| Management doc. | 8% |
| Others | 11% |
| Date | % |
| 2020-2018 | 46% |
| 2017-2014 | 18% |
| 2013-2011 | 10% |
| Older | 26% |
| Peer reviewed (articles) | % |
| Yes | 100% |
Scientific article is the type of resources mostly collected, followed by reports. We can find reports about other projects and studies, and institutional reports from bodies like the European University Association, the European Commission or different Quality Agencies. Management document, books or conference proceedings are other type of resources considered. Almost a half of resources analyzed where published/released in the last 3 years, so we can confirm that we have analyzed quite updated sources that lower the risk of ignoring current trends and uses in Quality Indicators. Besides, all scientific articles are peer reviewed, ensuring a minimum level of quality. Reports are not normally peer reviewed but, as the intellectual output of an institution or project, its multilateral nature guarantee a level of consistency. Finally, we would like to point out some strengths of the analyzed resources:
- Some relevant projects are considered. For example, the SQELT project is another Erasmus+ initiative aiming to build a core dataset focused on Learning & Teaching indicators. 4 out of 9 partners have considered outputs related with this project, representing an undeniable precedent of SMART-QUAL project. This project identified more than 800 indicators used in HEIs. Other relevant projects or studies carry out a collection and clustering of quality indicators in different contexts: Anglo-Saxon, LatinAmerican and European.
- Management documents are also analyzed, focusing on quality indicators currently used by HEIs in different context and getting a point of view different than the scholar one.
- It is easier to find resources about Teaching & Learning, but different resources collected are focused specifically on Research and Relations with Society. This will help us to cover all the project scope.
- Up to 12 resources analyzed propose a subset of common quality indicatorsto be used in different scopes (6 for Teaching & Learning, 6 for Research, and 5 for Relations with Society), regions (Europe, the Nordic countries, Latin-America or world-wide) or idiosyncrasies (open science, politecnic HEIs, sustainability assessment…). This 12 resources will be a good starting point to build the SMART-QUAL QIS.
Quality indicators collected
During this activity partners have collected 302 quality indicators, that will be added to the 223 quality indicators collected in the previous activity. Some characteristics are described below. Above 75% of the quality indicators are classified as Quantitative by partners. In Figure 2 we can see the predominance of Teaching & Learning indicators as it was expected, but a remarkable fact is that the other scopes are also covered. There are also some combinations of scopes, as some indicators could be suited in the monitoring of more than one university missions, depending on the approach considered.
| Scope | % |
|---|---|
| Teaching & Learning | 46% |
| Research | 25% |
| Relations with Society | 14% |
| Combinations | 15% |
As far as decision-making level is concerned (Figure 3), project partners have classified the indicators mostly in tactical and combination of levels. This make sense as the strategical use of indicators depends on the approach and objectives of each HEIs, and the tacticaloperational nature are more generalizable between HEIs. The high amount of combinations of decision-making levels (around a third of indicators), indicates also different uses per each indicator. Despite this fact, some indicators have been identified in a strategical level and will be a good basis for the future SMART-QUAL proposal.
| Decision-making level | % |
|---|---|
| Strategic | 13% |
| Tactical | 36% |
| Operational | 20% |
| Combinations | 21% |
Finally, in Figure 4 the coverage of ESG is described. Almost every standard is covered by indicators. The biggest amount of indicators is related with Research & development (70), followed by indicators about Student admission, progression, recognition and certification indicators (43), and External relations (40). It would be interesting to carry out further research of indicators related with Public Information and Cyclical External Quality Assurance, though the latter is quite more a qualitative condition determined by national legislation than a quantifiable standard.
| ESG (A3ES adaption) | Indicators |
|---|---|
| 1. Policy for quality assurance | 14 |
| 1.1 Policy for quality assurance and pursuit of quality objectives | 14 |
| 2. Quality assurance in the nuclear processes of the institutional mission | 203 |
| 2.1 Design and approval of programs | 4 |
| 2.2 Student-centred learning, theaching and assessment | 31 |
| 2.3 Student admission, progression, recognition and certification | 43 |
| 2.4 Ongoing monitoring and periodic review of programs | 6 |
| 2.5 Research and development/ragated research and high level professional development | 70 |
| 2.6 External relations | 40 |
| 2.7 Internationalisation | 9 |
| 3. Quality assurance in the management of resources and support services | 32 |
| 3.1 Human resources | 19 |
| 3.2 Material resources and services | 13 |
| 4. Management and publication of information | 20 |
| 4.1 Information management | 13 |
| 4.2 Public information | 1 |
| 5. Periodical assessment | 9 |
| 5.1 Cyclical external quality assurance | 0 |
| 5.2 Cyclical internal monitoring, evaluation and continuous improvement of the QMS | 9 |
A list of a subset of the indicators collected, clustered by university mission adressed, can be found in the Annex. Partners have collected the most relevant indicators based on each resource approach or conclusions, and their own qualified experience. As the codification and integration of indicators will be a task of the next activity, the indicators in the list are neither treated nor harmonized.
Principal findings to consider in QIS
To sum up, we can identify some main findings from the literature review carried out in this activity of SMART-QUAL project:
- Adequate variety, updating and relevance of resources analyzed. Almost a half of resources analyzed where published/released in the last 3 years, and relevant antecedents, like SQELT project, are considered. Up to 12 proposals of common indicators to be used have been identified.
- The resources analyzed and the quality indicators collected cover the scope of the project. The risk of undervalue Research and Relations with Society missions, in front of the prolific topic of Teaching & Learning, is correctly managed. As far as ESG coverage is concerned, Tab 4 shows also the variety of quality dimensions covered.
- Further conceptual leveling will be required between partners in order to agree on main classificatory elements. This need is identified in the fact that the same indicator has been classified in a different nature, decision-making level or mission, depending on the resource and/or the partner background. Far from being a limitation it is a challenge that an harmonized and international QIS should address if it pretends to be useful and understandable.
- The latter is related with the idea that the SMART-QUAL project is not a theoretical project about indicators, quality or Higher Education. Our aim is to collect and analyze what is being done and propose a harmonized, synthetic and applicable scoreboard of indicators for QMS based on resources, best practices and relevant experiences. Accordingly, SMART-QUAL will have to put forward a well-founded proposal of, for example, relevant indicators for a decision-making level.