Department of Computer and Systems Sciences
Stockholm University, Sweden
E-mail: ulf.olsson@su.se
This study focuses on teachers' awareness of quality issues in relation to Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG). Semi-structured interviews were conducted with 20 teachers at six Swedish HEIs while they developed open courses (MOOCs). The interviewees' comments are divided into five categories, but the overall finding shows that the teachers were not part of any transparent quality assurance system. This was despite the fact that there were several examples of quality assurance work. The result relates to prevailing standards and guidelines for quality assurance ESG (2015) and Conole's (2016) characteristics of good learning. The question of the adequacy of a quality system for innovative activities is raised.
Keywords: Quality assurance system, ESG, MOOCs, Higher education
Sharing and openness are expected to be cornerstones for both teaching quality and a global perspective with regards to access to education (Weller, 2014). Accordingly, knowledge, information, technological communication and open educational resources used in a way that enhances the quality of education is becoming an expectation of teachers in higher education (HE) (Kirkwood, 2013; Janssen, Nyström Claesson, & Lindqvist, 2016). However, the adoption of these new demands at many higher education institutions (HEIs) has been slow (Nascimbeni & Burgos, 2016; Singh & Hardaker, 2014). The teachers interviewed in this study were expected to develop massive open online courses (MOOCs) using these new demands by engaging in internal and externally funded projects. Their initial attitudes were investigated in a prior study (Olsson, 2014) and were found to be highly positive overall. However, their ambitions were often difficult to see in their day to day work. This study is a follow-up of this, but with a focus on quality, and questions were raised about the teachers' awareness of quality assurance whilst in the middle of the open courses' innovation and development phases. The quality assurance awareness is related to the ESG described below.
The Standards and Guidelines for Quality Assurance in the European Higher Education Area, ESG, (ENQA, ESU, EUA & EURASHE, 2015) is a reference document for both internal and external quality assurance systems in HE. At a generic level, it gives reasons for quality assurance in Europe to be accepted and has been divided into three parts: Quality assurance agencies, External quality assurance and Internal quality assurance. The latter part is most interesting for this paper as the purpose of the current study is to investigate the awareness of any current quality assurance system among academics developing MOOCs. Most notably, it will appear in the component for internal quality assurance. A solid quality system involves several stakeholders in the process of designing education, and academics directly involved in a course design process should be significant players in a quality assurance task. To be involved, the stakeholders need to be aware of the system as a starting point.
The principles of ESG give guidance for the quality assurance system in Sweden, and other European countries. The Swedish Higher Education Authority's (UKÄ) new quality assurance system for HE has been based on ESG and is due to be implemented in early 2017. In line with the development of the ESG is broader access to HE and a change towards student-centered learning and teaching.
Although MOOCs have not yet celebrated their 10th anniversary, quality issues surrounding them have been frequently discussed (Kocdar & Aydin, 2015; Creelman, Ehlers, & Ossiannilsson, 2014; Daniel, 2012; Conole, 2016; Hayes, 2015; Hood & Littlejohn, 2016). This is likely due in part to the discussion of MOOCs that, for a period, was carried out outside of the academic field. Some argue that the new open and scalable courses are disruptive and have changed the preconditions for the whole sector, while others believe that it is just another hype around technology. Admittedly, the MOOCs had many registered individuals, but could easily be criticized for lack of interaction among the participants and other pedagogical issues. These criticisms have also been compounded by the fact that a majority of participants do not complete all of the tests and even fewer obtain a certificate (Jordan, 2015). MOOCs are not a solution to every educational need and Lowenthal & Hodges (2015) argues that it is an opportunity to rethink how to design and teach online courses rather than either a complete threat or gimmick.
A common skepticism of online learning has contributed to a number of quality assurance programs for online courses (Lowenthal & Hodges 2015), and there are a number of quality agencies and European initiatives focusing on quality assurance in MOOCs (Conole, 2016; Creelman, Ehlers, & Ossiannilsson, 2014). Some discussions have centered around whether MOOCs can use the same criterion for quality assurance as for other types of education that take place on campus (Hood & Littlejohn, 2016; Yepes-Baldó et al., 2016). These discussions are largely an extension of those focusing on the aspects of quality in distance education, in which categories of participants do not fit into the traditional model of a campus student. Quality assurance as a tool for enhancement is emphasized, and several recommendations highlights the value of using different criteria for the quality work. Martin Weller argues in a contribution to the EFQUEL MOOC Quality Project
“One last plea – MOOCs are still a new kid on the block. Let them make mistakes, let them be experimental, let people play and explore in this space without tying it down with the types of quality overhead we already have in formal education."(Creelman, Ehlers, & Ossiannilsson 2014, 83)
The relationship between quality systems and innovation comes to the fore when new types of activities are taking off, such as the development of MOOCs. To measure the degree of excellence of something demands an existing scale, but even with some scale, you cannot discuss if innovation and quality assurance systems are incompatible entities. Ng states, when discussing the school system, that
”Quality assurance in an era of diversity and innovation is a delicate affair and there is an inherent tension.” (2007, 246)
The aim of this study is not to discuss any aspect of quality of the MOOCs themselves. However, it is obvious that some issues regarding pedagogical methods, adequate guidance and feedback are quite different compared to ”traditional"courses. It can be discussed if Conole's (2016) characteristics of good learning are relevant in the MOOC setting. Those characteristics are: Encourages reflection; Enables dialogue; Fosters collaboration; Applies theory learnt to practice; Creates a community of peers; Enables creativity; Motivates the learners. Good learning is important as Daniel (2012) argues that MOOCs of bad quality can obstruct the development of open education. MOOCs must be shown to meet some of the same quality standards that other online courses are expected to meet (Lowenthal & Hodges, 2015). In the same way that quality assurance systems are relevant for all institutions, the question of MOOCs have become a present issue of every strategic board in the higher institutions' management.
ESG and current quality systems convey an ambition that quality assurance issues should be transparent and in all parts of the business, from individual teachers, administration, management, and external stakeholders and, preferably, in all phases of the educational pathway. The starting point is that quality assurance is everybody's concern and should be embedded in all activities, which means engagement of stakeholders of every category involved in a project is important. In that sense teachers developing new courses should be an active part of the quality assurance process at an institution. Due to this, the focus of this study is teachers that are highly engaged in developing open courses. The study poses the question: How aware are teachers of quality assurance systems when developing MOOCs? It is not an inventory of any quality assurance work at any level even though some teachers in this study pointed at quality assurance work conducted by other units and stakeholders at the University. It is nor any evaluation of the quality work of any existing or absent parts in conjunction with the MOOC development projects respectively.
A qualitative approach to data collection and analysis by using interviews was adopted and a purposive sampling method (Cohen, Manion, & Morrison, 2011) was used to access the ”knowledgeable people”. The knowledgeable people in this case were academics at six Swedish Universities. All 20 interviewees have been involved in the planning, production and delivering phases of MOOCs. The interview sample has representation from a range of disciplines and experience levels. The interviewee's contribution to the MOOCs' design and production varied from one minor ”part"of a larger course to a teacher/producer of all of the material in a course. The interviewees have combined experience from 10 MOOCs. The study is not conducted with the purpose of being able to generalise outside the sample and the result should be interpreted accordingly.
Thirteen semi-structured interviews were conducted face-to-face and seven by using Skype. Fifteen were conducted in Swedish and five in English. Any translated citation below is indicated. The interviews varied from 40 minutes to 100 minutes, but included some issues about personal incentives and ”going public”, which are not reported here. All interviews were transcribed in full.
Interview data was analysed inductively by using MAXQDA11. The transcribed material was coded in an iterative process and the interviewees´ standpoints were compared and categorised (Anfara, Brown, & Mangione, 2002). These were all possible to verify by data, as presented below.
The presentations of the categories below are not any ranking of a project's total quality of work as the quality assurance work was done at different levels. The focus is what the teachers involved expressed opinions about, and not any mapping of the quality assurance systems at the institutions.
The overall finding shows that the teachers involved in developing the MOOCs were not part of any transparent quality assurance system, even though there were several examples of quality assurance work, both by the teachers in the project and by other parties (internal or external) involved in the development of the courses.
Different kinds of support were promised when the projects were initiated as the projects were financed by internal funds or externally funded by a granting body. The support and advice from an academic development unit, a project manager or an external production company were described as being very important. Several decisions about quality issues were also taken by other staff or units, and this was described as a presumption for progress within the projects. It was clearly expressed in the interviews that the production made it necessary to collaborate with other teachers and parties.
The interviewees' comments about quality assurance work are divided into the categories below. The categories are the interviewees' expressions about quality assurance systems and are more or less overlapping. The name in the brackets is an attempt to give each category of quality assurance process a short name.
An internal review by colleagues can be done more or less without being outspoken as a method for quality assurance. At one department, this was explained as a way to work with video recordings in the project.
“We looked at each others and would help each other, and it is also the manager, project owner [x], who was in control and had the big picture and see if this is correct or not."(P11)
In this case, it goes hand in hand with “in-department production."The teachers were using their computers and software for several components of the production.
Several teachers had knowledge about the routines for course evaluations and standardized quality assurance activities at their institutions, but they did not follow the required routines. Problems with surveys adapted for campus courses and not for the actual open courses decreased the ambition to use them.
“..we have problems with our evaluation because [the institution] wants you to ask a lot of standard questions and earlier we had a tailor-made evaluation that I worked with and had easy to produce. When it is not your questions, I don't know, I lost interest in working with the course evaluation.” (P32, translated)
Another comment about the course evaluations was that they were very also sloppy handed when collected. As such, it was not a serious alternative to suddenly implement that kind of information retrieval.
Routines and infrastructure in use in the ordinary operation were not always considered usable for the new format.
”Oh, to gore me bloody against the IT department, one want to do things that are not standard, noo, it´s just a lot of work. And people who think that you should use this or that, no it is backward, I do not.” (P21, translated)
This teacher found that there were possibilities outside the IT department.
Quality assurance by checklists was used by the external platform providers.
”The server belongs to Canvas network, then we had a supervisor who was very kind and she asked us supervisors if we had any questions about any slide or what to do. She took part and went through all the pages, all modules, that everything is correct and nothing odd.” (P11)
This kind of quality assurance is a review that the course would follow acceptable standards of the platform and that the navigation and design issues were correctly implemented. In some cases, the platform providers communicated deadlines for the production and publishing process. Trying to keep up with the deadlines became a challenge as the work took a lot more time than had been estimated.
Don't know (Laissez-faire QAP) Some interviewees expressed that they did not know anything about any quality assurance system or simply ignore it. The quality issue was assumed to be handled by someone else in the project or organization. The project management or the institution needs to solve that issue.
“No, no .. I do not know if we have any quality system. Do you know?” (P52, translated)
This question exemplifies the teachers' vague idea about any quality assurance system, even if they know about some routines for new courses as in the next and last category.
Several interviewees commented that the syllabusis was usually required to be approved by a committee. Publishing material in the institutions learning management system (LMS) also requires approval. One unit was developing a new quality assurance process, based on the European standards, and the teacher saw no problems with a system that covered all kinds of education. Several teachers mentioned the help from the educational department and one teacher appreciated the possibility to attend a course about online teaching as professional development in parallel to his MOOC development.
The interviewees expressed that they learned a lot when designing and producing the courses. The need to be on schedule and to plan the course in detail was mentioned several times, and the material that should be put online must be of good quality.
”I feel pressure. Is the material I am creating, is it up to the standard?” (P11)
Aside from the view that the production of video and other material was more time consuming than expected, it was a challenge to get ”the essentials"out of the research. One teacher was already looking forward to the possibility to improve the course when it runs for the second time. One teacher expressed the opinion that
“One must be aware that if you work with an open education you let some quality go in both learning and in the specific interaction above all and in the learning process. But if you can accept it one can get more students.” (P52, translated)
However, this interviewee was the only one that expressed that the quality can be worse as a result of the new format. All other interviewees expressed that it was a lot of work and a new form that has its quality gatekeepers but not necessarily the traditional quality standards.
It is not surprising that the teachers' stories are focused on video recordings. The produced courses are among the first of its kind at the institutions and to convert presentations of short videos is a common way to start using media in teaching. Discussions of more student-centered activities were rare. It was not evident that MOOCs were popular among students, but the form was seldom problematized. The effort generated material for courses, and the teachers were satisfied, perhaps both because of and in spite of the fact there was more work than expected. One exception, however, is a teacher who was not completely satisfied with what was achieved.
”We are still to record a script and has a little material at hand. It is not so much more. It has not happened so much more. That's really the question of what can we do in 10 years."(P62, translated)
Despite this, most teachers mentioned different motifs for them to be part of the development of MOOCs but did not stress quality assurance nor as a control function, nor as a support for development.
The ESG Part 1: Standards and guidelines for internal quality assurance states with regards to the policy issue that a quality culture that involves all internal stakeholders at all levels should be supported (ENQA et al. 2015, Part 1:1, page 11). The above result shows examples of internal stakeholders as the education department, support unit, MOOC project group and also external stakeholders as platform providers and video companies. The outspoken awareness of general quality reviews or quality assurance were quite vague. Instead, examples of colleagues as reviewers and the above-mentioned platform provider as the reviewer were mentioned. A high level of ambition was built in all of the projects and the teachers were clearly focused to deliver a course that would be completely public and estimated to attract considerably more participants than existing courses. Teachers in one MOOC can ”meet"more students than they do in a whole career at the campus. The institutions and the external bodies also invested heavily in the projects. This is reflected in the stories of weekend work and that virtually all teachers express that it took more time than expected.
ESG part 1:1 also indicates that ”How the policy is implemented, monitored and revised is the institution's decision.” (ENQA et al. 2015, Part 1:1, page 11) The interviewees' thoughts about any quality assurance were outside the ordinary processes except for some teachers who mentioned the process of approving new or changed syllabus and publishing routines on the learning management system. ESG gives advice about the processes of the design work, how education should be delivered, evaluation and competence of teaching staff. ESG also recommends that the institutions should undergo external quality assurance on a cyclical basis. The recommendations consist of various perspectives and are clearly adapted to ongoing programs even if the teachers' competence should include “encourages innovation in teaching methods and the use of new technologies.” (Part 1.5, page 13)
The focus of this study was not to ask the teachers about the pedagogical values behind the design of MOOCs. However, almost no concerns about the possibility for the student to learn from MOOCs were raised except for discussion about separate parts of the course's design. As previously stated, Conole's (2016) characteristics of good learning are: Encourages reflection; Enables dialogue; Fosters collaboration; Applies theory learnt to practice; Creates a community of peers; Enables creativity; Motivates the learners. The interviewees have an engagement and ambition to deliver an open course of the highest possible quality, despite more work for them than originally expected. A structured quality review of all of Conole's seven characteristics seems, however, to be absent in this first run, and may not even be adequate in this initial phase and for the MOOC format. ESG can be interpreted as a framework for the ambitious, innovative and tentative ways of teaching and learning that may not fit in the first round of MOOCs.
The diversity of participants' inducements for participation is highlighted when MOOCs are criticized for high drop out rates. Conole's characteristics for good learning can hardly be relevant for all the participants as some registered participants use the MOOC as a site to get a brief clue about something interesting to them, while others are curious about how (other) teachers present the field. Some participants may also want to get some brief information on a subject. All these participants may not want to reflect, communicate, etc. as Conole describes as being good learning, even if her characteristics are highly relevant for good learning as such.
A quality assurance system that has the flexibility to embrace new activities can support the institution to be prepared for changes and new operating conditions, but that is eventually not possible by definition of innovative activity. The institution needs to choose a strategy on how to deal with open courses. Clarke (2013) suggested four possible strategies which all need quality assurance work:
The decisions taken by the majority of the institutions for the MOOCs projects in this study is to join an existing MOOC consortium, which reached many students. The institutions that got external funding for developing open education were more geared to their existing communication platforms. The latter institutions' ambition to reach new students and position their research worldwide were quite low compared to the first category. By viewing the different decisions, different conditions for the teachers' quality of work existed.
This study has shown the absence of teachers' (those engaged in projects developing MOOCs) awareness of any structured quality assurance systems although different reviews and checks are mentioned. This does not indicate the absence of quality assurance as other parties in the projects can be assumed to have a holistic picture of the process. However, the result shows that the teachers interpreted the quality assurance process more as an internal control system than as a tool for strategic change. The result raises the question of the adequacy of a quality system for innovative activities. Suppose that an activity requires full resources in order to develop. This makes Björn Stensaker's question highly relevant as a closing remark: ”How can we create QA systems that stimulate creativity, personal engagement, local initiatives, and innovation?” (Stensaker, 2009, 2).
Björn Stensaker's question above (Stensaker, 2009) is in line with the concern by one of the teachers.
“But if there is a project and we do not test different paths. What kind of academy are we then? Then we roll's just on one track, and then, we will, of course, do not get any answers to questions.."(P21 translated)
How can a quality assurance system support development of HE and thereby become embraced by innovative teachers developing new teaching and learning activities?
Anfara, V. A., Brown, K. M., & Mangione, T. L. (2002). Qualitative Analysis on Stage: Making the Research Process More Public. Educational Researcher, 31(7), 28–38. https://doi.org/10.3102/0013189X031007028
Bates, T. W. (2015). Teaching in a Digital Age. Tony Bates Associates Ltd. Retrieved from http://opentextbc.ca/teachinginadigitalage/
Cohen, L., Manion, L., & Morrison, K. (2011). Research methods in education (7.ed.). Oxon, UK: Routledge.
Conole, G. (2016). MOOCs as disruptive technologies: strategies for enhancing the learner experience and quality of MOOCs. Revista de Educación a Distancia, 50(2), 1-18. http://dx.doi.org/10.6018/red/50/2
Creelman, A., Ehlers, U.-D., & Ossiannilsson, E. (2014). Perspectives on MOOC quality - An account of the EFQUEL MOOC Quality Project. International Journal for Innovation and Quality in Learning, 2(3), 78–87. Retrieved from http://lup.lub.lu.se/record/4648237
Daniel, J. (2012). Making Sense of MOOCs: Musings in a Maze of Myth, Paradox and Possibility. Journal of Interactive Media in Education, 2012(3), p.Art. 18. DOI: http://doi.org/10.5334/2012-18
ENQA, ESU, EUA, & EURASHE. (2015). Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG). Brussels, Belgium: ENQA. Retrieved from http://www.enqa.eu/index.php/home/esg/
Hayes, S. (2015). MOOCs and Quality: A Review of the Recent Literature. Gloucester, UK: The Quality Assurance for Higher Education 2015.
Janssen, M., Nyström Claesson, A., & Lindqvist, M. (2016). Design and Early Development of a MOOC on ‘Sustainability in Everyday Life': Role of the Teachers. In W. Leal Filho & S. Nesbit (Eds.), New Developments in Engineering Education for Sustainable Development, 113–123. Springer. http://doi.org/10.1007/978-3-319-32933-8
Jordan, K. (2015). Massive open online course completion rates revisited: Assessment, length and attrition. International Review of Research in Open and Distributed Learning, 16(3), 341–358. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/2112/3340
Kirkwood, A., & Price, L. (2013). Missing: evidence of a scholarly approach to teaching and learning with technology in higher education. Teaching in Higher Education, 18(3), 327–337. http://dx.doi.org/10.1080/13562517.2013.773419
Kocdar, S., & Aydin, C. H. (2015). Quality Assurance and Accreditation of MOOCs: Current Issues and Future Trends. In Proceedings of Open Education Global 2015: Innovation and Entrepreneurship. Banff, Canada. 22-24 April, 2015.
Lowenthal, P., & Hodges, C. (2015). In Search of Quality: Using Quality Matters to Analyze the Quality of Massive, Open, Online Courses (MOOCs). The International Review of Research in Open and Distributed Learning, 16(5), 1–8. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/2348/3436
Nascimbeni, F., & Burgos, D. (2016). In Search for the Open Educator: Proposal of a Definition and a Framework to Increase Openness Adoption Among University Educators. International Review of Research in Open and Distributed Learning, 17(6), 1–10. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/2736/3974
Olsson, U. (2014) A preliminary exploration of operating models of second cycle/research led open education involving industry collaboration. European Journal of Open, Distance and E-learning. 17(1) http://www.eurodl.org/?p=archives&year=2014&halfyear=1&article=610
Ng, P. T. (2007). Quality assurance in the Singapore education system in an era of diversity and innovation. Educational Research for Policy and Practice, 6(3), 235–247. http://doi.org/10.1007/s10671-007-9018-x
Singh, G., & Hardaker, G. (2014). Barriers and enablers to adoption and diffusion of eLearning. Education + Training, 56(2/3), 105–121. http://doi.org/10.1108/ET-11-2012-0123
Stensaker, B. (2009). Innovation, Learning and Quality Assurance - Mission Impossible? In Presentation at European Quality Assurance Forum, Copenhagen (p. 2). Retrieved from http://www.eurashe.eu/library/quality-he/Plenary_I_Bjorn_Stensaker.pdf
Weller, M. (2014). The Battle for open: How openness won and why it doesn't feel like victory. London: Ubiquity Press. http://doi.org/10.5334/bam