Monday, 27 April 2015

EMA and the White Rose Learning Technologists' Forum 21 April 2015

I was very pleased to be invited along to the latest WRLT Forum on the 21st April 2015. One of the main drivers for my attendance was down to my involvement with Electronic Management of Assessment (EMA) here at the University of Sheffield; this particular event was dedicated entirely to this theme.

 It was held at York St John University’s Skell Building on a frankly rather beautiful sunny spring day, which made both the University and the City of York stand out even more; a very pretty location!

We were also extremely lucky to have with us Lisa Gray (Programme Manager) and Lynette   Lall from JISC (Joint Information Systems Committee) who would be conducting the EMA workshop with us. Lisa Gray has completed a lot of extremely important work in the EMA arena where Institutions, including ours, are really looking for the ways forward in the implementation of electronic assessment at scale.

We kicked things off with a brief introduction to the Forum by Roisin Cassidy (TEL Adviser) at York St John University followed by the main presentation from Lisa Gray.

JISC - EMA Project Presentation


Lisa Gray Presenting at the EMA WRLT - image courtesy of Sarah Copeland, University of Bradford

Lisa gave the forum an overview of JISC’s EMA project. The presentation started with some background and context, which included the completion of a 3 year technology enhanced assessment and feedback programme. EMA was one of four main themes within this programme, and this background work helped highlight that there were clear tangible benefits to the use of EMA, but that there were also some serious hurdles to jump over in terms of its implementation. This essential background work led onto the EMA study itself.

We were presented with the study’s headline findings which included charts around levels of EMA system integration, variations in business processes and the variability in the take up of e-marking and e-feedback.

In particular we were shown a chart that showed to us very clearly where the main “pain points” were in adoption of EMA.  The top spot on this chart went to system integration (lack of), with staff resistance coming in a close second. Perhaps not surprisingly, student resistance was last on this chart…

Figure 1 - Pain points in EMA

Image courtesy of  JISC. Taken from the "Electronic Management of Assessment (EMA): a landscape review" publication

The study then explored the reasons for the pain in EMA implementation, including the lack of a central joined up approach, and the fact that trying to implement EMA at scale exposes limitations in the technology.

Finally Lisa moved on to the excellent “assessment life cycle” model developed by Manchester Metropolitan University.  You can catch that, and their excellent suite of resources on assessment here.

Figure 2 -The assessment lifecycle


Image courtesy of Manchester Metropolitan University - reproduced under CC license (BY/SA)
Lisa explained how, by breaking down the components of assessment in this way, we can then map the main challenges of EMA into each area which highlights where in the cycle our EMA issues lie. As we might imagine a large portion of them reside in number 5, “Marking and Production of feedback”.

Workshop activities


For our first workshop activity we were asked to work in pairs or small groups. Working off a “challenge sheet” that outlined the main EMA challenges, we were tasked with  reviewing and ticking off the challenges that we thought were causing the most pain. We were then asked to join forces with a larger group to rank those challenges in order of importance (1 - most important, 5 - least important).

Having reviewed all the challenges our group thought that, although there were  no real level 1 (urgent and biggest impact) challenges, there were a couple of level 2 challenges that were the biggest hurdles:

  • Ability to handle a variety of typical UK marking and moderation workflows : We felt that this challenge encompassed a lot of the pain points in adoption of EMA. Certainly within our institution there are many local variations of workflows that, when you apply EMA, it highlights issues such as anonymous marking, double blind marking, and moderation of marks 

  •  Ability of systems to handle off-line marking: Currently, there is only really the Turnitin iPad app that can offer true off line marking. This is quite a limitation when we  consider that offline marking may well offer the biggest step forward in making EMA a solution, that makes marking at least as easy, if not easier than its paper equivalent.

The resulting group feedback to the room generated some really useful discussions around groups pain points, some of which was directed towards either the incompatibility between Institutional student systems and marking systems or the current inflexibility of the technology.

EMA Solutions and workshop 2

As part of the final workshop of the day we were asked to move into four groups to look at each of the projects, decide what detail we would like to see within them and then provide feedback to the room. I chose group 4: the EMA tool kit. You can view the EMA toolkit description and the other projects here.

EMA toolkit

Our group discussed the idea that such a resource should work reciprocally with local Institutional produced resources. For example at the University of Sheffield we have our recently devised TEL Hub  which is continually evolving.  We are looking to  build our own set of EMA resources within the Hub for our Institution, and the content that is within this resource should work in tandem with resources in a centrally devised hub. Whether this be policy, process guidance or case studies. Anything we develop locally could potentially feed back into the central hub.

In addition we felt that it would be important to for hub users to be able to utilise different views or “lenses” on the toolkit to encompass all the different stakeholders: academic staff, admin, learning technologist (or equivalent), and student.

Feedback and Finish

The groups provided feedback to the room regarding the separate projects and three institutions were given a brief moment (due to time restraints) to mention their EMA themed work .

Paul Dewsnap at Sheffield Hallam University: Great work on looking end to end at their assessment and feedback processes:

Joel Mills at the University of Hull: Their use of Sakai (ebridge) to ensure that e-submission from students is captured successfully even if Turnitin is encountering issues.

Phil Vincent from York St John University: Their EMA policy development, and in particular, describing how you can reduce staff resistance to EMA through the deployment of two monitors!

Sarah Copeland from the University of Bradford: The University policy that requires e-submission where practical and for electronic feedback to be given within 20 days. The Faculty of Health studies policy of anonymous e-marking and use of core technologies: Pebblepad, Blackboard and Turnitin for areas of EMA.

We would like to thank once again, our hosts St Johns University of York, for providing with us a fantastic location and to Lisa and Lynette for a fab workshop that has helped provide some much needed light at the end of a dark EMA tunnel!

If you wish to get involved with the White Rose Learning Technologists' Forum, you can subscribe to our mailing list:

Please also see our Google+ Community page  at the link below:

Hope to see you at the next meeting!


1 comment:

  1. This comment has been removed by a blog administrator.



Related Posts Plugin for WordPress, Blogger...