Tags

Last year I was given the responsibility of organising the tendering process to acquire a University wide license for a on-line language pack for all staff and students.

RS 260The decision to buy a language pack was directly related to the aims of the University’s Development Plan 2011-2020.  “During the decade, the University has further strengthened its language instruction and has promoted study abroad.  RUL continues to teach language and culture associated with nine modern languages and also operates a highly rated English language school”. Also, the aim is that by 2020 over 50% of the student population study for at least a semester in a country that is neither the UK nor their home country, as part of acquiring their degree. Students will have access to the online language pack whilst they are abroad on their SPA.

The Self-access Language Pack would be accessible to all students and staff (up to 4000), on and off campus and fully integrated into Regent’s Blackboard Virtual Learning Environment. Also it would provide English Language support to the whole College as well as an opportunity for everyone to learn a wider variety of languages, starting at an introductory level and increasing to at least an intermediate level (defined as level B1 by the Common European Framework of Reference for Languages – CEFR).

I drew up a check list of essential and desired requirements of the language pack (see previous blog post https://totallyrewired.wordpress.com/2012/07/04/evaluating-rosetta-stone-and-tell-me-more/ ) and then looked at the different alternatives available within the budget. I invited three companies; Rosetta Stone, Tell Me More and EuroTalk to give demonstrations of their products. After a extended evaluation of all three products the decision was made to buy Rosetta Stone.

Regent’s University London will be the first be the first University to have a full site license (fully integrated into Blackboard) for all its staff and students.

Why evaluate?

Given the current economic pressures demonstrating how we make a difference as teachers, lecturers or Learning technologists is even more important. Throughout Higher education at the moment there is a relentless drive to improve results and raise achievement. Coupled with this is (in the UK) is the rise in tuition fees that have raised expectations about what learners can expect from their learning experience. If we believe that we make a difference we need to be able demonstrate how we do it by effective evaluation of our teaching methods.

In a wider context, Chelimsky (1997) suggests three possible purposes for evaluation. Firstly to check, or audit – to assure those who funded the project that the project has done, produced and achieved what it planned to do and done these things to an appropriate standard and in an appropriate way. (Summative evaluation). At the end of the academic year I will provide the senior managers at RUL quantitative data on Rosetta Stone’s usage. . Second, to improve the project or venture. ‘Evaluation can be a form of consultancy and, as such, do a lot for enhancing the thinking and work of those being evaluated’ (Knight, 2003). In other words the evaluator is a respected outsider or critical friend. (This is formative evaluation). In terms of the Rosetta Stone package I have no direct control over but there are a number of things that can be changed and improved upon, such as Rosetta Stone’s integration with Blackboard or staff training in its use to support classroom teaching. Thirdly to know or understand – what is working and what isn’t, how, and above all why, in order to improve the activity being evaluated. The last of these points is concerned with not just how a project is not working but to explore the reasons why it is not working. Here it would be interesting to explore the learning theories that underpin Rosetta’s notion of ‘Dynamic immersion’.

Evaluating eLearning.

Rob Phillips et al, in their recent book Evaluating e-Learning (2011) start their discussion of how best to evaluate a e-learning project by outlining their Learning Environment, Learning processes and Learning Outcomes (LEPO) Framework. First, the learning environment provides the context in which the learner works. Therefore it encompasses both the physical and ‘virtual’ learning space of the learning.

Secondly, the Learning Processes, which can mean both the participation in learning activities (completing tasks, in groups or individually, interaction with the learning environment) and the internal (cognitive) learning process such as Problem solving, reflection etc. In the guide ‘Getting started with Rosetta Stone Manager’ the basic learning method of Rosetta Stone is described as ‘Dynamic Immersion’.  “Rosetta Stone uses ‘Dynamic Immersion’ a method that simulates a real life emersion experience and relies on active participation. All lessons are taught on the target language ….Tasks and activities engage the learners in the learning process. Throughout the programme new information is presented systematically so that words or grammatical forms are easily identified and understood”.

Thirdly, the learning outcomes which are the things learners can actually do as a result of their engagement in a course of study. In terms of language acquisition, how effective is Rosetta Stone in supporting the face to face teaching and acquiring new language skills?

 Evaluating the users experience.

To fully evaluate the learners experience of using Rosetta Stone I intend to use a mixture of quantitative and qualitative methods.

Quantitative Data.

As the System Administrator of Rosetta Stone here at RUL I will have access to the ‘Rosetta Stone Manager’. This is the ‘behind the scene’ recording of who is using it and who and how long they are using it for to generate ‘usage reports’. The usage reports displays the following information of learners by selective groups or classes; hours spent on Rosetta Stone, overall score for activities completed, curriculum and language levels, overall progress for through the displayed curriculum, time spent on the curriculum or the activity. This is the type of information that is relatively easy to quantify and is easy to present in a form that will make sense to the senior managers who want to see have often the software is being used by the students.

However, there are also limitations with this type of evaluation. It does not tell us what the learning experience is like for the learners. Are they enjoying the learning process? To what extent is it supporting their classroom activities? Is the learning experience a dynamic process that builds motivation and increases the desire to learn another language? How much language can be learnt and to what level?

Qualitative Data.

Rosetta Stone will be used in the university to support language teaching from an introductory level to intermediate and advanced levels. There are 9 foreign languages taught, plus English as a second language. I think it will be interesting to contrast its usage, first between staff and students and secondly between students studying in a language class and those using Rosetta but who are not taking formal classes in language tuition.

1. Evaluating staff’s experience of using Rosetta Stone

Evaluation of Rosetta Stone usage. Interviews with Language teaching staff and non-Language teaching staff.

  • Identify number of users of the product and how long they have spent using Rosetta Stone and how much progress did they make on the system for their chosen      language.
  • Interview some language staff – how useful did they find the software – do they think it has it helped their students with their language learning / confidence /      improved their speaking skills?
  • Create an online survey/questionnaire for students/staff to complete about their use of Rosetta Stone. Analyse the responses to this survey.

2. Evaluating students experience of using Rosetta Stone

  • Focus groups/Interview some ‘language students’ – how useful did they find the software – do they think it has it helped with their language learning?
  • Look at T1 module feedback forms for languages to see if there is any improvement in student feedback since the same time last year. Is there anything specific to use of Rosetta Stone.
  • Focus groups/Interview some non-language students – how useful did they find the software – do they think it has it helped them to learn a language? what will they use      these language skills for?

Processing the results of this research is a time consuming process but the analysis will be helped with the use of ‘Nvivo’ which I have used on previous research projects.

 

References.

 

Getting started with Rosetta Stone Manager.  Available at: http://www.lerum.se/Documents/F%C3%B6rvaltningssidorna/Utbildning%20och%20barnomsorg/Verktyg%20i%20skolan/manager_administrators_guide.pdf?epslanguage=sv (last accessed July 2013)

Chelimsky, E. (1997) Thoughts for a new evaluation society, Evaluation, 3(1), 97-118.

Knight, P. & Yorke, M. (2003) Assessment, learning and employability. London: The society for research into Higher education and Open University Press.

Phillips, R., McNaught, C., Kennedy, G. (2012) Evaluating e-Learning: guiding research and practice. Abingdon: Routledge.

Advertisements