15 Friday Apr 2016
Posted in #OER16, Apps, CPD, Evaluation, OER's, Open Access
03 Friday May 2013
Posted in Evaluation, Technology Enhanced Learning, TheBrain
Interview (Transcript) with Mikko Arevuo about TheBrain
Intro
1. When did first become aware of the TheBrain?
Mikko: It must be about 4 years ago, I was a doing a paper with a quite a few reference documents and what I wanted to have was a repository to drop them in. All at the same develop the paper and the structure of the paper and I wanted to have a repository to tie it into the structure. I was looking for some sort of mind mapping type tool and I just stumbled them on the web.
Chris: You didn’t see a demo of it or anything?
Mikko: No
Chris: just found it by searching or googling it?
Mikko: Right
2. Why did you want try it out with your students?
Chris: What were you thinking at the time?
Mikko: What I find is that the students don’t read enough and just assigning chapters and books or a lot of the web resources that comes with textbooks and articles, that they do not engage. If I present a pretty massive thing on the screen and I’m able to put materials onto wiks, some videos, articles, weblinks and so on to the actual programs and actual slides. My thinking was that they would start exploring materials and hopefully reading around the topic. As you know I teach history and management and there’s strategies happening all around us so when I’m reading the Financial Times or the Wall Street Journal, I pick this other snippets and I put it into a relevant week so I’m bringing the materials to the students.
Chris: We’ll develop some of those things in a bit more time as well.
3. a) How easy was it for you to start adding content to the TheBrain?
Chris: When you first started it.
Mikko: Yes 4 years ago. It’s relatively easy because it’s just a click and drop. The difficulty is to understand the linkages because it works on notes because you have parent, you have child thoughts, you have system thoughts.
Chris: But that’s with the structuring but the initial putting stuff into TheBrain?
Mikko: Yes very easy but it becomes a model if you don’t organise it.
b) What were the main difficulties you faced when uploading content to the TheBrain (if any)?
Mikko: The uploading is very easy, you drag and drop.
Chris: So you’re uploading files, word documents..
Mikko: Word documents, I can put in pdfs, I can do weblinks, videos, I have my blogs, the whole thing is there.
Chris: You don’t have problems with any of that, putting that in?
Mikko: No it’s fine it’s a very solid and robust.
4. Have you experienced any technical difficulties using the TheBrain? (Did the students have any problems accessing the TheBrain)
Chris: Did you have problems
Mikko: No.
Chris: What about your students
Mikko: We’ve been running it for 2 terms. The first term the way, the licence I have is that I can, I create the brain, in this case the business strategy module brain. And I’m the administrator of that so I can then send emails through the settings function on the brain to my students to access the materials as a reader and email a no reply email is generated by the brain. And then effectively what the students do is they should just click on the link and fill their details and get into the system. Now I have not spoken to the brain but a number of students were not able to log in. and my thinking is and it is something we perhaps do is talk to the brain and perhaps get some the materials to them. Is there might be some sort of a time limit after which the email link becomes obsolete.
Chris: Stuff is working.
Mikko: Yes stuff is working. So last term I had to ask a number of students to re send their email to me at different email addresses so then I can invite them in.
Chris: Is that roughly a big percentage of students?
Mikko: No and again, based on my completely honest scientific sampling of this, this is where the students were not really engaged in the classes. I think they were in the back end and just didn’t bother and didn’t get the links. Or not getting them to sign in.
Chris: We’re not really sure whether it’s them who can’t actually be bothered to go in and register TheBrain or whether it is something wrong. Maybe we need to investigate that a little bit.
Mikko: But I’ll come to that in a minute because this term things have been working tikity boo.
Chris: We can come back to that.
Teaching
5. Has TheBrain affected (or could affect) the way you teach in the classroom?
Chris: Maybe describe what you do in the classroom first?
Mikko: What I do is open up the WebBrain which is the cloud functionality of the brain in the classroom. And in effect you have the totality visible so you have all the materials on it, so for example today’s the first week of the term and I have a one note that goes through or details of the content. And today is to say for the next 3 weeks we are going study the following areas and as I said these are the top line headings and will be drilling down into the following components. I’m clicking on the notes, opening up, drilling into the content so for the next 3 weeks they know what’s coming. And then I say let’s move on to today’s lecture. So I have one note that has the lecture slides so I open up with 1, lecture 2 and out pops the overhead but also on that weekly programme or today’s lecture I have links that come which I have stored down in videos and companies in the and so on. News and so on. So I have other materials that links into that specific session. So I say to the students by the way, we’re going through the slides the formal lecture but there’s additional information that relates to what we are discussing. And the resources are here. So going back into the question you asked me earlier, what I’m trying to get the students to do, often they are just looking for the PowerPoint slides but now what they do is click on that slide content they also get additional links to additional materials within that session.
Chris: Just again as an aside the first time you use it with students, I presume it’s the first time they’ve seen it as well?
Mikko: Yes.
Chris: Do you get any initial reaction in the classroom?
Mikko: They sit up, it’s something that looks different. It grabs attention and that was part of this thing is, is grabbing the attention. Something slightly different they can face. The question is then is when they start using it.
Chris: Which is another issue, which relates to the next question.
6. How does the WebBrain compare to using Blackboard?
Mikko: My personal opinion is, because in the way I think is more in pictures, maps and connections and I can deal with the messiness and dynamic nature of thoughts that come into these notes. To me that is the better way of thinking of what I’m doing rather than in a linear filing system which is just clicking into files and opening them up. Because those files are not really interlinked are they? and that to me is the biggest difference. That’s the personal way of learning.
Chris: If you got that way of thinking then a proportion of your students will have that way of thinking, it’s across the board.
7. What do you think were the benefits of TheBrain to the students and yourself?
Mikko: The benefit to me is I use the brain in a lot of my work for various different projects so the brain is always open on my laptop. Coming in the morning and I open up the financial times or there’s an article about Apple’s market capitalisation yesterday in Forbes and I say what an interesting article and I just click on it and dump it into the relevant week in the student brain so it’s instantaneous. So rather than say I have to open up Blackboard and need to think about what files I want to put it in, it’s a natural process because I’m teaching, I’m lecturing, I’m developing materials, so when I see something I can do very easily the transfer process.
And for students, because I’m able to make that connection because when I see something it goes in instantaneously into the student brain. They probably get more materials from me then if I had to say well ok today I’m going to do something on Blackboard. I open up the Blackboard and say what kind of materials I’m going to go and put on there and start searching for it. This is slightly a different way, I see something it goes in.
8. In your opinion what were the main disadvantages of using the TheBrain?
Chris: Is there any disadvantage to you?
Mikko: Over the past 4 years I have become so comfortable using it that in fact I was toying with the idea of putting all my files into the brain rather than having anything on my Mac.
Chris: What about the students?
Mikko: The students I think it’s the messiness because it is slightly messy because it is a dynamic knowledge repository and they are ‘thoughts’. And I had a student today who said ‘I was on the brain but I was only able to see the videos’ I said you probably just clicked on one and didn’t spend some time on it. It takes some time for the students to understand it and spend time exploring the linkages and if you open up to explore it all the materials are there, then yes its a real mess because there’s so much stuff in there, all visible. So if you are just looking for one piece of information without having gone through all the various different top level information that I have categorized, it’s not as simple as going ok, week 1, week 2, Blackboard ‘boom’, open up the file.
Chris: We may come back and look at some of the solutions to that if there are, I don’t know.
9. What changes are you planning to make next Semester to the TheBrain?
Chris: Compared to last semester, are you delivering the same module, obviously to a different cohort of students, what are the changes you are planning to do.
Mikko: The first change was to give to students a user guide.
Chris: You’ve done that already. Did that get a reaction at all?
Mikko: I think it went down well because in fact they had something tangible first of all, they were given a 2 page user guide that you wrote and that prompted them to take action. So the first lecture was on Monday and this morning, about 2/3 of the students have already logged in.
Chris: How does that compare to last semester, if you remembered at all?
Mikko: I think it’s significantly higher. Nobody has come to me saying ‘I have a problem I cannot log in’, it’s brilliant. And the stats that is provided to the owner of the site, I think I have so far had about 80 openings and 450 clicks. So it shoots up. So that was a major improvement.
The second thing what I’m doing is that if you look at the bottom of the screen you can put notes in, so you have the presentation mode or just have the blue space with the thoughts. You can adjust that and in the bottom you have notes so what I’m doing is I’m also writing a very brief couple sentence summaries of the materials that will be covered at each of the lectures and if I add an additional link I say ‘by the way I have added a link about this company or a video about this company that relates to the …
Chris: That gives students more guidance around TheBrain.
Mikko: Yes. And again that is, ok Mikko is writing about this video that is relevant to the lecture so maybe they will actually play it and that is exploring and reading around the topic more.
Chris: Which were the objectives in the first place.
Mikko: Exactly.
10. Do you use the TheBrain to evaluate how much work the students are doing on your module?
Chris: I suppose you are looking at the stats so you’ve already answered that in one sentence.
Mikko: I look at the stats but I’m not able to really have access to what they are doing with the brain. I can see they have access to that but I am not able to see the total or which areas they go to or which students and so on.
Chris: If you had a different licence, would it provide that?
Mikko: I don’t know
Chris: I suppose we can do a bit more investigation about that and see if it can give us that type of information.
Mikko: Yes
11. In your opinion does the TheBrain engage students with low levels of motivation?
Chris: Because that was one of your objectives, you said at the start you wanted to provide the information to your students and you wanted to do it in a more engaging way. Now that sounds like it’s doing that to some extent but what about the students who are really, I wouldn’t say totally cut off from what you’re doing, they want to get through the module with minimum effort whatsoever, is it touching those students?
Mikko: I can’t tell you. As I said there’s still 1/3 of students who have not signed on, are those students who don’t care. What we have done is driving students on this platform because we are saying we are not making these materials available on Blackboard, driving the other students to the brain because we’re not making the materials available on Blackboard. So at minimum I would expect that students if they come to a lecture they would then also open up the lecture slides. Then again there are students in this place who never come to lectures they only come to seminars because they take attendance at the seminars. I think students who make the effort of logging in and make the effort of at least opening up some of the files. My thinking is that the materials because it is accessible is there although it might be slightly overwhelming, will encourage students to explore. Students in the past who just open up the lecture slides probably have a bit more incentive to see what else is there. But students who are completely disengaged they don’t open up Blackboard or they don’t open up anything, they don’t open up lectures. They overstep that.
Workload
12. Has the introduction of the TheBrain changed your overall workload?
Mikko: It has changed in a way that it’s constantly in the background. Because like I said, if I’m reading something I might be at home at 10pm in the evening and something pops up and I say oh that’s an interesting article to put into my week 5 lecture. So click and put it in, so in that sense the work is there constantly me in a background.
Chris: So the answer is yes but it doesn’t feel onerous – I guess you’re wanting to do it yourself you can see the advantages as a pay back for you
Mikko: Yes. In that sense with the Blackboard. I just say I’m going to do a Blackboard session now, I’m going to create this stuff with Blackboard I can upload my lectures etc.
Chris: This changes the way you work, with the brain you’re constantly there doing it in small bits, where as if you’re working with Bb and VLE you have to sit down with a pc somewhere and do it in a time slot. And maybe change the hours you’re doing it.
Of course if you have the brain opened up in a small window on my laptop and I have the financial times open on the other window and there’s an interesting article, I just drag it over and drop it into the brain. And of course with Blackboard you’re not able to do that sort of thing you need to have the uploading coming from one program to another, it is actually more timely, time consuming on Blackboard then it is on the brain.
Future needs
13. Would you like Regent’s College to give you more support to develop your TheBrain? If so, what further help do you need
Mikko: What I like to do is to allow students to do some assignments on the brain and for that I need to have is what I think is called the team brain.
Chris: How would an assignment work on TheBrain.
Mikko: Good question, let me explain that to you. So I have a few course work assignments and one of the assignments is thesis analysis of a firm within a script and it is effectively a desk research assignment and the students have to collect quite a bit of material from various different sources to do the written report. And with the technology students work at different locations and very often with the assignments they have to physically come together and play with a piece of paper, ‘I printed that out, you bring what you printed out, and this is what we’ve done’. The team brain is a collaboration tool, so although I have not explored it that much, what you’re able to do is you have a ‘create a work space so people can drop them I’m doing an analysis of this industry, I’m looking at these competitors, and I’m looking at bringing materials and they will be able to drop this.
Chris: I see so you would have students working in a team, each team would have its own brain.
Mikko: Absolutely, for that project. And they are going to start developing issues, with note on that, recommendations, and note on that. And start putting materials and then all the materials, the knowledge, the documents and the web links and so on, are on that brain. At some point they’ll just need to say, ‘how are we going to then put this together and write it up?’
Chris: And then you would have a collection of those little brains and case studies around those issues?
Mikko: And I think that would be a fantastic assignment. Because again, don’t forget the other part of this, my drive for the brain is to show students that in business, as it is in life we are increasingly having to deal with more and more information and we need to be able to put that information, put that thinking into somewhere. And technology is an enabler. It’s there and they are in their twenties and I’m a dinosaur compared to them but all this thinking will, why not take advantage of the technology while it’s there? It would be great to get that type of licence.
Chris: Definitely need to check that out actually and see how much that will cost and the practicality it doing that. Is that shifting the way you are using the brain from a repository of information into actually an interactive thing that students are working with it would be quite a different way of using it isn’t it?
Mikko: Everything student would have a study group and they could create their own virtual team with the platform and repository
Chris: I will have to check that out in a bit with you.
14. Would you like to make any other comments about the TheBrain?
Mikko: It takes a while to get comfortable with it, but again when we were talking, I have all kind of stuff I put in the first year or 6 months I used it. And I deleted them because they were messy and so on and so forth, and I guess our brains are messy in the way we hold this information. And there is a technique to start thinking about how things work. We are trained very often in the west to think in a sort of linear way.
Chris: Even if our brains don’t work like that because we’ve been trained in that way we end up using that as a way of learning.
Mikko: Exactly. Little boxes of open it up. But here all the time this is top of the roster here You know blank canvas and you start throwing ideas.
Chris: How did you learn this, through trial and error?
Mikko: Yes just through trial and error. I mean the good thing about the brain is that they have a significant repositories or use of videos and for various different. And I used to listen to them, the webinars. You have people who write books, they do the book structures on that and that in fact is what draw me to this in the first place. I do my doctorate research on that. And also the book structure, the book brain, is something I think we can also pass on to the students for them to do on a collaborative basis, do a big term paper. I think it’s a great platform. I haven’t across anybody in the UK who uses it in the academic context so I hope they will carry on developing it, and do well. I think what we need to check out is obviously is if you’re going to take this further at the university is to see, what sort of company it is how solid it is, there been upgrades coming all the way in the past 4 years, so its pretty solid to me.
05 Tuesday Feb 2013
Posted in Blackboard, Evaluation, Technology Enhanced Learning, Turnitin, VLE's
I have been supporting one of our lecturer’s here at Regent’s College, Anabel Gutierrez who is using Turnitin Peer Review.
In my role as a Learning Technologist I did a demonstration of Turnitin Peer Review to staff in September 2011. Anabel saw the demonstration and set up Peer Review assignment with her students studying the Information Management module. 
The immediate feedback from the students was very positive. At the end of the module when the assessment had been completed I went into the lesson and spoke to the students. What they really liked about it was the opportunity to see their colleagues work. They said that they really liked to see how their work compared to others in the class.
This academic year (2011-12) Anabel repeated the assignment with a different class doing the same module (Information Management) which is a module on the BA in International Business.
Evaluation. First, student Peer review questionnaire was created and given to students. Turnitin Peer Review Questionnaire Results. Secondly I interviewed Anabel Interview questions on Turnitin Peer review for lecturer. You can hear the interview here http://soundcloud.com/rowellc/usining-turnitin-peer-review?utm_source=soundcloud&utm_campaign=share&utm_medium=twitter&utm_content=http://soundcloud.com/rowellc/usining-turnitin-peer-review or alternatively read the Transcript of interview with Anabel Gutierrez
This term we plan to do the same evaluation again, but this time with post-graduate students on the MBA. So we will do a questionnaire with students in week 9 of the course and I will do a similar interview with Anabel when the module is complete. it will be interesting to compare the undergraduates results with the post graduates and compare the similarities and differences.
At the end it would be great to turn this into a conference presentation or journal article…watch this space!
10 Monday Dec 2012
Posted in Blackboard, Evaluation, Technology Enhanced Learning, VLE's
Tags
Blackboard, Distance Learning, Education, Educational assessment, Formative assessment, Higher education, Virtual learning environment, VLE
As more research is done into the teaching and learning by the use of VLE‘s such as Blackboard (Kramarski and Mizrachi, 2006; Saurers and Walker, 2004, Rodriques, 2006, Lobel. 2005)) there is a realisation that there are positive benefits to a more student centered perspective. Kramarski and Mizrachi’s (2006) study of secondary school maths classes found that students who engaged in online discussions outperformed their face to face counterparts in maths and real life tasks. Saurers and and Walker (2004) looking at business writing classes and Rodrigues, et al (2006) investigation into a biology class found similar results. Furthermore Lobel, et al (2005) discovered that students interacted more directly with one another rather than the classroom where they tended to relate more with the tutor.
One critical success factor for establishing a functioning VLE is usability, relating to the ease by which the VLE can be used by staff and students is also an important factor in assisting and increasing collaborative learning. Bradford et al’s 2007 study of Blackboard highlights the fact that it has been criticised for being hard to learn. They quote a 2003 survey of staff and students at the University of Wisconsin who found the Blackboard VLE harder to use than they expected. Faculty members found the VLE “time-consuming and inflexible”. Cole Lewis’ (2006) study, based on a UK university found similar problems regarding usability for some its students. Although this evidence is contradicted by a UK study of first year Business Studies students who state that “‘usability’ is confirmed as the most favoured feature of Blackboard VLE…” (Roberts 2002) It must also be noted that these studies were based on a earlier version of Blackboard (Version 5.5) which has changed considerably in recent years.
Yet there are still many who are sceptical of the effectiveness of VLE’s. There are some barriers for the VLE to help develop a student centered learning perspective, it lacks the opportunity for face-to face interaction and socialising which develops trust (Gibson and Manuel 2003) and therefore some individuals may prefer to operate on their own. Also,
“The Blackboard learning management systems provide mechanisms for students to interact directly with each other and the instructor. But these platforms have yet to move beyond fairly basic communication features. What has changed is the instructor’s ability to track student’s use of the class Web site; number of messages posted, number of messages read, and how many times various pages or sections are accessed” (Coopman 2009). Mullen (2002) goes further and argues that whilst this may seem to be a measure of how much students are participating in the VLE it is in fact recording something different, simply whether they are using the course or not. “Students are treated not as learners, as partners in an educational enterprise, but as users”, Mullen (2002).
From a much more critical standpoint, Rose (2004) argued in her critique of VLE’s that their structure, layout and tools used to teach the students is not value free. She states that “there is no acknowledgement of the fundamental transformations that must be wrecked upon content imported into platforms such as ….Blackboard, nor of the fact that the very structure of these systems constrains instructional possibilities and decision-making”, Sandvig (2006) develops this point saying that once the structure is built into a VLE it becomes very difficult to change it.
One recent study on the evaluation of Blackboard’s use provided some interesting results. Yaneske and Bingham ‘s 2006 study of second year multimedia students at the University of Teeside found that mature students were shown to have a preference for face to face classroom techniques (which is not surprising) but their research also revealed that lecturers and students expectations of Blackboard were mismatched with each other and with its intended usage. In terms of usability both staff and students mentioned the need for a less confusing user interface. Students were also frustrated that not all modules and their resources were available on Blackboard. Also students had no single point of access for course related information (individual exam time tables, exam results etc). Staff felt that Blackboard did not effectively enable them to implement online tasks which prompted collaborative approaches to learning. The survey results suggest that some students use blackboard inappropriately as a distance learning tool with 22% of students saying that Blackboard enhanced learning by allowing them to work anytime, anywhere without the need of a tutor. The courses were not designed as distance learning courses with materials provided online intended to support face to face methods, not replace them. The finding suggest that both staff and students could benefit from being prepared in how to use blended learning and also in assessing how effective the technology has been.
In terms of assessment in In Higher Education there is a large body of evidence that formative feedback has an impact on the learners experience (Black and William, 1998; Knight and Yorke, 2003; Hounsell 2003). However, over the last 10 to 15 years larger student numbers, increased modualisation and increased diversity have all affected formative assessment. The knock on effect of these changes is that there an been an increased emphasis on summative assessment as opposed to formative assessment, a reduction in the times when students can meet lecturers individually and less time for lecturers to mark and give feedback to their students. It has also meant that both students and lecturers are increasingly concerned with increasing grades rather increasing the degree and type of learning that is taken place. The two are not necessarily the same thing! (Gibbs 2006).
David Nichol’s article (2006) on increasing success rates on first year courses clearly shows how learning technologies can be used to increase and redesign formative assessment. Based on the Re-engineering Assessment Practices project (REAP) it highlights two case studies, one in Psychology and the other in Mechanical Engineering and shows how a standard tool available in every VLE (a discussion board) can provide many opportunities to increase feedback both from the lecturers and from peer and self assessment. It article goes further by discussing the possibility to strengthen learner autonomy and further relax teacher control when discussing the use of electronic voting systems and the use of multiple-choice questions within the VLE.
References:
Black, P. 7 Wiliam, D. (1998) Assessment and classroom learning. Assessment in Education. 5(1), 7-75.
Coopman S.J. (2009) “A critical examination of Blackboard’s e-learning environment”. First Monday, Volume 14, Number 6 – 1 June 2009. Available at: http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/viewArticle /2433/22
Gibbs, G. (2006) Why assessment is changing. In C. Bryan & K. Clegg (Eds), Innovative assessment in Higher Education. London; Routledge.
Gibson C.B & Manuel J.A. (2003) Building trust; effective multicultural communications processes in virtual teams. In Gibson & Cohen (Eds) Virtual Teams that Work. San Fancisco. Wiley & Sons.
Kramarski B. and Mizrach N. (2006) “Online discussion and self-regulated learning; effects of institutional methods on mathematical literacy”, Journal of Educational research, Volume 99, number 4, pp. 218-230.
Mullen M. (2002) “‘If your not Mark Mullen, click here’: Web-based courseware and the pedagogy of suspicion”, Radical Teacher, number 63 (Spring), pp14-20.
Nichol D. (2006) Increasing success in first year courses: assessment re-design, self-regulation and learning technologies. Available at: http://tltt.strath.ac.uk/REAP/public/Papers/DNicol_Ascilite_26oct06.pdf
Rose E. (2004) “‘Is there a class with this content?’ WebCT and the limits of individualisation”, Journal of Educational Thought, Volume 38, number 1 (Spring) pp. 43-65.
Sandvig C. (2006) “The structural problems of the Internet for cultural policy”, in Silver D. and Massanari A (Eds). Critical cyberculture Sudies. New York: New York University Press, pp. 107-118.
Sauers D. and Walker R C. (2004) ” A comparison of traditional and technology-assted instructional methods in the business communications classroom“, Business communications Quarterly, volume 67, number 4 (December),pp 430-442.
Lobel M., Neubauer M. and Swedburg R.(2005) Comparing how students collaborate to learn about the self and relati0onships in a real-time and non-turn-taking online and turn-taking face-to-face environment, Journal of Computer-Mediated communication, Volume 10, Number 4 (July), Available at: http://jcmc.indiana.edu/vol10/issue4/lobel.html,
Yaneske E. & Bingham A. (2006) Mismatched expectations of staff and students towards Blackboard. Part 1 Paper 721 ALT-C Research Proceedings 2006
03 Wednesday Oct 2012
Posted in Blackboard, Evaluation, Tricks of the Trade, Turnitin
Last week Dr. Anabel Gutierrez, Senior Lecturer in ICT at Regent’s College gave a short presentation on using peer review assignments based on her experience of using this type of assessment here at Regent’s College. Her talk addressed these three issues:
10 Monday Sep 2012
Posted in Blackboard, Evaluation
Abstract This is a short paper looking at the pros and cons of different ways to evaluate a VLE (Blackboard). Blackboard was initially analyzed using the ‘official’ statistical reports within Blackboard and the ‘Bb Stats’ building block. The quantitative data generated by these systems was complemented by a qualitative study of staff opinions about Bb. Recently we have implemented Google Analytics as an additional method of gathering data regarding institutional Blackboard usage.
First I will give a brief overview of the data provided by ‘Bb Stats’ and the inbuilt analytical tools provided by Blackboard. Secondly I will provide of overview of the qualitative methodology used to evaluate staff’s perceptions of Blackboard. Thirdly, I will describe the process of installing Google Analytics and how we interpreted the statistical information it provided.
Google Analytics, Bb Stats and the inbuilt tools provided us with a variety of quantitative information. The quantitative reports enabled us to see how many students visited the VLE and how they interacted with its content. It also showed us the kind of browsers they were using and geographically where the users were coming from. The qualitative data based on staff interviews gave us much more information on how the VLE was being used. I will show how Google Analytics data compares and contrasts with the inbuilt analytical tools and ‘Bb stats’. Finally, I will also look at some of the questions about Google Analytics limitations as an evaluation tool, such as, can a piece of software designed for commercial motives be useful for learning technologists?
Overall using a combination of quantitative and qualitative data enabled us to have a much better understanding of how the VLE is being used, which in turn informed decisions about the formatting and appearance of the learning resources in the VLE. The results determined the type of training we provided to our staff and students and informed discussions with senior managers to show how the VLE is contributing to the successful outcome of the institutions strategic objects.
04 Wednesday Jul 2012
Posted in Evaluation, Language tuition, Rosetta Stone
This week I invited two companies to come into the college to give demonstrations of their products. On Wednesday morning ‘Tell Me More’ gave their demo and this was followed by ‘Rosetto Stone’ on Friday. I’m not a linguist so I wasn’t sure what to expect. I found it difficult to learn a language at school so the prospect of an online self study programme didn’t seem that appealling…. to be honest.
Anyway the process started a few months ago when I sat down with the Head of Languages, Amparo and we discussed what we actually wanted from a languages self study pack. We came up with a list and then I visited other interested individuals in the College. I contacted the VLE manager, the project manager in IT, deputy head of the English Language School, head of Learning Reources and a representative from the library team. They all made contributions to what we required and then we ordered them into what is ‘essential’ and what is ‘desireable’. As a result we produced a chech list that I emailed to the two companies before they did the demo:
Self-access Language Packages tick list RS
I was quite surprised by the demonstrations. I wasn’t expecting great differences between the two products but they both had very different approaches to language learning. So the evaluation is not just about the technical differences of the two products. Evaluating the technical differences is a relatively easy thing to do. . So for example, we can easily assess how many languages are provided or to what level they can provide support. What was clear to me was that these two products fit the needs of different users within the college. Tell Me More was much more closely alligned to the rigors of learning languages in a academic environment. It is much easier to identify parts of its progamme that match what the students are doing on their course. Whereas, Rosetta Stone is a much more emersive way of learning a language where grammer and vocabulary are picked up unconciously as the learner passes through the learning activities. I could see that the former might be more suited to the higher level language courses whilst the later would be more applicable to rest of the College’s non-language provision.
I was also really pleased with the turnout for both sessions. We had over 20 members of staff who came to both sessions. Whilst we had a few technical specialists the majority of the audience was made up of language lecturers. They all gave feedback on the two products and I look forward to reading their comments.
Anyway, this is just the first stage of evaluating what is ‘out there’ for this type of product. My intention is that we look at a couple more other products before we make a decision. We are possibly going to have a look at ‘Eurotalk’ in September and maybe a couple of others at the same time and make a final decision in October. It would be great to have the language packs fully installed and ready to go for Jan 2013!
….for more information on this project see my second blog post:
https://totallyrewired.wordpress.com/2013/10/01/evaluating-rosetta-stone-part-2-my-future-plans/
07 Thursday Jun 2012
Posted in Blackboard, Evaluation
Tags
Devising interview questions to evaluate staff’s perception of a VLE was a difficult process. Here is what I ended up with:
Interview questions for lecturers
Starting Bb
1. a) How easy was it for you to start adding content to your module area? b) What were the main difficulties you faced when creating a new module area in Bb (if any)?
2. Have you experienced any technical difficulties using Bb?
Staff/Student interaction
3. Do you think the students have an increased expectation of your availability since Bb was introduced?
4. Do you believe that Bb can replace a part of your face-to-face contact with your students?
5. Do you think that Bb affects the attendance in your lessons?
6. In your opinion, has Bb changed the quality of interaction with your students?
Communication
7. Has the introduction of Bb affected your relationship with students?
Teaching
8. Do you think that Bb enables you to become more or less student-centred in your teaching?
9. Has Bb affected (or could affect) the way you teach in the classroom?
10. How effective is Bb at supporting collaborative activities on your course?
Assessment
11. How has Bb affected the way you assess students work?
12. Have you changed the amount of feedback to students since the introduction of Bb?
13. Do you use Bb to evaluate how much work the students are doing on your module?
14. Do you use Bb to identify any students who might be “at risk” on your module?
Workload
15. Has the introduction of Bb changed your overall workload?
Future needs
16. Would you like more training or on-to-one support to develop your Bb module? If so, what areas of Bb would you like to develop?
17. Would you like to make any other comments about Bb?
01 Friday Jun 2012
Posted in Evaluation
Tags
Given the current economic pressures demonstrating how we make a difference as teachers, lecturers or Learning technologists is even more important. Throughout Further and Higher education at the moment there is a relentless drive to improve results and raise achievement. Coupled with this is (in the UK) is the rise in tuition fees have raised expectations about what learners can expect from their learning experience. If we believe that we make a difference we need to be able demonstrate how we do it.
We need to show how we evaluate what we do. Kirkpatrick (1992) provides a good starting point on how to approach an evaluation project. He suggests four levels of evaluation. Originally developed for training staff, Kirkpatrick’s model can be further adapted to the evaluation of most educational projects:
1. What are ‘participants’ (students/staff) immediate reactions to the event, often obtained with a a simple ‘tick box’ questionnaire. 2. What have participants (students/staff) learned from the event, perhaps also obtained with the end of-event questionnaire. 3. How and how far have participants (students/staff)in the event later used what they have learned. 4. The most important question – what has been the impact of changed practice on student/staff learning?
Baume (2010) paraphrases Kirkpatrick (1992) in much more simple terms:
It is the first of these three questions that are relatively easy to address. It is possible to ask the participants if the liked the activity or training they were asked to undertake. If the participants enjoyed the activity they are more likely to use and engage with it. The second question can again be evaluated, after the event by asking the participants directly. The third issue can also be evaluated by asking the participants, possibly using a questionnaire or interviews or even monitoring changes in their working practices.
“The final question is arguably the most important but in an educational context probably the hardest to answer”, As Baume (2010) points out, “how can we be sure that any improvements or increases are attributed to the changes that we are evaluated? Our students results may have increased, which we can measure by analysing and comparing present results with previous years but can these changes be attributed to say, the increases use of the VLE?”
It might be really difficult question to answer when there are so many variables but its something we have to show. This might not result in quantifiable data in terms of improved exam results or increased retention rates, it might just result in an ‘improved educational experience’ that can only be captured by recording the story of the student’s journey as they aquire the knowlede or skills they are learning.
References.
Baume, C., Martin, P. and Yorke, M. (2002) Managing Educational Development Projects: Effective Management for Maximum Impact. London: Kogan Page.
Baume, D. (2010) ‘How do academic developers make an impact – and how do they know they have done? SEDA Conference Presentation 2010.
Kirkpatrick, D. L. and Kirkpatrick J.D. (2006). Evaluating Training Programs (3rd ed.). San Francisco, CA: Berrett-Koehler Publishers
Useful Links:
