May 312012
 
Screenshot of Dave Pattern's Webinar

I am attending a webinar from Dave Pattern, Library Systems Manager at University of Huddersfield today.  He will be talking about their Library Impact Data Project and I will be taking notes throughout. The usual caveats apply that this is a liveblog and potentially a bit for of typos etc. The session has been organised by Scotland’s Colleges and you can find out more about this webinar and other events on their Webinars page.

The hashtag for this project is #lidp and the whole team is available via Twitter. We have been using the data we collect for a number of years. We are often guilty of making a lot of assumptions about how students use our resources. We have also been trying to measure the impact the library has on student attainment and experience, particularly in the current funding era.

Defining Usage Data – this might be circulation transactions, e-resources usage and also building entry stats indicating use of physical facilities. JISC recently completed an activity data programme and they broke it into user activity data and attention data.

About 6 years ago we started to look at our own library data, with a view to thinking about those students that do not tend to use the library, to look for trends, see what has changed… things like seeing a huge increase in use of e-resources. Not a huge declince in print usage despite this. For the librarians we have been pulling out quite detailed data on usage. We use a system called MetaLib for eresources. In 2005/6 you see very little use of eresources but each year it increases. We also see stats for our Horizon system – our catalogue – and for Sentry, the system for access. It’s useful to look for patterns and trends here.

Screenshot of Dave Pattern's Webinar

One of the other things we wanted to do was identify students who would want library training etc. When we talked to student reps they asked if there was a link between how students do and how much they use the library. When we looked at the stats we didn’t see any significant difference between use of the library and grades, we’d just had a refurb so sort of hoped for a pattern but just reenforced that the library is for all.

BUT when we looked at borrowing levels there were double the number of loans for those who got a first and those who got a third. e-Resources seemed to have a linear relationship. Definitely something there. Looking in more detail we could see that some students who never used the library got a 1st! Many logins to the eresources system was strongly correlated to grades of a first (borrowing more than 180+ books over 3 years). This is interesting stuff…

Complex graph of usage and grade

As we started to show these around we had questions about whether this is just at Huddersfield or the same elsewhere. And we had lots of questions about whether what was happening was statistically significant. So we applied to the JISC Activity Data Programme. Our project aimed to prove there is a statistically significant correlation between library use and attainment amongst other things.When the project kicked off we set up a hypothosis to test – that more book use results in better grades in a statistically significant ways. We had multiple prokect partners who each were asked for appropriate data (most could provide at least two of the three types of data requested).A few challenges here. Concerns on the UK Data Protection Act 1998 (University Student Handbooks), Data anonymisation challenges, data extraction (one security system regarded this is as their own data), finally we did manage to release the data though ??

Broadly speaking it transpired that Huddersfiels’d usage data is broadly representative of UK HE Libraries. But there was an issue here… continuous data made it hard to indicate a true correllation. Some libraries could include renewals as well as original loans, but not all.

So if we look at our own data from Huddersfield… it appears that students who get a first are using an increasing number of books and resources. One of my favourite graphs shows that students who get a first are already doing so in their first year, they seem ahead already. Are those students also used to using public libraries? Do they have better study skills? Really interesting to more in this area.

Slide showing stats on first year usage of resources

Looking at non and low usage of materials… Seems to be a real clustering around non and low usage… and indicates a real need to do more to get students aware of the library.

Non and low use of resources vs grade


Phase 2 of this work is now underway and we are looking at Huddersfield data specifically and using final %age marks and looking in much more depth to try and show significance and causation. We are also looking at UCAS entry points and if improvement between entry points and degree result includes any sorts of patterns. Is non or low usage a warning of drop out or incompletion numbers?

 

Looking at EZProxy stats. Here we seem to have a strong correlation in the average usage and final %age grade. Same for item loans. Looking at the time of day accesses we see most heavy use in the core daytime hours. In the very late night we see those with a third using more eresources than their peers. It’s not a significant effect but notable. And we see early morning as the time when those likely to get firsts are using eresources and physical library resources.

Usage and grade by hour of the day

Looking at nationality we see that the UK born users use books more often whilst EU students seem to have much higher use of resources and really higher use of PDFs and eResources, particularly those from New Europe.

Usage by nationality

Students who drop out appear to be 10 times more likely to not use eresources. This may be an indicator of those that are struggling with courses. Want to look at stats like VLE usage etc. to see what else we can learn here.

It seems that prior library usage before university is pretty important. There is a statistically significant correlation between HE library activity and final degree outcome. And evidance elsewhere seems to confirm this. See ACRL Value of Academic Libraries work for instance – in the US they don’t tend to collect usage data because of issues such as the Patriot Act.

So, what next?

More data to be released. Keep an eye on the blog: http://library.hud.ac.uk/lidp. And keep in touch via Twitter (#lidp).

Q&A

Q1 – Nicola) Is that the stat on use of non e-resources also suggesting something about physical stock and relevance not just usefulness of eresources?

A1) Yes, it probably is… we know students do borrow more books and use good resources… but what value they get out I’m not sure. We need to do focus groups… it may be that if you pick good resources you may borrow fewer books but are more focused on what you need to download etc.

Q2 – Lynn) Dave, assuming all your students joined in 1st year? Any research as to library use/attainment for those joing eg 2nd year via college articulation?

A2) We haven’t considered that but it’s a really good question for us to think about.

Q3) Are many students distance learners at your uni? Does this influence eresource usage?

A3) We do have data at partner colleges and abroad – the eresources data would be really interesting to look at and see if there are big differences.

Going back to that first year data… there is a sense that it’s almost too late to teach students information literacy at that stage, they are already set in their ways and studying practices

Q4 – Nicola) Have you had any feedback from students about this data?

A4) We have made findings available to academics. We have had them trying to scare their students a bit but we have been careful to say that it’s early data though. Will be interesting to

Q5 – Lynn) With library loans, have you looked at number of different titles (ie breadth of usage) as opposed to simply number of loans?

A5) Potentially we could use that data. For a number of years we’ve had recommendations in our catalogue… perhaps we should do a “students who got a first and borrowed this also borrowed that”. And

Q6 – Nicola) Do you have any way to compare the usage in the library with usage of non library stuff e.g. Google Scholar/repositories elsewhere

A6) A lot of the eresources data is via EZProxy. We moved to Summon rather than MetaLib. We push both on-campus and off-campus use of that. We obviously can’t capture use of Google Scholar etc. so I think that will need to be something for focus groups I think. Would be interesting to look at 3rd year students, yet to graduate but likely to get a first, and see what their habits are, how they are using things like Google Scholar etc.

Dave will be at CILIPS Scotland Conference in a few weeks so do say hello if you are too!

 

 

Apr 252012
 
Screenshot from the ViTAL Webinar on "Flipping"

This lunchtime I have been attending a ViTAL webinar (held via Adobe Connect here) on “flipping” which they describe as “the video-based approach that emerged in the US and has raised huge interest in the UK and Europe”. There is more background in an article on flipping in the UK edition of Wired this month: http://www.wired.com/geekdad/2012/04/flipping-the-classroom/

Our presenter for this session is Carl Gombrich, Programme Director for UCL’s undergraduate interdisciplinary degree: Arts and Sciences BASc. Carl has Maths, Physics and Philosophy degrees and is a professional opera singer!

 

Screenshot from the ViTAL Webinar on "Flipping"

Screenshot from the ViTAL Webinar on "Flipping"

So here are my notes from Carl’s talk:

This is my first webinar – in fact I’m really pretty new to technology in general. He’s currently setting up an interdisciplinary degree of Arts & Sciences. It’s a major launch of a degree for UCL, it starts with 80 students this year. And we’re really thinking in this climate – and the recent changes to student fees, funding etc – about how we can best engage our students. I am entirely focused on teaching – I’m not involved with the REF at all – and I am desperate to do something better than huge lectures to foster engagement with students.

So about 18 months ago I started to hear about “Flipping” with the launch of the Khan academy. I’m a fan of those and would have loved to have had access to those videos at school. So I wanted to think about how lectures could share content and do this ahead of the lecture so that contact time is really saved for stuff that really counts.

The idea of Flipping comes from about 2007 – Bergman and Sams although some say they have been doing this for much longer – where there was real questioning of why we gather students together in person in a room. I wanted to think about their model and think about how to make contact time more useful, more valuable, so wanted to add polling to the face to face sessions so that lecturers can really get a handle on what students want, to foster engagement through questions and why that’s a good idea.

You can see a 12 minute presentation on my blog about the kit I used but lets just run through quickly. I used the Echo 360 lecturcast system – the tool used at UCL. You just download it and it’s a few clicks to get started. I used a bog standard camera and mic – the built in options on laptops are fine. The lecturecast system could pair an image of the speaker with any materials. You can switch between the materials as you want. You can use MS Office docs along with any bespoke images you want. The exciting thing about video is that you can make it pretty interactive. You can stop the material, you can replay it to engage more with something you don’t understand etc. The other kit I used was a tablet – a little graphics tablet – I use Wacom/Bamboo – it just lets you underline, circle, highlight content as you want.

Actually after the presentation I did for the HEA I have learnt far more about how you do this stuff… some of the technologies are far more fluent, allow realtime noting etc. I think PowerPoint for Mathematics is a real killer. You have to see the process as you do in music, it’s visual, you learn best from seeing people thinking aloud. I think Khan does that so well, not everyone agrees but I think he’s a really excellent teacher.

So, that’s what I did. I think that sort of model is transferable to any old-style model. Any old knowledge transfer system should be transposable to the idea of making videos in advance. But if you want to do that what do you do?

Well you need to record lectures in advance – at home, in the office, event outside. Use lecturecast – this bit is easy. Then you ask your students to view the lecture before the timetabled lecture slot. Now that, of course, may not work… So… ask your students to upload 3 questions each with timings based on the video lecture (to indicate when questions arise) and send these questions to Moodle – everyone can see the questions that way and you also have evidance that the student has viewed the lecture and raised a question. Cognitively I think that’s very interesting but inevitably there’s also a command and control aspect here about ensuring students are taking part. And my colleague Matt Jenner has helped me set up some basic tracking in Moodle to know that students are participating. The other thing we dop is take a poll of the most popular, say 10 questions.

I was recently at a conference with Thrum, the man behind the Audacity web programming course at Stanford which you should look at as that is truly revolutionary, and he also uses polls and questions to gauge student need, to shape the teaching.

So back to what to do… the final stage is to go to the timetabled lecture slot with questions – interact, debate, solve problems with the students. That’s where it’s really pedagogically interesting. You get to know the students really well, you can get a sense of learning type (if you believe in those) and you can really get a sense of how they are doing. It’s a way to get back to more personal relationships in learning.

So the good things about this approach are that students can interact with lecturers on questions that interest them, problems they want to work through. Students can be split into groups and perhaps support each other (see Mazur) but the key bit is they get their questions answered. Better relationships are built up especially around mentoring, contact, etc. And submitting questions could be part of formative assessment so that everyone is involved in learning and that can really soldor that engagement. And that old lecture time can be used for summative assessments – short tests, blog pieces, group work, longer assessments etc.

And the bad things here?

Well some are concerned about the kit working, technology issues. But I am really a middle aged late adopter and I can manage, we owe it to our students to engage in this stuff and it’s easy to do.

“It will take me double the time – 1 hr to record the lecture, 1 hr for the interactive class” – well perhaps in the current fee climate we owe it to our students to spend that extra time. But being kinder on the lecturer you also do not have to rerecord the lectures every single year but you can rerecord as needed to update or correct anything. And like writing lecture series you can do this far ahead of term. And colleagues have pointed out to me that we don’t have to spend a full hour video – a series of shorter more intense videos might be better and allow you to really focus on the threshold concepts. I don’t know how much more work this would be – maybe 25% more in the first year but reducing over time. But the gains are so much more than any additional time one puts in.

“I hate working to camera” – I loathe working to camera, particularly I hate still images. It’s a real issue for me. But it’s where we are with the technology… I remember my grandparents generation refusing to use the telephone! We all use email now and I think video is really becoming that ubiquitous. We just have to go through that process of getting used to it.

“Students and colleagues will make fun of me or say inappropriate things about my style or the lecture” – this is falling away because of the ubiquity of video. There is an issue with trolling but it’s not a big issue with this sort of video. BUT there is a good reference in my slides here – students have other things to do, we need to rise above those concerns.

References:

And references from the community in the chatroom here:
Q via John Conway (Moderator)) We’ve had a comment about the Panopto product – it lets students annotate notes and save to their own profile, and they can then make them available online for discussion.
A – Carl): Lecturecast isn’t well used yet in UCL. The idea of polling questions in advance is the reflective thing – students can go away, come back, think about the questions. We learn when we aren’t thinking directly on the topic so those gaps can add some real advantage.
Q) What is the difference of Camtasia and Echocast 360?
A – Carl) I think they are versions of lecturecast systems but fairly similar
A – John) Lecturecast is the concept really. Camtasia is a vendor of several sets of softwares. It’s something that we’ve had to be careful to phrase things – see the previous presentation on Lecturecasts on Ning.
Q) What about doubling student study time?
A – Carl) Well we know the thing students most value about studying at university is the contact time and so I think making that more useful will be appreciated. But perhaps it does require reshaping of expectations. perhaps you shave reading time to allow this video engagement. I don’t think you add too much time and hopefully it will be something they value.
Q) Our experience at Aberystwyth is that lecturers are not keen to videoed and students are not that bothered to see them. The audio and the content are the key thing.
A – Carl) Speaking to colleagues there I have a sense that a face is really important for younger students – perhaps children/young people not adults. The audio is the key bit for older learners. But I’m not hugely sold on video particularly. The ability to draw on the screen, to show the process etc. is really important here.
A – John) We have some material on the usefulness of capturing body language – adding additional feedback and information here.
A – Carl) Matt here at UCL has made another point – there’s something on my blog about “do you need to see your lecturer”. I think a few minutes to see them on video may be enough. If you never see/meet someone in the flesh you lose something BUT once you have that, once you have a sense of them as a human, then you can go back to the virtual and use that sense of them to really better understand what you are engaging with online. I think there not meeting/meeting via video/meeting in the flesh. Both of the latter are important but perhaps we don’t have to do as much in person as we once did.
Comment) In teaching negotiation video is hugely important
A – Carl) That is a hugely important point I hadn’t considered – any teaching that requires understanding human interaction – psychology say – will really make the
Q) Do you make any of your material available under an Open Educational Resource model?
A – Carl) I’m not sure if we’ve worked out the economics of this… if a lecturer makes their materials available for free what does that mean for the lecturer and for the institution, doesn’t it undermine that? I certainly don’t want to release them all before students get here. Maybe I’m just not brave enough here!
Q) Many lecturers are used to presenting materials but some are not used to being facilitated? Should we offer training on how to be a good facilitator? For instance would they need training on how to handle debates in the classroom?
A – Carl) Gosh, maybe. I’ve always done my teaching the way I do. I suppose I just expect teachers to have those skills and I’m lucky that setting up a new degree I can choose my colleagues here. But if you don’t naturally engage with clickers, with new technologies that have proven pedagogical value then yes, you would want/need access to training.
Q) What is you say something untoward on camera?
A – Carl) That’s a really interesting issue and is far beyond just education. I would hope that we would really learn to handle this as society in a sensible way. As educators we should lead though. I think if you make a comment to a group of 200 people that isn’t being recorded should be fine with doing that when you are being recorded and be backed up by your institution.
Q) Could you use some of the captured content in the classroom?
A – Carl) I think you would not want to show long clips but with a bit of planning using a clip related to the key questions as you are addressing those.
Q) What feedback have you had from students?
A – Carl) As I mentioned earlier I am setting things up for September 2012 so I don’t have research base for this teching method yet but we do have research that what students value most is contact time. We are also trialling some split screen head to head debates for students to engage with
Q) How will you evaluate this approach?
A – Carl) Some open ended questions at the end of term will probably be the way to do this. I am cautious about over scrutinising students – I just think that’s the wrong atmosphere for what we’re trying to achieve.
Really most of the first and second year undergraduate courses you might be teaching are already on the web in some way – via existing educational materials online. But you really add the value meeting the teachers face to face and discussing and engaging with them.
Comment) Isn’t this the same as reading before a lecture?
A – Carl) Yes, some of my colleagues have said that! But the medium is really changing. In a way we’ve always asked students to do pre-reading – and they have rarely done that. But I think video, I think polling students is a qualitative shift that makes this difference.
John) Thank you all for coming along today and if you have any further questions and comments do take a look at the ViTAL (Video in Teaching And Learning) Ning community:  http://vital-sig.ning.com. We will address any questions raised there on Ning and perhaps in a webinar in the future.  The next webinar will be on video and pedagogical design.