Sep 282015

Today I am at the National Library of Scotland for a Clipper project workshop (info here). Clipper is a project to create a content creation tool for multimedia, with funding from Jisc.

After an introduction from Gill Hamilton Intro it’s over to John Casey who will be leading this through the day…

Introduction – John Casey

The tagline for the project is basically Clipper 1. 2. 3: Clip, Organise, Share.

We want your input early on in the process here but that means we will be trying out a prototype with you – so there will be bugs and issues but we are looking for your comments and feedback etc. The first outing of Clipper was from 2009, as a rapid development project which used Flash and Flex. Then it went to sleep for a while. Then we started working on it again when looking at Open Education in London

Trevor: I’m Trevor Collins – research fellow at the Open University. My background is very technical – computer engineering, HCI but all my research work is around the context of learning and teaching. And we have a common interest in HTML5 video. And my interest is working, with you, to ensure this will be helpful and useful.

Will: My name is Will and my background is engineering. Originally I worked with John on this project in Flash etc. but that’s really died out and, in the meantime HTML has really moved on a long way and with video in HTML5 we can just use the browser as the foundation, potentially, for some really interesting application. For me my interest today is in the usability of the interface.

With that we have had some introductions… It is a really interesting group of multimedia interested folk.

John Casey again:

This project is funded by Jisc as part of the Research Data Spring Initiative, and that is about technical tools, software and service solutions to support the researchers workflow, the use and mangement of their data. Now it’s interesting that this room is particuarly interested in teaching and learning, we are funded for researcher use but of course that does not proclude teaching and learning use.

The project partners here are City of Glasgow College as lead, The Open University and ?

So, what is Clipper? One of the challenges is explaining what this project is… And what it is not. So we are punting it as a research tool for digital research with online media / time-based media (ie audio/video data). The aim is to create a software toolkit (FOSS) deployed in an institution or operated as a n national service. We are about community engagement and collavorative design delivering a responsive design. And that’s why we are here.

So, why do this? Well time-based media is a large and “lumpy” data format, hard to analyse and even harder to share your analysis. There are barriers to effective (re)use of audio and video data including closed collections (IPR) and proprietary tools and formats. So we want to be able to create a “virtual clip” – and that means not copying any data, just metadata. So start and stop points on reference URI. And then also being able to organise that clip, to annotate it, and group into cliplists. So playlists of clips of excerpts etc. And then we can share using cool URIs for those clips and playlists.

This means bringing audio and video data to live, enabling analysis without breaking copyright or altering the soure data. We think it had streamlined workflows and facilitate collaboration. And we think it will lead to new things. It is secure and safe – respecting existing access permissions to data and does not alter or duplicate the original files. And it creates opportunities for citizen science/citizen research; user generated content – e.g. crowd sourcing etdata and user analytics. Colleagues in Manchester, for instance, have a group of bus enthusiasts who may be up for annotating old bus footage. The people who use your archives or data can generate analytics or para data and use of that can be useful and interesting as well.

So Clipped is… An online media analysis and collaboration tool for digital researchers (ie it supports human-based qualitative analysis, collavoboration and sharing. It is not an online audio/video editing tool. It is not a data repository. It is not using machine analysis of time based media. 


John: The best way to understand this stuff is to demonstrate and test this stuff out. We are going to take you through three workflows – these are just examples: (1) One source file, many clips, (2) Many source files, many clips, (3) Many source files, many clips, and annotations.

Over to Trevor and Will for examples.

Trevor: Hopefully as we work through these examples we should get more questions etc. and as we look through these examples.

Do bear in mind that what we will show you today is not a finished product, it’s a prototype. We want you to tell us what is good, what needs changing… You are the first of our three workshops so you get first say on the design! We want clear ideas on what will be useful… We hope it is fairly straightforward and fairly clear. If it isn’t, just tell us.

So, Workflow (1): Analysing a source file – the idea is an app developer (researcher) interviewing a user when testing an app. So the flow is:

  • Create and open a new project
  • Add the source file to the project
  • Preview the file – to find emerging themes etc.
  • Create clips – around those themes.
  • Add clips to cliplist

Now Will is demonstrating the system.

Will: I am going to create a new project, and I can edit the details later if I want to. And then I go in to edit the cliplist… And one of the collections included here is YouTube, as well as the BBC Collection (looks like their journalism trainee stuff), etc. I can choose a video, preview it, then I can choose to create a clip… I do this by watching the video and clicking “start” and “end” at the appropriate sections of the clip. I can then give the clip a title, and add a description and save it. Then close it. And any clips I create are kept in a “Project Cliplist”. Behind the scenes this is also getting saved to a database behind the scenes…

Trevor: So here we use the original source file, and we select a stop and start point… All the examples are based on video but the same player will do the same thing for audio. The intention is to support both audio and video within the same video.

Q1: What happens if you have a clip on a password protected Vimeo, etc.

Will: You have to have access permissions to the video… So it would attempt to play a video, and then that would be stopped.

Q1: But you would want students to be able to login, perhaps

Will: Would the tool then direct you to login, to anticipate that up front when you watch the list

Trevor: If you are signed in to Google, or signed in to your VLE, then as they would elsewhere in the browser, it will play clips. But if you are not logged in, no it won’t work. It would be nice to offer a pop up for sign in when needed. But we’ve tried that with private YouTube (only) so far.

Q2: Is there a way to separate out audio and video to save to different channels… So that you can strip down to just the audio… Maybe you just want to capture that.

Will: It’s not something that can be done in the browser, that’s more a server side function…

John: You could mark up the audio and video in Clipper. And then use that server side to extract the audio… And put the time references onto that.

Comment: Could hide the video, and play the sound…

Trevor: Hadn’t heard of that… But viewer is under our control… Could put a black filter over the video for a section.

Q3: The clips you are generating, can you tag them?

Will: Yes

Trevor: Thinking of ways to tag them, and to scale that up, is something to think about…

Workflow 2: Analysing Multiple Files

  • Create and open a new project
  • Add multiple source files to the project
  • Preview the files
  • Create clips
  • Add clips to cliplist

And the scenario we have in mind here is labs reviewing results across a distributed research team.

Will demonstrating a clip creation process using a Wellcome Collection video. 

Will: For some of our videos we can create thumbnails but that varies depending on rights etc. so instead we have a generic icon at the moment. And, as you can see, you can combine videos from multiple sources. So no matter what the resource you can create groups of clips.

Workflow 3: Adding Annotations to clips

  • Create and open a new project
  • Add multiple source files to the project
  • Preview the files and create clips
  • Add annotations to clips
  • Add clips to cliplist
  • So the example scenario is representations of climate change in mass media.

Trevor: Now this is where we’d particularly appreciate your comments on structures or tagging or approaches that might work, and that might scale.

Over to Will to demo again.

Will: So, when I have selected a clip I can click to annotate, and add an annotation to that clip at that moment. These annotations can then be associated with a particular second or moment in the video. And that is added to the clips metadata. And so we have time based annotation. And we will be adding a play button that enables us to jump to that moment in a video… And that information can be sharable – the clip,  the annotations, and the jumping to a moment in time.

Trevor: So it’s fairly light weight and pretty much wire framed… But hopefully enough there to understand the functionality.

Q4: Will annotations pop up when you reach them?

Will: Could highlight the clips…

Q4: Would be really useful, somewhere on the screen.

Comment: Even just a scrolling panel.

Q4: But also thinking about how it plays in fullscreen…

Will: Have seen demo on full screen video…

Q5: If you wanted to annotate a whole video would you have the option to do that as one clip?

Will: Yes, just use beginning and end of the video for a clip.

Q6: Would be useful to be able to use the hashtag or keywords etc. that a researcher wants to use – to easily find all the clips…

Will: So you could tag an annotation, or search for a keyword.

Comment: And see spread of tags etc.

Q5: The different ways the researcher wants to catergorise things.

Q7: All the moving image content on our site is only licensed for one site… Would this sit on the organisations site.. Where is it going?

Trevor: The YouTube videos are on their server… Played with this tool but the file stays on your server. Rights wise it would depend on how it is phrased. If hosted on your domain, then this would break it… But you could do this in house on your own system… Installing this software.

Q8: What if you have a video that specifies only the servers/IPs that can be used – which you can do on Vimeo – how would that work with Vimeo?

Trevor: I think it would work the same way… So if the user accesses the video from an appropriate IP range, it should work etc. But examples like that would be great to hear, so that we can address this.

Q9: How does transition between clips work?

Will: We can determine end of a clip, and fade out, fade in… But there are some buffering challenges potentially.

John: In the tool being tried out, the clips are on a host site… They are out on the web… Not on our demo site. Wellcome, BBC, YouTube is all coming in from different sites… So transitions have to take account of that.

Will: I am using the open source VideoJS player here… It does fire off nice events that allows us to indicate where clips begin and end… with a bit of jQuery.

John: Colleagues in North West Film Archive want to join clips up fairly seamlessly… But a gap or clear demarcation may be interesting.

Will: On the original flash version, to mask interruption, we took description from next clip and displayed that to smooth the transition.

Q9: Should leave to end user.

Trevor: Should maybe leave to end user for two or three options….

JOhn: We have been discussing options for end users… Because of how it is coded, it would be very feasible to have different options. Do I want to see the annotations this way or what… That flexibility does seem like it should be on the road map.

Trevor: May need to be decided at point of viewing.

Q10: Is this something Final Cut Pro could help, in terms of approach?

Will: Could be…

Trevor: Range of options is good.

Will: Almost drag and drop there…

Q11: Can you reorder the clips?

Will: That’s the intention, so yes. And likely drag and drop.

Q12: What about a web resource becomes available… And disappears… Hyperlinks can disappear and that would be a concern when I come to share it… And when I invest that time. And it’s quite likely… If a web link is dead, it’s a problem.

Trevor: With the Clipper server thing… If it was NLS, or a service based with the archive might be more trusted?

Q12: Not about trust, but fragility of web links…

Trevor: If we can surface the availability of content – if a source we know expires – we can show this.

Q12: I think that notifications would be useful here. But maybe also something that can be cached or kept so there is a capture of that.

Trevor: You don’t create the clip of video… But the annotation can be retained… And it can be saved and downloaded. So that even if the clip disappears, you might be able to switch the video URL and reapply that annotation.

Q12: Notifications would be really important.

Trevor: Managing a service that pushed out those emails could be really useful.

Will: We discussed that it would be possible to have video, captured by fancy proprietary video – once converted to e.g. MP4 – to annotate, but then also direct back to the original format.

Q13: You are pulling things through from websites elsewhere. If you make your own interview, can you upload it here… Or do you upload elsewhere and pull in URL?

Trevor: You can refer to a file on your own machine, or a repository, or on a private YouTube. But annotating a video that sits on your own machine is a good one for some researchers, e.g. on sensitive work etc.

Will: We have one challenge here… A fake path is used in the browser, and that can change… So you might have to browse to recreate that fake path…

John: But markup should transfer when you upload a video somewhere else – and upload a Clipper document that matches up with it…

Now watching a locally stored example – school children’s perceptions of researchers…

Q14: Question from me: Can you display rights information here – they should be available in metadata with video and/or APIs and are really important to indicate that.

John: We do take in that information, so it should be possible to display that… And we could do that with icons – e.g. Creative Commons symbols etc.

Q14: You might also want to include icons for locally hosted items – so that the playlist creator knows what can or cannot be seen by others (who likely won’t be able to access a file on a local machine).

Comment: For our collections the rights information is available in the API so it should be straightforward to pull in – that will apply to many other collections too (but not all)

Trevor: In addition to those indications it could be useful to foreground where rights information isn’t available.

Q15: My question is a bit different… Maybe how the clip is created… There are so many people who share clips and compilations of video items…

Trevor: We get to the same place really, but without reediting those videos etc.

Q16 – Me again: US vs UK copyright, particularly thinking about Fair Use content which might be legally acceptable in the US, but not in the UK.

John: Increasing ubiquity of video and audio online makes this stuff easier… But legal issues are there…

Q16 – Me again: In a way that level of usage, and so that issue would be a great problem to have though!

And now we are moving into testing out Clipper… So things will be quiet here… 

Comments on Demo

C1: You’ve only got one timestamp for annotations – would be useful to have end point too. And being able to annotation a particular frame/part of the frame to annotate as well. There are plugins for VideoJS with Overlay HTML. Being able to link annotations – link one to another would be useful.

Trevor: We thought about clips as URLs, and playlists as URLs. But we could also think about annotations as URLs.

C1: Version control on annotations would also be useful.

Trevor: Useful to think of that…

C2: A slide for the beginning or the end with credits etc. generated in the system would be useful. Would help with rights information.

Will: Also in Overlay VideoJS as well.

C3: General comment – do not understand technophobia of your audience. Web based service is a real advantage. Not many options, nothing to download, that is important. Capitalise on that… At the moment it looks more complex than it is. Has to not just be simple, but also look simple and user friendly.

Trevor: Absolutely. And that interface will change.

C4: I was wondering about nudging start and stop points…

Will: Set to read only now, was thinking about nudge buttons.

Trevor: Would you want to type or to have forward/back nudge buttons.

C4: probably both.

C5: I think you will need real usability testing to watch people using the tool, rather than asking them… And that will highlight where there is any misunderstanding. When I chose a video for a collection. How do I do anything creative with those clips… To merge or play all etc…

Trevor: Some of that sounds like video editing… If for those clips you want to change the order… You can shuffle them. You can’t merge them…

C5: Maybe you’d edit down elsewhere… Something to do with the content I have.

John: Are you wanting to select clips from different clip lists and then use in a new one?

C5: Yes, that’s one thing…

Will: That’s come up several times, and we do feel we need to add that to a roadmap… Perhaps creation of new video file maybe as compilation…

C6: From a users point of view you need confirmations on screen to highlight things have been created, saved, etc. For creating a clip, start and end, I didn’t get any visual confirmation. Need that to make it clear.

Trevor: Those are critical important things… Hopefully as we go through these workshops we’ll add that functionality.

Will: Notification systems might be useful in general within the system.

C7: It would be helpful to have maybe a pop up, or information symbol to remind you to cut off the clip. Thinking about the likely users here. Would be useful to have reminders.

Will: I think there is a lot to do on annotations.

C8: Searchable annotations would be really useful. And find all the relevant annotations. Things like NVivo do that.

Will: If anyone has looked on the JSON, I’ve had a tags property on the clip, but I can see we need that on the annotations.

John: On the annotations, people from Arts and Humanities suggest that an annotation could be an essay or an article. Several projects want storytelling tools using archives… The annotations side is potentially quite big in terms of length, and function it plays. From a rights point of view, an annotation could have it’s own rights.

C9 (me): That issue of annotations, also raises the issue of what the playback experience is. And how annotations etc. are part of that…

C10: How do you publish this content? Do you share the playlist? Do you need a Clipper account to view it?

Trevor: Well it may be the document of different clips… Maybe for projects you can invite people to join that project. Talking through the workflow might be useful. Sharing the link out there is something to think about.

Will: It may be just having a player, with a pane to the annotations. With a URL that works through the playlists, just as read only view. So we hope to have a sharable published HTML document to share. And could be maybe cached/saved for the long term (but not including original videos).

John: Could also have an embed code. Clipper fires information to a database, also into directories as HTML documents. If the database goes down, you still have reclaimable HTML documents. And you can send an embed code OR the HTML documents. Very transportable and friendly to Web 2.0 type stuff. But because in HTML, could deposit into catalogues etc. So good for long term.

Trevor: Any other ideas or comments please note them and share them with us – all of your comments are very welcome.

Now, after lunch we will have more discussion which includes implications for data management, service development and policy, etc. And then we’ll talk a bit more about technical aspects.

And now, for lunch… 

Discussion: Implications for Data Management

John: When we are looking at data management and implications: whose data? where stored? how is it stored and managed? why store and manage it? formats? retention? archive/deep freeze (available but maybe off site/harder to get to)?

Trevor: So, in your tables have a chat at your tables. And then we’ll feed back from these…

We’ve been discussing this so now for responses/ideas/comments… 

Table 1: If it’s research data a lot of this will be sensitive, and have to be within your control and your own students…

John: May also be issues of students data.

Table 1: We do use some cloud based services for student data though, so there must be some possibility there.

John: There is some of this in the paper economy, e.g. with assessment. But we find ways to do this. We are transitioning paper based to digital model… Perhaps we see problems as bigger than they are… And how long would you want to keep for a long time?

Table 1: Some for long term, some quite short.

Table 2: Some funders will have requirements too. But we were also talking about non-public video content… Maybe need two systems with permissions lined up… Asking students to sign in twice can be confusing. Institutional single sign on might be useful – map permissions across. But can the system recognise right to access data.

John: It could, and single sign on as a solution.

Comment: My students have access to very private recordings that have to be secure and has to be retained in that way, and keep it secure.

John: This can work as creating annotations, and can share pointer to the video clips… Outsider could view the annotations… It’s both a technical and policy issues. So you would tell students about protective identities etc.

Comment: password protection, encryption etc. might be important.

Comment: security of annotations may also be quite important.

Table 3: A question really: if it is someone else’s data and shared under CC licence (ND) – do clipper clips count as modifications or not?

Trevor: We think not but we should look at that.

John: But it might be fine, you are just excerpting the content, not cutting it. But could risk “passing off”.

Comment: You are still only showing part of a video, the whole video is available.

Comment: Could ensure links to full video… to ensure context is there.

Trevor: Again about how we present the content and it’s context, rights, etc.

John: It’s a user education issue, and a policy issue…

Table 4: We didn’t get beyond “whose data” and were particularly thinking about researcher data, and whether that data should be available to reuse by the institution, the funder, other researchers etc. And what are the funders requirements for that data etc. So really about how Clipper might be used inside that data environment.

Trevor: Funders are requiring data – some of it – to be made available openly.

Comment: Although not totality of data, it’s usually what supports publications. But open access aspect is certainly important. Clipper could find its way into that kind of environment and could be a good tool to show off some of your research.

John: And to do that in an efficient way… Maybe that FigShare concept of sharing data, even if not successful… Could have optional access to wider data sets, to the compressed video for easy viewing but maybe also HD huge files too…

Discussion: Policy

John: So what we’ve talked about already leads us to policy implications for service development. This may be legal issues (e.g. copyright, IPR); user generated content; licenses; access management; content management; data protection; data ownership and institutional IPR. Traditionally publishers owned the means of production and distribution and have high status with the University. But those issues of data ownership and institutional IPR are not well thought through. And that user generated content has issues of rights, license, access management.

After a lively discussion…

Table 1: How much do you need to worry about, how much is for institutions to worry about. Like data ownership etc. But you may need to worry about as a platform.

John: But we may need platform to support that, and therefore need to understand local platforms.

Table 1: And for access you’d want a lot of granularity of who might access these things, might be a large group or public, or might just be you, or just be a small group.

John: Clarity that that is possible could be a big winner.

Table 1: Having users fill in a field where they can state what they think the copyright is.

Trevor: A statement of intent?

Table 1: Yes, something that allows you to have a comeback is a collections owner comes back…

John: So it’s good for tracking, for due diligence. And maybe good for institutional procedures – for research projects where you need to know the rights involved. Might help raise awareness.

Table 2: Policy implications wise, there aren’t really any cases that shouldn’t already be covered by institutional policies. Licenses, derivative works, etc. should already by covered by institutional policies. Maybe some special cases…

John: Are the policies fit for purpose?

Comment: It is usually awareness not existence of policies which is usually

Table 3: Possibly a pop up indicating license and appropriate usage, so you know what you can do. Second aspect, if you can legally modify videos – why not do on desktop system offline, if not then how can this comply. Only the making of copies that this removes the issue for. Sorry for a super defeatist comment but how does this differ from what else is there.

Comment: I come at this from two places… Both the way into lumpy content, interrogate, search it, etc… And then also this more creative tool where you make something else available on the internet – alarm bells start ringing. For the creative side, why not use iMovie etc.

Comment: It’s not a video editing tool, it’s annotation. So clearly not that…

John: Useful to use, to make sure we describe it appropriately. It’s a challenge. We need to make it clear what we think can be done with it. We’ll take those comments on board and blog about it to try and make this all clearer.

Trevor: If you were just making clips.. .but in the context of research it’s more about annotations and descriptions etc. But when you have gone to that effort, you want it to look nice.

John: One of our original ambitions was to make it as easy for researchers to quote video and time based media as for print…

Comment: For digital preservation… preserving video is relatively difficult and is an ongoing process. Clips are basically JSON descriptions – easy to preserve.

Comment: A very good content. But I think being very clear on what this thing is for… And making it really good for these things. Really focusing on the annotations and textual aspects more.

Discussion: Service Development Implications

Trevor: Now for our final section we will talk about service development implications: scale – should it be individual, institutional, regional, national, international? Why bother? Benefits? Technical challenges – storage (e.g. 132 MB/s or 463 GB/h), transcoding and archiving; costs; metadata and data models.

Again, much discussion… 

John: We talked about scale of this system… There may be a role here for an individual service… For many here will be institutional… But may be national or international. Bandwidth could be an issue depending on resolution.

Table 4: Embargoes, on metadata, and issues of privacy, access, and license for annotations for the same reasons.

John: What about bandwidth?

Table 2: It depends on the video delivery…

Table 1: It’s not your issue really. It’s for content providers…

Trevor: It’s more institutional stuff then…

Comment: The system depends on you having a consistent URI for a playable version of a video… That may be an issue depending on how files are held.

John: What about a Service Level Definitions around persistent URIs? Would that fly?

John: And what about the role of cloud providers?

Several in the room indicate they are using them… 

Comment: Making annotations public will help others find your data.

John: Annotations coming up and up as being the things.

Comment: Costs wise it needs to be open source for people to import themselves? And if so, how can you skin it and brand it. And how often does it need maintenance and updates.

John: We are looking at sustainability options, that’s something we want to look at.

Trevor: This is currently funded under Jisc Research Data Spring initiative, and that is done in 3 phases… First stage is reaching out to show there is demand. This phase we are in now is developing our prototype it. And the third phase is to look at sustainability, things like support, update, development community, etc.

Trevor: The last bit for the day is to cover some technical stuff and go through some of that…

Technical Overview – Will

The system generates and stores HTML5 documents. And generates sharable URIs of playable clips and cliplists. JSON data structures (import/export CSV or XML). PHP scripts data handling with MySQL database and JavaScript interface. Responsible layout – computer, tablet and phone (already tested on iPad). And actually as you use a video on your system you can take a video in situ on tablet/phone. Will be free and open source software – the code will be posted to:

So, just to demonstrate, when you have a playlist you hit “publish” to publish your playlist in various formats. At the moment generates JSON data. A nice quick way to describe data. Annotations are becoming very important so we will need some comma separated tags, and access privileges as well.

Comments: Is there documentation for the code so far?

Trevor: Not yet but software and documentation

Will: Does anyone have any questions about technology elsewhere. We are using VideoJS. We are hosting this in a WordPress installation at the moment – that’s for logins and id generation as well.

Comment: API for Clipper? So others can use the annotations etc.

John: Also discussing a metadata editor for those creating their own annotations.

Comment: If sensitive data, and videos, then annotations might also want to be private… Rather than being on your server..

Trevor: We’d suggest an institutional instance.

Comment: Or could they get a private instance from you?

John: We are not at that stage yet, but that could be an option.

Complex: We haven’t talked much about searching capabilities.

Will: Anything in this text content should be searchable… Might be able to searchable across the board… Might be that when sensitive and private you might have to request access rather than seeing it.

John: Worth making the point that it has to be easy to import data into Clipper, and export data out of it. If this is in a library or archive… We could ingest catalogue information… Could ingest metadata and then come up with an instance to point to. So, e.g. for Scottish Screen Archive you could use shotlist to create clips automatically. So lots of potential when metadata rich environment. So could take in metadata to help generate your collection.

Trevor: Within a project you can search within that project, or more when at the higher level… So we want search to be contextual…

Comment: I think for effective searching you are going to want to have a more complex annotation data structure – so you can do filters, indexing etc. so less computationally taxing and more accurate for users.

Comment: Does the system log who has created which annotation? So you can track who does what on a research project.

John: And with that we will bring it to a close… Thank you all for coming.

Thanks to John, Trevor and Will for today’s workshop and to Gill and the NLS for hosting. If you are interested in attending the next Clipper workshops you can register/find out more here:

Sep 172015

This afternoon I am attending a seminar from Gregor Kennedy, University of Melbourne, organised by the Digital Cultures and Education research group at University of Edinburgh.

As usual this is a liveblog so please let me know if you see any typos, have corrections to suggest, etc. 

My background is in social psychology and I decided to change fields and move into educational technology. And when I started to make that change in direction… Well I was studying with my laptop but I love this New Yorker cover from 1997 which speaks to both technology and the many ways in which Academia doesn’t change.

I also do a lot of work on the environment, and the ways that technology effects change in the wider world, for instance the way that a library has gone from being about physical texts to a digital commons. And my work is around that user interface and mediation that occurs. And in the first 15 years of my career was in medical technology, and in interfaces around this.

Now, the world of Digital Education is dominated by big platforms, from early to mid-2000, enterprise teaching and learning systems that provide, administer, etc. Platforms like Blackboard, turnitin, Moodle, Echo. And we have tools like Twitter, blogging tools, YouTube, Facebook, Second Life also coming in. We also see those big game changers of Google and Wikipedia. And we have companies/tools like Smart Sparrow which are small adaptive learning widgets with analytics built into them. And we see new big provicers of Coursera, EdX, Future Learn, the mass teaching and learning platforms.

So, as educators we have these fantastic tools that enable us to track what students do. But we also can find ourselves in an Orwellian place, where that tracking is all the time and can be problematic. But you can use all that VLE data in ways that really benefits education and learning. Part of that data enables us to see the digital footprints that students make in this space. And my group really look at this issue of how we can track those footprints, and – crucially – how we can take something meaningful for that.

Two of the early influential theorists in this space are Tom Reeves and John Hedberg. Back in 2003 they wrote about the problematic nature of auditing student data trails, and the challenges of doing that. Meanwhile there has been other work and traditions, from the Intelligent Tutoring Systems in the 1970s onwards. But part of the reason I think Reeves and Hedberg didn’t think meaningful interactions would be possible is because, at their most basic level, the data we get out of these systems is about behaviour which is not directly related to cognition.

Now we have to be a bit careful about this… Some behavioural responses can be imbued with a notion of what a student is thinking, for instance free-text responses to a discussion list; responses to multiple choice questions. But for much of what we collect, and the modern contemporary learning analytics community is talking about, that cognition is absent. So that means we have to make assumptions about students intent, motivation, attitude…

Now, we have some examples of how those sort of assumptions going wrong can be problematic. For instance the Amazon recommendation system deals poorly with gifts or one off interests. Similarly Microsoft Clippy often gets it wrong. So that distinction between behaviour and cognition is a major part of what I want to talk about today, and how we can take meaningful understanding from that.

So I want to start with an example, the Cognition and Interaction project, which I work on with Barney Dalgarno, Charles Sturt University; Sue Bennett, University of Wollongong. We created quite flat interactive learning objects that could work with learners who were put in an fMRI machine, so we could see brain activity. For this project we wanted to look at how learning design changed cognition.

So, we had an “Observation Program” – a page turning task with content screens and an introductions with background terminology. They saw changes in parameters being made. And an “Exploration Program” where students changed parameters themselves and engaged directly with the material. Both of these approaches were trialled with two examples: Global Warming adn Blood Alcohol. Now which one would you expect to be more effective? Yup, Exploration. So we got the results through and we were pretty bummed out as there was very little difference between the two. But we did notice there was a great deal of variation in the test scores later on. And we were able to use this to classify Students Aproaches:

  • Systematic Exploration – trying a variable, seeing the result. Trying another, etc…
  • Non-Systemaic Exploration – changing stuff all over the place.
  • Observation group – observation

So we re-ran the analysis and found there was no difference between the Non-Systematic Exploration and the Observation group, but there was a difference between the Systematic Exploration and the other groups.

So, why is this interesting? Well firstly students do not do what they are supposed to do, or what we expect them to do. The intent that we have as designers and educators is not manifest in the way students engage in those tasks. And we do see this time and time again… the digital footprints that students leave show us how they fail to conform to the pedagogical intent of the online tasks we set for them. They don’t follow the script.

But we can find meaningful patterns of students behaviour using their digital footprints… interpreted through the lens of the learning design of the task. These patterns suggest different learning approaches and different learning outcomes…

Example 2: MOOCs & LARG

One thing, when we set up our MOOCs, we set up the Learning Analytics Research Group, and this brings people together from information technology, informatics, education, educational technology, etc. And this work is with members of this group.

So, I want to show you a small snapshot of this type of work. We have two MOOCs to compare here. Firstly Principles of Macroeconomics, a classic staged linear course, with timed release of content and assessment at the end. The other course is Discrete Optimization which is a bit more tricksy… All of the content is released at once and they can redo assessments as many times as they want. There is a linear suggested path but they are free to explore in their own way.

So, for these MOOCs we measured a bunch of stuff and I will focus on how frequently different types of students watched and revisited video lectures across each course. And we used State Transition diagrams. These state transitions illustrate the probability of people transitioning from State A to State B – the footfall or pathways they might take…

We created these diagrams for both courses and for a number of different ways of participating: Browsed – did no assessment; Participated – did not do well; Participated – did OK; Participated – did well. And as outcomes improve these transitions/the likelihoods of state transition increases. And the Discrete Optimisation MOOC saw a greater level of success.

So, again, we see patterns of engagement suggesting different learning strategies or approaches. But there is a directional challenge here – it is hard to know if people who revisit material more, do better… Or whether those who do better revisit content more. And that’s a classic question in education, how do you address and help those without aptitude…

So, the first two examples show interesting fundamental education questions… 

Example 3: Surgical Skills Simulation 

I’ve been working on this since about 2006. And this is about a 3D immersive haptic system for e-surgery. Not only is the surgeon able to see and have the sensation of performing a real operation, but the probe being used gives physical feedback. This is used in surgical education. So we have taken a variety of metrics – 15 records of 48 metrics per second – which capture how they use the surgical tools, what they do, etc.

What we wanted to do was provide personalised feedback to surgical trainees, to emulate what a surgeon watching this procedure might say – rather than factual/binery type feedback. And that feedback comes in based on their digital trace in the system… As they show novice like behaviour, feedback is provided in a staged way… But expert behaviour doesn’t trigger this, to avoid that Microsoft paperclip feedback type experience.

So, we trialled the approach with and without feedback. Both groups have similar patterns but the feedback has a definite impact. And the feedback from learners about that experience is pretty good.

So, can we take meaningful information from this data? Yes, it’s possible…

I started with these big buckets of data from VLEs etc… So I have four big ideas of how to make this useful…

1. Following footprints can help us understand how students approach learning tasks and the curriculum more broadly. Not so much whether they understand the concept or principle they are working on, and whether they got something correct or not… But more their learning and study strategies when faces with the various learning tasks online.

2. If we know how students approach those learning tasks and their study, it does give us insight into their cognitice and learning processes… Which we can link to their leanring outcomes. This method is a wonderful resource for educational research!

3. Knowing how students approach learning tasks is incredible useful for teachiers and educational designers. We can see in fine detail how the educational tasks we create and design are “working” with students – the issue of pedagogical intent, response, etc.

4. Knowing how students approach learning tasks is increadibly useful for designing intervantions with students. Even in open and complex digital learning environments we can use students digital footprints as a basis for individualised feedback, and advise students on approaches adopted.

So, I think that gives you an idea about my take on learning analytics. There are ways we can use this in quite mundane ways but in educational research and working across disciplines we have the potential to really crack some of those big challenges in education.


Q1) For the MOOC example… Was there any flipping of approaches for the different courses or A/B testing. Was there any difference in attainment and achievement?

A1) The idea of changing the curriculum design for one of those well established courses is pretty difficult so, no. In both courses we had fairly different cohorts – the macroeconomics course . We are now looking at A/B testing to see how potential confusion in videos compares with more straightforward “David Attenborough, this is the way of the world” type videos, so we will see what happens there.

Q2) What

A2) There is some evidence that confusion can be a good thing – but there is productive and unproductive confusion. And having productive confusion as part of a pathway towards understanding… And we are getting students from other disciplines looking at very different courses (e.g. Arts students engaging with chemistry courses, say) to cause some deliberate confusion but with no impact on their current college courses.

Q3) On that issue of confusion… What about the impact of learning through mistakes, of not correcting a student and the impact that may have?

A3) A good question… You can have False positive – provide feedback but shouldn’t have. False negative – don’t provide feedback but shouldn’t have. With our system we captured our feedback and compared with a real surgeon’s view on where they would/would not offer feedback. We had about 8% false positives and 12% false negatives. That’s reasonably good for teaching excercise.

Q4) How do your academic colleagues respond to this, as you are essentially buying into the neo liberal agenda about

A4) It’s not a very common issue to come up, its surprising how little it comes up. So in terms of telling teachers what they already know – some people are disheartened by you providing impirical evidence of what they already know as experienced teachers. You have to handle that sensitively but many see that as reenforcement of their practice. In terms of replacing teachers… These are helper applications. The feedback side of things can only be done in a very small way compared to the richness of a human, and tend to be more triage-like applications that forms a small part of the wider curriculum. And often those systems are flagging up the need for a richer interaction or intervention.

Q5) Most students think that more time on task maps to more success… And your MOOC data seems to reinforce that… So what do you do in terms of sharing data with students, and especially students who are not doing as well?

A5) It’s not my research area but my colleague Linda does work on this and on dashboards. It is such a tricky area. There is so much around ethics, pastoral care, etc.

Students with high self efficacy but behind the pack, will race to catch up and may exseed. But students to low self efficacy may drop back or drop out. There is educational psychology work in this area (see Carol Dykal’s work) but little on learning analytics.

But there is also the issue of the impact of showing an individual their performance compared to a group, to their cohort… Does that encourage the student to behave more like the pack which may not be in their best interests. There is still a lot we don’t know about the impact of doing that.

Q6) We are doing some research here with students…

A6) We have a range of these small tasks and we ask them on every screen about how difficult the task is, and how confident they feel about it and we track that along with other analytics. For some tasks confidence and confusion are very far apart – very confident and not confused at all although that can mean you are resistent to learning. But for others each screen sees huge variation in confidence and confusion levels…

Q7) Given your comments about students not doing what they are expected to do… Do you think that could impact here. Like students in self-assessments ranking their own level of understanding as low, in order to game the system so they can show improvement later on.

A7) It’s a really good question. There isn’t a great motivation to lie – these tasks aren’t part of their studies, they get paid etc. And there isn’t a response test which would make that more likely. But in the low confusion, high confidence tasks… the feedback and discussion afterwards suggests that they are confused at times, and there is a disjoint. But if you do put a dashboard in front of students, they are very able to interpret their own behaviour… They are really focused on their own performance. They are quite reflective… And then my colleagues Linda and Paul ask what they will do and they’ll say “Oh, I’ll definitely borrow more books from the library, I’ll definitely download that video…” and six weeks later they are interviewed and there is no behaviour change… Perhaps not surprising that people don’t always do what they say they will… We see that in learning analytics too.

Q8) [Didn’t quite catch all of this but essentially about retention of students]

A8) We have tried changing the structure and assessment of one of our courses, versus the first run, because of our changed understanding of analytics. And we have also looked at diagnostic assessment in the first three weeks of a course as a predictor for later performance. In that you see a classic “browsing in the book store” type behaviour. We are not concerned about them. But those who purchase the first assessment task, we can see they can do well and are able to… And they tend to stick with the course. But we see another type – a competent crowd who engage early on, but fall of. It’s those ones that we are interested in and who are ripe for retaining.

 September 17, 2015  Posted by at 1:16 pm Events Attended, LiveBlogs Tagged with: , ,  No Responses »