Today I am at the National Library of Scotland for a Clipper project workshop (info here). Clipper is a project to create a content creation tool for multimedia, with funding from Jisc.
After an introduction from Gill Hamilton Intro it’s over to John Casey who will be leading this through the day…
Introduction – John Casey
The tagline for the project is basically Clipper 1. 2. 3: Clip, Organise, Share.
We want your input early on in the process here but that means we will be trying out a prototype with you – so there will be bugs and issues but we are looking for your comments and feedback etc. The first outing of Clipper was from 2009, as a rapid development project which used Flash and Flex. Then it went to sleep for a while. Then we started working on it again when looking at Open Education in London
Trevor: I’m Trevor Collins – research fellow at the Open University. My background is very technical – computer engineering, HCI but all my research work is around the context of learning and teaching. And we have a common interest in HTML5 video. And my interest is working, with you, to ensure this will be helpful and useful.
Will: My name is Will and my background is engineering. Originally I worked with John on this project in Flash etc. but that’s really died out and, in the meantime HTML has really moved on a long way and with video in HTML5 we can just use the browser as the foundation, potentially, for some really interesting application. For me my interest today is in the usability of the interface.
With that we have had some introductions… It is a really interesting group of multimedia interested folk.
John Casey again:
This project is funded by Jisc as part of the Research Data Spring Initiative, and that is about technical tools, software and service solutions to support the researchers workflow, the use and mangement of their data. Now it’s interesting that this room is particuarly interested in teaching and learning, we are funded for researcher use but of course that does not proclude teaching and learning use.
The project partners here are City of Glasgow College as lead, The Open University and ?
So, what is Clipper? One of the challenges is explaining what this project is… And what it is not. So we are punting it as a research tool for digital research with online media / time-based media (ie audio/video data). The aim is to create a software toolkit (FOSS) deployed in an institution or operated as a n national service. We are about community engagement and collavorative design delivering a responsive design. And that’s why we are here.
So, why do this? Well time-based media is a large and “lumpy” data format, hard to analyse and even harder to share your analysis. There are barriers to effective (re)use of audio and video data including closed collections (IPR) and proprietary tools and formats. So we want to be able to create a “virtual clip” – and that means not copying any data, just metadata. So start and stop points on reference URI. And then also being able to organise that clip, to annotate it, and group into cliplists. So playlists of clips of excerpts etc. And then we can share using cool URIs for those clips and playlists.
This means bringing audio and video data to live, enabling analysis without breaking copyright or altering the soure data. We think it had streamlined workflows and facilitate collaboration. And we think it will lead to new things. It is secure and safe – respecting existing access permissions to data and does not alter or duplicate the original files. And it creates opportunities for citizen science/citizen research; user generated content – e.g. crowd sourcing etdata and user analytics. Colleagues in Manchester, for instance, have a group of bus enthusiasts who may be up for annotating old bus footage. The people who use your archives or data can generate analytics or para data and use of that can be useful and interesting as well.
So Clipped is… An online media analysis and collaboration tool for digital researchers (ie it supports human-based qualitative analysis, collavoboration and sharing. It is not an online audio/video editing tool. It is not a data repository. It is not using machine analysis of time based media.
John: The best way to understand this stuff is to demonstrate and test this stuff out. We are going to take you through three workflows – these are just examples: (1) One source file, many clips, (2) Many source files, many clips, (3) Many source files, many clips, and annotations.
Over to Trevor and Will for examples.
Trevor: Hopefully as we work through these examples we should get more questions etc. and as we look through these examples.
Do bear in mind that what we will show you today is not a finished product, it’s a prototype. We want you to tell us what is good, what needs changing… You are the first of our three workshops so you get first say on the design! We want clear ideas on what will be useful… We hope it is fairly straightforward and fairly clear. If it isn’t, just tell us.
So, Workflow (1): Analysing a source file – the idea is an app developer (researcher) interviewing a user when testing an app. So the flow is:
- Create and open a new project
- Add the source file to the project
- Preview the file – to find emerging themes etc.
- Create clips – around those themes.
- Add clips to cliplist
Now Will is demonstrating the system.
Will: I am going to create a new project, and I can edit the details later if I want to. And then I go in to edit the cliplist… And one of the collections included here is YouTube, as well as the BBC Collection (looks like their journalism trainee stuff), etc. I can choose a video, preview it, then I can choose to create a clip… I do this by watching the video and clicking “start” and “end” at the appropriate sections of the clip. I can then give the clip a title, and add a description and save it. Then close it. And any clips I create are kept in a “Project Cliplist”. Behind the scenes this is also getting saved to a database behind the scenes…
Trevor: So here we use the original source file, and we select a stop and start point… All the examples are based on video but the same player will do the same thing for audio. The intention is to support both audio and video within the same video.
Q1: What happens if you have a clip on a password protected Vimeo, etc.
Will: You have to have access permissions to the video… So it would attempt to play a video, and then that would be stopped.
Q1: But you would want students to be able to login, perhaps
Will: Would the tool then direct you to login, to anticipate that up front when you watch the list
Trevor: If you are signed in to Google, or signed in to your VLE, then as they would elsewhere in the browser, it will play clips. But if you are not logged in, no it won’t work. It would be nice to offer a pop up for sign in when needed. But we’ve tried that with private YouTube (only) so far.
Q2: Is there a way to separate out audio and video to save to different channels… So that you can strip down to just the audio… Maybe you just want to capture that.
Will: It’s not something that can be done in the browser, that’s more a server side function…
John: You could mark up the audio and video in Clipper. And then use that server side to extract the audio… And put the time references onto that.
Comment: Could hide the video, and play the sound…
Trevor: Hadn’t heard of that… But viewer is under our control… Could put a black filter over the video for a section.
Q3: The clips you are generating, can you tag them?
Trevor: Thinking of ways to tag them, and to scale that up, is something to think about…
Workflow 2: Analysing Multiple Files
- Create and open a new project
- Add multiple source files to the project
- Preview the files
- Create clips
- Add clips to cliplist
And the scenario we have in mind here is labs reviewing results across a distributed research team.
Will demonstrating a clip creation process using a Wellcome Collection video.
Will: For some of our videos we can create thumbnails but that varies depending on rights etc. so instead we have a generic icon at the moment. And, as you can see, you can combine videos from multiple sources. So no matter what the resource you can create groups of clips.
Workflow 3: Adding Annotations to clips
- Create and open a new project
- Add multiple source files to the project
- Preview the files and create clips
- Add annotations to clips
- Add clips to cliplist
- So the example scenario is representations of climate change in mass media.
Trevor: Now this is where we’d particularly appreciate your comments on structures or tagging or approaches that might work, and that might scale.
Over to Will to demo again.
Will: So, when I have selected a clip I can click to annotate, and add an annotation to that clip at that moment. These annotations can then be associated with a particular second or moment in the video. And that is added to the clips metadata. And so we have time based annotation. And we will be adding a play button that enables us to jump to that moment in a video… And that information can be sharable – the clip, the annotations, and the jumping to a moment in time.
Trevor: So it’s fairly light weight and pretty much wire framed… But hopefully enough there to understand the functionality.
Q4: Will annotations pop up when you reach them?
Will: Could highlight the clips…
Q4: Would be really useful, somewhere on the screen.
Comment: Even just a scrolling panel.
Q4: But also thinking about how it plays in fullscreen…
Will: Have seen demo on full screen video…
Q5: If you wanted to annotate a whole video would you have the option to do that as one clip?
Will: Yes, just use beginning and end of the video for a clip.
Q6: Would be useful to be able to use the hashtag or keywords etc. that a researcher wants to use – to easily find all the clips…
Will: So you could tag an annotation, or search for a keyword.
Comment: And see spread of tags etc.
Q5: The different ways the researcher wants to catergorise things.
Q7: All the moving image content on our site is only licensed for one site… Would this sit on the organisations site.. Where is it going?
Trevor: The YouTube videos are on their server… Played with this tool but the file stays on your server. Rights wise it would depend on how it is phrased. If hosted on your domain, then this would break it… But you could do this in house on your own system… Installing this software.
Q8: What if you have a video that specifies only the servers/IPs that can be used – which you can do on Vimeo – how would that work with Vimeo?
Trevor: I think it would work the same way… So if the user accesses the video from an appropriate IP range, it should work etc. But examples like that would be great to hear, so that we can address this.
Q9: How does transition between clips work?
Will: We can determine end of a clip, and fade out, fade in… But there are some buffering challenges potentially.
John: In the tool being tried out, the clips are on a host site… They are out on the web… Not on our demo site. Wellcome, BBC, YouTube is all coming in from different sites… So transitions have to take account of that.
Will: I am using the open source VideoJS player here… It does fire off nice events that allows us to indicate where clips begin and end… with a bit of jQuery.
John: Colleagues in North West Film Archive want to join clips up fairly seamlessly… But a gap or clear demarcation may be interesting.
Will: On the original flash version, to mask interruption, we took description from next clip and displayed that to smooth the transition.
Q9: Should leave to end user.
Trevor: Should maybe leave to end user for two or three options….
JOhn: We have been discussing options for end users… Because of how it is coded, it would be very feasible to have different options. Do I want to see the annotations this way or what… That flexibility does seem like it should be on the road map.
Trevor: May need to be decided at point of viewing.
Q10: Is this something Final Cut Pro could help, in terms of approach?
Will: Could be…
Trevor: Range of options is good.
Will: Almost drag and drop there…
Q11: Can you reorder the clips?
Will: That’s the intention, so yes. And likely drag and drop.
Q12: What about a web resource becomes available… And disappears… Hyperlinks can disappear and that would be a concern when I come to share it… And when I invest that time. And it’s quite likely… If a web link is dead, it’s a problem.
Trevor: With the Clipper server thing… If it was NLS, or a service based with the archive might be more trusted?
Q12: Not about trust, but fragility of web links…
Trevor: If we can surface the availability of content – if a source we know expires – we can show this.
Q12: I think that notifications would be useful here. But maybe also something that can be cached or kept so there is a capture of that.
Trevor: You don’t create the clip of video… But the annotation can be retained… And it can be saved and downloaded. So that even if the clip disappears, you might be able to switch the video URL and reapply that annotation.
Q12: Notifications would be really important.
Trevor: Managing a service that pushed out those emails could be really useful.
Will: We discussed that it would be possible to have video, captured by fancy proprietary video – once converted to e.g. MP4 – to annotate, but then also direct back to the original format.
Q13: You are pulling things through from websites elsewhere. If you make your own interview, can you upload it here… Or do you upload elsewhere and pull in URL?
Trevor: You can refer to a file on your own machine, or a repository, or on a private YouTube. But annotating a video that sits on your own machine is a good one for some researchers, e.g. on sensitive work etc.
Will: We have one challenge here… A fake path is used in the browser, and that can change… So you might have to browse to recreate that fake path…
John: But markup should transfer when you upload a video somewhere else – and upload a Clipper document that matches up with it…
Now watching a locally stored example – school children’s perceptions of researchers…
Q14: Question from me: Can you display rights information here – they should be available in metadata with video and/or APIs and are really important to indicate that.
John: We do take in that information, so it should be possible to display that… And we could do that with icons – e.g. Creative Commons symbols etc.
Q14: You might also want to include icons for locally hosted items – so that the playlist creator knows what can or cannot be seen by others (who likely won’t be able to access a file on a local machine).
Comment: For our collections the rights information is available in the API so it should be straightforward to pull in – that will apply to many other collections too (but not all)
Trevor: In addition to those indications it could be useful to foreground where rights information isn’t available.
Q15: My question is a bit different… Maybe how the clip is created… There are so many people who share clips and compilations of video items…
Trevor: We get to the same place really, but without reediting those videos etc.
Q16 – Me again: US vs UK copyright, particularly thinking about Fair Use content which might be legally acceptable in the US, but not in the UK.
John: Increasing ubiquity of video and audio online makes this stuff easier… But legal issues are there…
Q16 – Me again: In a way that level of usage, and so that issue would be a great problem to have though!
And now we are moving into testing out Clipper… So things will be quiet here…
Comments on Demo
C1: You’ve only got one timestamp for annotations – would be useful to have end point too. And being able to annotation a particular frame/part of the frame to annotate as well. There are plugins for VideoJS with Overlay HTML. Being able to link annotations – link one to another would be useful.
Trevor: We thought about clips as URLs, and playlists as URLs. But we could also think about annotations as URLs.
C1: Version control on annotations would also be useful.
Trevor: Useful to think of that…
C2: A slide for the beginning or the end with credits etc. generated in the system would be useful. Would help with rights information.
Will: Also in Overlay VideoJS as well.
C3: General comment – do not understand technophobia of your audience. Web based service is a real advantage. Not many options, nothing to download, that is important. Capitalise on that… At the moment it looks more complex than it is. Has to not just be simple, but also look simple and user friendly.
Trevor: Absolutely. And that interface will change.
C4: I was wondering about nudging start and stop points…
Will: Set to read only now, was thinking about nudge buttons.
Trevor: Would you want to type or to have forward/back nudge buttons.
C4: probably both.
C5: I think you will need real usability testing to watch people using the tool, rather than asking them… And that will highlight where there is any misunderstanding. When I chose a video for a collection. How do I do anything creative with those clips… To merge or play all etc…
Trevor: Some of that sounds like video editing… If for those clips you want to change the order… You can shuffle them. You can’t merge them…
C5: Maybe you’d edit down elsewhere… Something to do with the content I have.
John: Are you wanting to select clips from different clip lists and then use in a new one?
C5: Yes, that’s one thing…
Will: That’s come up several times, and we do feel we need to add that to a roadmap… Perhaps creation of new video file maybe as compilation…
C6: From a users point of view you need confirmations on screen to highlight things have been created, saved, etc. For creating a clip, start and end, I didn’t get any visual confirmation. Need that to make it clear.
Trevor: Those are critical important things… Hopefully as we go through these workshops we’ll add that functionality.
Will: Notification systems might be useful in general within the system.
C7: It would be helpful to have maybe a pop up, or information symbol to remind you to cut off the clip. Thinking about the likely users here. Would be useful to have reminders.
Will: I think there is a lot to do on annotations.
C8: Searchable annotations would be really useful. And find all the relevant annotations. Things like NVivo do that.
Will: If anyone has looked on the JSON, I’ve had a tags property on the clip, but I can see we need that on the annotations.
John: On the annotations, people from Arts and Humanities suggest that an annotation could be an essay or an article. Several projects want storytelling tools using archives… The annotations side is potentially quite big in terms of length, and function it plays. From a rights point of view, an annotation could have it’s own rights.
C9 (me): That issue of annotations, also raises the issue of what the playback experience is. And how annotations etc. are part of that…
C10: How do you publish this content? Do you share the playlist? Do you need a Clipper account to view it?
Trevor: Well it may be the document of different clips… Maybe for projects you can invite people to join that project. Talking through the workflow might be useful. Sharing the link out there is something to think about.
Will: It may be just having a player, with a pane to the annotations. With a URL that works through the playlists, just as read only view. So we hope to have a sharable published HTML document to share. And could be maybe cached/saved for the long term (but not including original videos).
John: Could also have an embed code. Clipper fires information to a database, also into directories as HTML documents. If the database goes down, you still have reclaimable HTML documents. And you can send an embed code OR the HTML documents. Very transportable and friendly to Web 2.0 type stuff. But because in HTML, could deposit into catalogues etc. So good for long term.
Trevor: Any other ideas or comments please note them and share them with us – all of your comments are very welcome.
Now, after lunch we will have more discussion which includes implications for data management, service development and policy, etc. And then we’ll talk a bit more about technical aspects.
And now, for lunch…
Discussion: Implications for Data Management
John: When we are looking at data management and implications: whose data? where stored? how is it stored and managed? why store and manage it? formats? retention? archive/deep freeze (available but maybe off site/harder to get to)?
Trevor: So, in your tables have a chat at your tables. And then we’ll feed back from these…
We’ve been discussing this so now for responses/ideas/comments…
Table 1: If it’s research data a lot of this will be sensitive, and have to be within your control and your own students…
John: May also be issues of students data.
Table 1: We do use some cloud based services for student data though, so there must be some possibility there.
John: There is some of this in the paper economy, e.g. with assessment. But we find ways to do this. We are transitioning paper based to digital model… Perhaps we see problems as bigger than they are… And how long would you want to keep for a long time?
Table 1: Some for long term, some quite short.
Table 2: Some funders will have requirements too. But we were also talking about non-public video content… Maybe need two systems with permissions lined up… Asking students to sign in twice can be confusing. Institutional single sign on might be useful – map permissions across. But can the system recognise right to access data.
John: It could, and single sign on as a solution.
Comment: My students have access to very private recordings that have to be secure and has to be retained in that way, and keep it secure.
John: This can work as creating annotations, and can share pointer to the video clips… Outsider could view the annotations… It’s both a technical and policy issues. So you would tell students about protective identities etc.
Comment: password protection, encryption etc. might be important.
Comment: security of annotations may also be quite important.
Table 3: A question really: if it is someone else’s data and shared under CC licence (ND) – do clipper clips count as modifications or not?
Trevor: We think not but we should look at that.
John: But it might be fine, you are just excerpting the content, not cutting it. But could risk “passing off”.
Comment: You are still only showing part of a video, the whole video is available.
Comment: Could ensure links to full video… to ensure context is there.
Trevor: Again about how we present the content and it’s context, rights, etc.
John: It’s a user education issue, and a policy issue…
Table 4: We didn’t get beyond “whose data” and were particularly thinking about researcher data, and whether that data should be available to reuse by the institution, the funder, other researchers etc. And what are the funders requirements for that data etc. So really about how Clipper might be used inside that data environment.
Trevor: Funders are requiring data – some of it – to be made available openly.
Comment: Although not totality of data, it’s usually what supports publications. But open access aspect is certainly important. Clipper could find its way into that kind of environment and could be a good tool to show off some of your research.
John: And to do that in an efficient way… Maybe that FigShare concept of sharing data, even if not successful… Could have optional access to wider data sets, to the compressed video for easy viewing but maybe also HD huge files too…
John: So what we’ve talked about already leads us to policy implications for service development. This may be legal issues (e.g. copyright, IPR); user generated content; licenses; access management; content management; data protection; data ownership and institutional IPR. Traditionally publishers owned the means of production and distribution and have high status with the University. But those issues of data ownership and institutional IPR are not well thought through. And that user generated content has issues of rights, license, access management.
After a lively discussion…
Table 1: How much do you need to worry about, how much is for institutions to worry about. Like data ownership etc. But you may need to worry about as a platform.
John: But we may need platform to support that, and therefore need to understand local platforms.
Table 1: And for access you’d want a lot of granularity of who might access these things, might be a large group or public, or might just be you, or just be a small group.
John: Clarity that that is possible could be a big winner.
Table 1: Having users fill in a field where they can state what they think the copyright is.
Trevor: A statement of intent?
Table 1: Yes, something that allows you to have a comeback is a collections owner comes back…
John: So it’s good for tracking, for due diligence. And maybe good for institutional procedures – for research projects where you need to know the rights involved. Might help raise awareness.
Table 2: Policy implications wise, there aren’t really any cases that shouldn’t already be covered by institutional policies. Licenses, derivative works, etc. should already by covered by institutional policies. Maybe some special cases…
John: Are the policies fit for purpose?
Comment: It is usually awareness not existence of policies which is usually
Table 3: Possibly a pop up indicating license and appropriate usage, so you know what you can do. Second aspect, if you can legally modify videos – why not do on desktop system offline, if not then how can this comply. Only the making of copies that this removes the issue for. Sorry for a super defeatist comment but how does this differ from what else is there.
Comment: I come at this from two places… Both the way into lumpy content, interrogate, search it, etc… And then also this more creative tool where you make something else available on the internet – alarm bells start ringing. For the creative side, why not use iMovie etc.
Comment: It’s not a video editing tool, it’s annotation. So clearly not that…
John: Useful to use, to make sure we describe it appropriately. It’s a challenge. We need to make it clear what we think can be done with it. We’ll take those comments on board and blog about it to try and make this all clearer.
Trevor: If you were just making clips.. .but in the context of research it’s more about annotations and descriptions etc. But when you have gone to that effort, you want it to look nice.
John: One of our original ambitions was to make it as easy for researchers to quote video and time based media as for print…
Comment: For digital preservation… preserving video is relatively difficult and is an ongoing process. Clips are basically JSON descriptions – easy to preserve.
Comment: A very good content. But I think being very clear on what this thing is for… And making it really good for these things. Really focusing on the annotations and textual aspects more.
Discussion: Service Development Implications
Trevor: Now for our final section we will talk about service development implications: scale – should it be individual, institutional, regional, national, international? Why bother? Benefits? Technical challenges – storage (e.g. 132 MB/s or 463 GB/h), transcoding and archiving; costs; metadata and data models.
Again, much discussion…
John: We talked about scale of this system… There may be a role here for an individual service… For many here will be institutional… But may be national or international. Bandwidth could be an issue depending on resolution.
Table 4: Embargoes, on metadata, and issues of privacy, access, and license for annotations for the same reasons.
John: What about bandwidth?
Table 2: It depends on the video delivery…
Table 1: It’s not your issue really. It’s for content providers…
Trevor: It’s more institutional stuff then…
Comment: The system depends on you having a consistent URI for a playable version of a video… That may be an issue depending on how files are held.
John: What about a Service Level Definitions around persistent URIs? Would that fly?
John: And what about the role of cloud providers?
Several in the room indicate they are using them…
Comment: Making annotations public will help others find your data.
John: Annotations coming up and up as being the things.
Comment: Costs wise it needs to be open source for people to import themselves? And if so, how can you skin it and brand it. And how often does it need maintenance and updates.
John: We are looking at sustainability options, that’s something we want to look at.
Trevor: This is currently funded under Jisc Research Data Spring initiative, and that is done in 3 phases… First stage is reaching out to show there is demand. This phase we are in now is developing our prototype it. And the third phase is to look at sustainability, things like support, update, development community, etc.
Trevor: The last bit for the day is to cover some technical stuff and go through some of that…
Technical Overview – Will
So, just to demonstrate, when you have a playlist you hit “publish” to publish your playlist in various formats. At the moment generates JSON data. A nice quick way to describe data. Annotations are becoming very important so we will need some comma separated tags, and access privileges as well.
Comments: Is there documentation for the code so far?
Trevor: Not yet but software and documentation
Will: Does anyone have any questions about technology elsewhere. We are using VideoJS. We are hosting this in a WordPress installation at the moment – that’s for logins and id generation as well.
Comment: API for Clipper? So others can use the annotations etc.
John: Also discussing a metadata editor for those creating their own annotations.
Comment: If sensitive data, and videos, then annotations might also want to be private… Rather than being on your server..
Trevor: We’d suggest an institutional instance.
Comment: Or could they get a private instance from you?
John: We are not at that stage yet, but that could be an option.
Complex: We haven’t talked much about searching capabilities.
Will: Anything in this text content should be searchable… Might be able to searchable across the board… Might be that when sensitive and private you might have to request access rather than seeing it.
John: Worth making the point that it has to be easy to import data into Clipper, and export data out of it. If this is in a library or archive… We could ingest catalogue information… Could ingest metadata and then come up with an instance to point to. So, e.g. for Scottish Screen Archive you could use shotlist to create clips automatically. So lots of potential when metadata rich environment. So could take in metadata to help generate your collection.
Trevor: Within a project you can search within that project, or more when at the higher level… So we want search to be contextual…
Comment: I think for effective searching you are going to want to have a more complex annotation data structure – so you can do filters, indexing etc. so less computationally taxing and more accurate for users.
Comment: Does the system log who has created which annotation? So you can track who does what on a research project.
John: And with that we will bring it to a close… Thank you all for coming.
Thanks to John, Trevor and Will for today’s workshop and to Gill and the NLS for hosting. If you are interested in attending the next Clipper workshops you can register/find out more here: http://blog.clippertube.com/