Today I am in Birmingham for day two of Jisc Digifest 2016 (catch up on Day One here). I’m particularly hear wearing my Jisc’s 50 most influential higher education (HE) professionals using social media hat, helping to share the event with the wider sector who aren’t able to be at the ICC.
As usual, this is a liveblog so all corrections, additions, comments, etc. are very welcome.
At the moment my expected schedule for day one (this will be updated throughout the day) is:
The chair for this session is Sarah Davies, head of change implementation support – education/student, Jisc.
I have missed the beginning of Heather’s talk, so catching up as she addresses the issue of Area Reviews in FE… Heather is talking about the uncertainty of mergers, and of needing to be confident going forward, ready to embrace a technology led future.
Technology, however, is also a real and substantial job threat. But this intelligence is only artificial – until recently it took huge amounts of computation to recognise an image of a cat. We need to get out there and teach to create the next generation of creative and innovative future employees. We need to address the needs of this changing world through chnging pedagogies, through empowering students – perhaps to organise and teach themselves. But what would Ofsted say about that? Well, it matters, a good Ofsted report is very important for FE Colleges, but I would rather have creative and innovative teaching methods. That means we have to, as Tim Marshall said last night, bring the regulators up to speed more rapidly. We should be looking for solutions through the digital lens of technology
Professor John Traxler, professor of mobile learning, Institute of Education, University of Wolverhampton
Prior to today some of what I will say has been pre-trailed on the blog. I was quoted as saying that “mobile learning” has stalled… But I essentially want to raise the issue of “mobile learning” and just the regular matter of learning with the tools that we have. I was making that distinction around a couple of issues… One is that the money had run out, and that money and that will had fuelled the rhetoric of what we did with innovation in the first decade of this century; the second is the developments and changes in mobile technology itself. About 15 years ago mobile was delicate, fragile, expensive, scarce, something for institutions, and to promulgate their solutions. But the money ran out. And we also focused too much on what we were building, less on who we were building it for… But meanwhile mobile has made the transition to cheap, robust, easy, universal, personal. It’s hardly notable anymore. And whatever constitutes mobile learning now is not driven from the top, but by our students. And the technology moves fast but social practices and behaviours moves even faster, and that’s the harder thing to keep up with. People share, disrupt, discuss… That happens outside the institution…Or inside the institution but on an individual basis.
This technology is part of this fluet, transient, flexible, partial world. It enables people to help each other to learn. And web access is significantly moving to mobile devices rather than desktop machines. But what does that do for the roles of educational designers, teachers, etc. What people call “phone space” is very different to cyber space. Cyber space is a permitted space, back to the world. Whereas phone space is multimodal, you are having conversations, doing other things, crossing roads, travelling… And this is a very different learning space from a student sat at a computer.
Now, looking back I’d consider “mobile learning” rather backward looking, something of the last decade. I think that we, as professional educators, need to look outwards and forwards… And think about how we deal with this issue of abundance – how do we develop the criticality in our students to manage that. And we should question why they still come to us for face to face experiences, and to think about what that mean. Hence, I’m not that bothered if mobile learning actually is dead.
We are a registered not for profit in the US, we have been described as an Apache Foundation for Education – that’s not quite right but gives an idea of what we do. We provide software including SAKAI, Xerte, and OpenCast (capturing and managing media at significant scale). But enough about us…
Next generation digital learning environment… Lots to say there but I will be focusing on a conversation that has opened up in the United States, and the relationship of that conversation to developing the discussion around Learning Analytics.
That conversation was started by Educause, which looked at the VLE – the benefits but also the drawbacks of being inflexible, of being very course or teacher-centred. And that work highlighted what a new VLE might want to look like – flexibility for different types of courses, that it should support collaboration across and between institutions, that it should support analytics for advising, and that this new environment should be a much more personal environment than what has gone before.
The analogy here perhaps is of Groundhog day. These are issues we have heard before over the last 10 years. But why do I think the environment is different now? Well, we are are more mature in our technology. We have gotten smarter and better at lightly working tools in and out of different environments. We are pragmatic about bringing functionality in pragmatically. And, lastly, we are starting to learn and develop a practical use of big data and learning analytics as a potential tool for personalisation.
I just want to pause to talk about academic analytics – about institutional trends, problems, etc. versus learner analytics – which are specific and personal, about interventions, retention etc. And we are already seeing some significant evidence about the effectiveness of learning analytics (see recent Bricks and Clicks report), with examples from the UK and US here. If one looks at the ends of the continuum here we are starting from prediction for retention intervention, but moving towards predictions for personalised learning.
There are several approaches to learning analytics at the moment. One is to buy in a system. We are taking a very different approach, developing a platform that uses various flexible components. That helps ensure data can move between systems, and that’s an issue Jisc has been raising – a national and international issue. And I think yesterday’s opening session was absolutely right about the importance of focusing on people, on humans. And if you look at the work Jisc has done, on ethical issues and informed consent, that is having an impact nationally and internationally.
We work with the society of analytics research. And there is a Solar analytics maturity framework. We have partnered with Solar and Jisc on our work and, to finish, I’d like to make a shameless plug for our Solar colleagues for LAK’16 which takes place in Edinburgh this summer.
I asked all of you, and those online, to help me to design the cover for “Wandering While Wondering” and I want to thank you all for that. We are doing a speed chat on Twitter and I’d invite you to participate there.
My background is in languages, and in being a translator. But mostly in the world of books… I felt privileged when authors of books I was translating were alive, and available to talk to – but usually asking for help wasn’t possible. Reference books are still important for me… But the world of web 2.0 has changed how we can ask for help, the resources we have to hand. And I stopped working in translation when the world of digital became much more exciting. I’m now a learning developer working with colleagues to do exciting things with technology.
I’ve been asked about my work on playful learning… technology shouldn’t be driving but we should be able to play with new things – like Google Cardboard – and discover how to use these things in teaching! But not everyone feels playful… And sometimes people’s wings are cut by the culture, by experience… Sometimes we have to remember that we are physical beings, with room to play and discover (Chrissi is throwing balloons around the room and walking the auditorium at this point).
We are very lucky in my university to have real support for creativity… We have a creative module, where teachers and learners work and learn together, locally and globally on projects. We can be tempted to put our learners and academics in boxes, but we have to get out of our silos, put our egos aside, and work for the common good. We all want to do it but often the fear is there, colleagues play it safe because they don’t know what is going to happen (by genuine coincidence one of the circulating balloons pops at this point!). Being silly, being playful, gives us that freedom to get out of our boxes, to engage, to empower, to educate.
Sarah: How do you give staff the space and encouragement and permission to play, in such a time-pressured environment.
Chrissi: We need to make that space by getting rid of stuff, to encourage experimentation. We stuff the curriculum with stuff… But that space to play is essential, we need space to play, to experiment, to think of new ideas…
Heather: I fully agree with that. You need to understand what happens in your organisation, and highlighting and encouraging good practice. Build models of DNA, rather than studying them. Go outside, rather than doing everything in the lab…
Chrissi: That is really important, that need to use the space creatively. We have outside spaces we can use… Even a change of scenery can refresh our minds. We have to have that openness.
Ian: I tihnk this is also true in digital space. One of the problems for staff in digital spaces is that you need to make business cases to the IT department, get approval, and then you can do something… I think moving to something like the App Store for educational technologies, where you can experiment more.
John: I’m nervous that for students being on campus endorses their identity as students – being a distance learner or part time learner is a very different identity – and play might not do that for them. Students can have more serious conceptions of learning. But I’m also nervous of play as prescriptive practice – our students come from a huge range of cultural backgrounds and in some of those contexts play is seen as frivolous, we have to be aware of that and not go in too fast and miss the opportunity to engage.
Q1 from David on Twitter) You haven’t talked much about the purpose of education, can you say a bit more about this.
A1 – John) If we are merely serving the economy in our work with students, we have to be aware of that, what that means etc. And that’s a concern about VLE – they won’t use those spaces as soon as they graduate. And that’s also part of criticality – our students have to have the skills to adapt and develop, to manage their
A1 – Sarah) They say that we can expect to have
Q2 from John Kirrimuir on Twitter) When the evidence for a technology is scant, how do we manage the risk of playing with it.
A2 – Heather) Use your nouse, but it is about risk taking.
A2 – John) We had a lot of small or limited experiments without evidence bases… But you do create that in those experiments…
A2 – Sarah) And it’s weighing up what you want to do in your teaching and learning.
Q3) I work in academic development around criticality and digital literacy skills, and this is very much about teaching students how to learn. That goes ok. But how about dealing with CAVEs: Colleagues Averse to Virtually Everything… those who don’t want to learn.
A3 – Chrissi) I think we need to immerse colleagues into these possibilities, being part of that community, not just modelling practice. If it’s something we look at rather than trying and engaging, then it doesn’t work. It’s hard to do but if we work together we can change everything. There are habits, including bad habits, but it takes time and can happen.
A3 – Heather) It’s also about providing a safe environment to experiment and take risks. A number of institutions have created “learning laboratories” to trial ideas, to film or share those… A safe space which can work well. There is also quite a bit of student pressure, in FE at least, as they are getting something different down the hall.
A3 – Ian) I think that criticality and willingness to fail is absolutely essential for creativity, for many forms of learning… That has to permeate the institution.
A3 – John) I’m nervous that we are too introspective, deciding what learning is or is not… that we should not define this.
Comment, Sarah) In that context where learners have so many tools and opportunities in their own hands, what is the role of the university?
A4 – John) Well we can give them a degree, a crude generalisation but still true.
And, with that slightly controversial comment, we are done for this session.
Chairing this session is Neil Jacobs, head of scholarly communications support, Jisc.
Neil: We have a very distinguished panel for this session on new directions for open research: Tom Crick, Ross Mounce, and Cameron Neylon. We’ll have all three talks and then opportunity for discussion and questions.
Cameron Neylon, professor of research communications, Curtin University, Australia
I’ve been involved in open research practices for a long time now but I think things have really started to change. Open data was fringe, maybe 5 years ago, now mainstream. Open research is increasingly of interest. And we are starting to think more about open source software, to deeper engagement. We started with people outside the standard practice, and it has now moved into the mainstream of what the institutions we work within do. Institutions are academic institutions, but also funders, also institutions in terms of the way that we communication what we do.
We have seen a move towards institutionalising these practices, primarily towards Open Access. And I’m sure many of you have faced your institutions trying to institutionalise those processes of open access… And as we do that it’s important to think about what we want these institutions to be in the future, what that should look like. We’ve been retrofitting this stuff into institutions, what they do, mechanisms for recognition and reward. What does it mean to institutionalise these things? What happens when we haven’t thought about the unexpected consequences of this. And I’m as guilty of this as anyone… Maybe those mandates haven’t played out how we’d expect in how researchers do their work.
I have moved from being at a publisher and back to being a researcher. And I’m now a researcher in a cultural studies department rather than my original research area in the sciences. And seeing the world through that cultural lens changes things. I can see the ways in which we see things as “cultural challenges” in particular disciplinary communities… In doing this… Well we say it’s cultural… Maybe it’s digital… But I think we’ve also used that as an excuse not to probe too deeply what’s going on. To ask about the culture of the research administrators in our community… These staff are familiar with traditional platforms, and publishers, and mechanisms… As we institutionalise open access practices, what does that mean about the cultures are, and how they can change. How do we move beyond “this is a cultural clash” to asking the question of how different cultures are in play, and how they might (and might not) change.
Ross Mounce, postdoctoral research associate, University of Cambridge
I’m going to talk about some of the tools and facilities already available for open practice. And I’ve been part of setting up the Research Ideas and Outcomes (RIO) Journal (riojournal.com), so I’m going to show you a promotional video for this. (The video is highlighting that the platform publishes research proposals, but also supports openness throughout the process of research. Whilst this plays I’m also going to note that I’m delighted to see that Ross is sporting a Software Sustainability Institute T-shirt today. Find out more about SSI’s brilliant work on openness and open software here, and through the JORS journal).
So, 90% of research proposals are never funded. And even those that are funded you see only a small part of that – maybe an abstract. As a researcher that makes it hard to see what is going on, to avoid repetition, and to collaborate. So RIO has been set up to share research proposals, as well as data, traditional papers etc. We aren’t the only ones – there is also F1000 Research and ThinkLab. And to find out more I’d recommend Mietchen 2014 (The Transformative Nature of Transparency in Research Funding).
Tom Crick, professor of computer science and public policy, Cardiff Metropolitan University
That was a really good framing from Cameron and Ross on the larger institutional stuff. We know that there are bigger funding and sustainability issues. But I wanted to come at this from a more personal perspective… As an academic and researcher I’ve always cared about openness and cross-fertilisation, at a personal level but also on a bigger governance and policy level. Openness promotes excellent research and science. How do we make sure that we know… There are clear economic levers, and broader economic and societal levers… So how do we ensure this approach is both top down and bottom up, and that that all works.
My job combines this odd combination of computer science and public policy, working with Welsh Government, with NESTA on data-driven policy making, particularly in Wales… Looking at how governments can do much more bespoke and evidence based policy making. Wales are keen to be a more agile economy, but how to do that. We have looked at available data sets – real time effective data for public policy making – and that is very hard. There’s a lot of “Open” out there, but having skills, tools, infrastructure and ability to use this effectively is harder. There is a UK Government push to openness, there is a big Welsh Government push to openness… And this work was only possible because there is so much open data out there… That’s a great way to look, to find structures and infrastructures that have impacts on us as academics but how do we really promote interdisciplinary practice, to have that funded without having to shape specific angles for funding from specific research councils.
We need to think about funding, reward, recognition, and governance structures that really can promote open research. Ross is actually wearing a t-shirt for the SSI, who I was going to mention, and they are supporting the research base, the openness of software. How can we embed a culture of openness throughout academia? I look forward to your questions and comments?
Q1 – Amber Thomas, Warwick) I have questions about drivers for openness. In sciences you talked about reproducability, in social sciences it’s about data literacy perhaps, in Humanities it’s much more about collaboration… How do you see this playing out?
A1 – Cameron) I think that thing of cultures, and of social as well as disciplinary and community drivers. I think we need to think about the stories that researchers are telling themselves. As someone new to humanities research I find discourse analysis really powerful, getting those things to align will get us towards progress.
A1 – Tom) A month ago I was chairing a panel at a Jisc/AHRC Digging Into Data workshop, that’s a clear interdisciplinary initiative. We had a data scientist, a poet, and a German Central Banker, all talking about projects funding under the DiD call… But there were some issues of common language or understanding but there was a commonality of what they wanted to find out. When you find cross cutting things – and data can be a great focus for that – you can find alignments of culture, a common aim to solve common real world problems. Maybe one of the things we can pitch openness as is for the greater good, but that doesn’t always work… But open can mean new possibilities and opportunities and solution.
A1 – Cameron) It can be tempted to talk about the objects, but actually we need to talk about who we want to talk to, why, what the research is… less about data etc.
A1 – Ross) Disciplinary boundaries really don’t matter that much, they are quite artificial. Some of the most creative and brilliant academics I know are totally interdisciplinary. I’m an SSI fellows, I can talk to other fellows who are historians, or are in totally different areas, but we find common interests and concerns.
A1 -Tom) Universities try to pump prime this stuff with early career researchers, but there has to be more fostering of that across the board, at other levels.
Q2 – Adrian Stevenson, Jisc) You gave examples of research, but I wonder if you have good examples of research being done in different ways because of the open agenda.
A2 – Ross) Yeah, there’s Timothy Gowers Polymath project, who put out his questions, did the work with the community, and entirely bipassed traditional examples.
A2 – Cameron) A classic example was the e-coli outbreak where raw data was released to GitHub, and all started working collectively on that. Similar has happened for Ebola and Zika. But those have been “turning on the open thing” as needed. But the real examples are often from open source software which have been built that way to avoid a lack of funding for anyone else to do it. If you take that radically open approach you can find you are outside the traditional reward and recognition structures. I know people in this room have been frustrated by people publishing on things which have already been shared/blogged before but not noted in the same way for being in that format. So that is a challenge.
A2 – Tom) As Cameron said, we can get too focused on data/DOI etc. when actually you need that big challenge, that big goal, that research that you are working towards. It’s not just to tick a box on a form. Whether that scales up and down, looking at standard research models that’s more challenging, but if you can start to tackle this super-nationally and nationally that gets things moving… Need that shift before we see things as out there, as radical.
A2 – Cameron) I am part of a project called the Open Collaborative Science in Development project, where we do research, we go to communities to find the needs. But when we say “open” we have to think about “to whom” and “to do what”. And that matters as openness can hit local political agendas at odds with the research agenda – for instance over water quality. But there are increasing numbers of examples, in different ways.
Q3 – Daniela, Jisc) Cameron, you were talking about institutionalising practices, and I was wondering if you could say more about your vision for what this is, and the better practices that could be institutionalised.
A3 – Cameron) Let me start with something that hasn’t worked as well as it should have, the RCUK open data requirements. The mandate there was ahead of where the institutional support and resources were in place. We can’t always build platforms to respond to change rapidly enough. Framing things as compliance issues created a compliance culture, and that doesn’t engage researchers to actually engage and change things. So, what could we do or should we do? We have to build institutions better. Our institutions were build for mid century needs and they need to change in terms of governance and structures. I think we have to look very closely at the way that institutions administratively engage with researchers. We need more research to actually understand what is happening there. I see a culture of researchers, administrators, and funders that are seemingly at odds, even though they don’t want to be… We have to dig into that before we can improve structures. But that’s a hard problem. Within the frameworks we have, what can we do to ensure researchers and administrators both meet each others needs, and can understand those needs. Administrators have to engage in compliance, standards etc. Researchers have a different set of concerns. How can we align, and do that at scale, to find how we can collaboratively work to solve these problems… Rather than fight based on misunderstandings and misconceptions. More collaborative practice would be really powerful, and that really changes the way that senior university administrators see their role – away from CEO type roles towards Community Leaders.
A3 – Tom) I can give an institutional profile. I am half time academic, and half time working on our university’s commercialisation and enterprise work. I have access to some of the levers. A lot of this working is about promoting the benefits of collaborative research, research tools… Understanding that collaborative models lead to funding, that there are easy workflows and infrastructures – I write all of my papers on GitHub with other people… Once you make it easy to do this, you can promote that and cascade it down as well. We care about compliance, but also try to be proactive in changing research culture across the institution and disciplines.
A3 – Ross) Collaboration is easier when you share research easier – the proposal, RDM plan, the data… Right now research proposals are really quite closed, and researchers want to keep it that way for competitive reasons. But actually sharing early helps with compliance.
A3 – Cameron) Compliance is a useful lever… But researchers tick the boxes and then forget about them… In the case of DMPs, how do we make those collaborative between the researcher, funder, and institution, so that they are part of the record of what is expected, what happens, part of the manifest of data at the end of the project. So, this stuff depends on local context but also DMPs can be dead ends now, we need to make them a real living breathing part of the conversation.
A3 – Ross) DMPs aren’t open, and that’s the issue… I am a data parasite, I use other people’s data… I’m a bioinformaticist. And if I see that data is being generated, and will be released… I will look for that data, I’m contributing to that compliance data and I’ll report that if the data isn’t available. We really have to open up DMPs.
Q4 – Rob Johnson, Research Consulting) Tom, I was really interested in your enterprise director role… And in my own work I speak to different communities but opening up data and research opens up lots of enterprise opportunities.
A4 – Tom) Actually I’m deputy enterprise director. Now enterprise can look like consultancy – which many academics don’t feel interested in that – or in commercialisation which can seem far from their research and teaching. But it’s about not being bogged down in terminology, but understanding the challenge you want to solve, the people you want to talk to… But not badging that in ways that may put people off. It’s hard though. I got involved through research data management… That’s tricky but means you can kick interesting projects together.
Q5 – Paul Stokes, Jisc) We’ve heard a lot about enterprise, economics… But ultimately someone has to pay for all this. Do you have examples of direct economic benefits of open work that helps make the business case.
A5 – Cameron) I can give small scale examples. I left PLoS a year ago and have become a consultant half time, that’s very
Public Genome Project – massive economic activity from that. But we keep missing the big things… GeneBank, PubMed etc. as they are so big and important, forgetting that they had to be fought for. And it can be hard to look at smaller stuff, and to account for that impact. But I also want to push back a bit about the economic impact for each part of the research, perhaps we should make the case that some infrastructure is not about a clear financial return – greater participation in democracy has an economic impact but not a clear financial one. So we should think beyond the financial ROI. But, that said, there are big examples, there are big platforms, there are great REF Case studies… But we have to be wary of focusing on financial ROI.
A5 – Tom) Thinking about open data in government, and in Wales we have projects around innovation in public services. If thinking of open data as financial ROI for private sector, that’s tricky and requires finesse. But there is public infrastructure that others can build upon… It’s a non zero cost but it doesn’t have to revenue generating, or producing financial ROI.
Q6 – Dashia, Cambridge) I am a PhD student and there seems to be a trend for publishing papers into social media – what does that mean for openness… Rather than repositories. People want research visible, not as clear if open.
A6 – Ross) ResearchGate and Academia.edu are not open and not the same as a repository… Many researchers don’t understand that difference between free and open. And they are indicative of a failure from institutions to present repositories in easy to use, beautiful looking ways…
A6 – Cameron) It’s symptomatic of failures of institutions and join up in institutions. There is a kind of Napster for research at the moment which is similarly emerging from disruption from an unmet demand. There are opportunities around that demand, and that could be met by the community at low cost… But we often miss that opportunity and that’s where those commercial platforms set up. The outreach director at Mendeley says “Academia is unwilling to invest in anything until it’s so important they can’t afford it”! And that usability etc. is part of that. What do we need to do to invest properly, particularly to move away from direct immediate financial ROI as the requirement.
A6 – Tom) Social media is another way to facilitate and enable openness in a general sense. For my discipline we have preprints in ArXiv. It seems crazy to write a paper collaboratively and openly and not put it there. But I use Twitter, I blog as an academic – good practice myself – but five years ago that was generally seen as a waste of time. Now we’ve seen a step change of the perception of the researcher. Social media is being promoted in institutions through courses etc. But those tools connect you to networks, to those beyond your discipline, and it makes your work visible.
Christopher Brown, senior co-design manager, Jisc
The context for data discovery here is that there is CKAN, but also an Australian national data discovery service. So, in this project we looked at what else was out there, and we selected CKAN as the solution. The aim was for a UK data discovery service to meet Jisc customer requirements. And this is a project of Jisc, the Digital Curation Centre and the UK Data Archive and their UK data centres.
We have 9 HEIs (including University of Edinburgh) and six data centres engaged in this project. And we have a governance structure and a researcher group advising us to ensure our solution meets researcher needs. Why do this? Well to make data more visible and transparent, to promote the HEI or Data Centre research, to encourage re-use and sharing of data, to validate research, but also to support various mandates on the sharing of data. And this sharing creates the potentiall for greater cross-disciplinary and cross-institution collaboration.
So, we gathered user stories through workshops, for instance around the researcher, the project manager, the machine (as user of M2M services), the data repository, the system manager, the funder. So, this accommodates a range of use cases, for instance discovering data or a lack of data as a researcher, to inform the shape of future research.
In order to make this happen we are aggregating metadata around a core schema that maps individual data repositories to the UK National Repository Discovery Service. We engaged with participants in workshops and online meetings, gathered those stories, selected CKAN software (used for CKAN and ANDS) and created a statement of requirements. And we now have a publicly abailable alpha site: http://ckan.data.alpha.jisc.ac.uk/.
There are a number of issues, including the quality of data, the completeness of the service – as you want a service like this to be as complete as possible. There are issues around open access, licensing, copyright. There are copyright issues around some metadata too. Access to external data may require a log in. And we need to ensure functionality meets requirements.
So, we have an alpha site, and we are using a rapid development cycle towards a Beta. To ensure we don’t leave system testing until the end, and to ensure our system meets user requirements. It’s all open and acceptable. There is an open metadata schema document (http://bit.ly/1QZVMCo) and you can also see the scope of the datasets in a shared document.
The timeline here is that we will be working on the business case and Beta site in Spring, before possibly going into production.
And now, for the demo. And, as I say, this is clearly an alpha site.
Chris: I’m Chris, developer for this project. We decided to use CKAN… The landing page presents a search of datasets, you can also explore by organisation – which shows all of those involved, including universities, research centres, data centres. You can search the data sets by keywords, and filter those by various things – institution, license, tag or format. When we look at a particular example we can see a description, the associated resources, the tags for that data, and then you have the additional information here. In a future version we’ll add four more metadata fields as well.
If we take an example document, you can either preview or download the data. The preview uses a plugin to let you take a quick look at the data, as well as a description of the data. And then, if you want to download the data you can download the file(s).
Christopher B: We don’t store the data, we just aggregate the metadata, the download link will take you to the data in the host data repository – whether at an HEI or data centre.
Chris: Some of the data we have has geospatial coordinates and you can use a map to select a region to explore/filter the search results to a particular area.
On the site, in addition to the pages shown, we also have About and FAQ sections.
Christopher B: That FAQ section is being populated as we go, with questions about the data, the metadata, etc.
Q1) How are licenses handled?
A1 – Chris) We are looking to have a standardised list to filter by.
A1 – Christopher B) We are working with existing data and that creates challenges – as there are different versions of the same licenses, and different licenses all in place. So we are trying to encourage standardisation.
Q2) Is the widget that previews files limited to specific files?
A2 – Chris) Yes, limited to PDF, excel and CSV files right now. There are other plugins that could be brought in for other formats potentially. I’m working on a visual search, using thumbnails – the VADS data set will support that. When we preview that there will be images.
A2 – Christopher B) Again, that’s about available metadata – VADS has that, but not all services do. Have to balance what we can do for some data with what can be done across all data sets.
Q3) Could you explain what needs to happen technically for an organisation to take part in this?
A3 – Christopher B) Mostly OAI-PMH.
A3 – Chris) Mainly harvested by CSW – Catalogue Service for the Web. Also supports OAI-PMH – a common standard for harvesting metadata, that supports a range of common metadata schemas including Dublin Core. And we set up periodic parses of data. Large data sets are checked every week, smaller data sets are updated every day. The metadata shown is what we can work with from organisations involved, and working with organisations to standardise.
A3 – Christopher B) There will be requirements for organisations of what is needed to supply data – like the European Data portal – but that guidance isn’t there yet. Coming soon though. There will also be the ability to search for any metadata field from the advanced search feature – coming soon.
Q4) Is it intended to be entirely open access to search?
A4 – Christopher B) Interesting question. Lots of additional functionality that could be solved by institutional login. But yes, right now, it’s open to use and search. But for the data download you will need to login.
Chris: I should say that there are different levels of production of metadata, depending on the institution. Is there anything else that people would like to see. That complex search functionality for instance?
This is available online already so do use it, test it, and give us feedback. And understand that it is in alpha. But it’s all open – CKAN is in GitHub, our schema is open, and as it develops we’ll be publicising the service more.
Q5) Do you have any way to check that the data archives being harvested are actually live an online?
A5 – Chris) That’s a great question. I do check the output from the harvest and any issues are automatically emailed to the admins at the appropriate organisation.
A5 – Christopher B) We are also producing reports, for instance of duplicate titles. And again the reports are being emailed back to the appropriate organisation responsible for that metadata. Also, one last thing… People have asked if you can save search results and you can do that through the URLs for those search results.
A5 – Chris) The reports are in development at the moment. I can show you a preview of these… These include duplicate titles – no good having the same data titles several times – so this flags all data of the same name, and can work with institution to resolve that. And other reports include view reports for the service.
Q6) Is there anything that indicates that data is open access before I download it?
A6 – Christopher B) Yes, you can filter by that, like you would on Flickr. But that will be in a later version as we need that data from the various contributing organisations.
Do keep an eye on the project as it develops and thank you all for coming!
13.30 – 14.30 Plenaries: the power of data
What can data mining the web tell us about our research?
This session is chaired by Dr Paul Feldman, chief executive, Jisc.
I’m going to be talking about the power of data, specifically data that can support the research process. I work for a company called Altmetrics (which around half the room indicates they’ve head of).
Data can help us capture several aspects of research:
- Quality – is this good research? Is it well researched? Is it reproducable?
- Engagement – is this research reaching it’s audience, the people it should reach.
- Impact – what do the audience do with that research once they encounter it.
These three things are different… They are related but not all one thing. You can have a great piece of work that gets no attention, that’s valid, that happens. And you can have a terrible piece of work that gets lots of attention – happens particularly in mainstream media coverage of science.
Today, I won’t talk about quality but will instead focus on engagement and impact. What I will say is that quality is often best assessed by someone who knows.
Impact is hard to define… It’s a bit like obscenity… You know it when you see it! And impact has different meanings for different people. For researchers it can be about reaching peers, for funders it is often about reaching out more widely thoug. As a very broad definition, impact is about showing that your work makes some kind of difference. And that matters in the REF because people want to know what has happened as a result in funding and investing in research. We have a responsibility in the academic and research community to show how our work has impact.
So, how do we know if our work has impact? Well it has impact if it is cited in Science or Nature of PNAS. But citations are just one measure of impact, that doesn’t account for wider practical applications of our work. So if I publish in Nature, I’ll get picked up by all kinds of citation tools. But what if my research is published in a document like the World Health Organisation Guidelines for the treatment of Malaria? It’s research based work, developed by a panel of experts. Why should you get credit for a journal article, but not a policy document?
Maybe you publish in humanities or social scientists… What if your work becomes a book like “Thinking, fast and slow” or “Freakonomics”? If my work is cited, is a chapter etc. why does that not count? What about patents? Shouldn’t I get credit for that impact? So, all of this data can be collected, as can Social Media and discussion of work there… There are traces that act as an indicator that can be presented to individuals for them to make a decision on what matters.
Now, I left social media until last because people often think of Altmetrics as being about tweeting and blogging and so on. It’s a very high volume source, but not always the most important for impact (more so for engagement perhaps). But there is value there too – tweets about a research paper by Greenpeace and Barak Obama shows real impact for that publication.
The data is online, there’s a lot of it to draw upon. But it’s not always immediately useful. The right tools, plus human context, moves us towards something useful and important. And so, we come back to altmetrics and the altmetrics manifesto… This document was written in 2010 by several people, including Jason Preen, Cameron Neylon, and was about these tracks and traces, it talks about recommendations and discovery and understanding that too. And various people are looking at this… Altmetric (that’s me), Impact Story, PLOS, and Plum Analytics.
The Altmetric journal is now on about 6000 journals and will let you explore the usage of the academic work… That brings together newspapers, magazines, tweets, blogs, etc. For example, we can look at the Gravitational Waves article that came out recently.
Now, I should talk about numbers. The score isn’t so important, it is the pooling of the data which you can then look at and explore in more detail. Only you know what impact you are really looking for.
Another solid example was on UAVs and drones, and in Altmetrics they found that their work was being cited in a government document. That’s impact and engagement and useful to know about.
Now, I’ve said this is all good but you have to be cautious about quality. Knowing the Altmetrics is just a starting point… The data needs critical engagement. And these measures are complimentary to the traditional measures – if peer review is the bread and butter, Altmetrics is the jam. And Alt suggests alternative… but it’s complementary. Metrics suggests quantitative as the focus but that’s not right either.
And when we talk about Altmetrics we need to be cautious of not making the same mistakes others have made before… With Bibliometrics we can all things of examples where they have been used poorly, but also good usage too. The Leiden Manifesto for Research Metrics was a Comment piece in Nature that I recommend reading and engaging with, it talks about what you should (and should not) focus on.
You’ve heard me talking, but the best way to understand this stuff is to try it for yourself – look up your work or that of your researchers. Look at http://altmetric.it/. And if you are a librarian you can access a bigger tool, free for libraries, http://www.altmetrics.com/.
Nice to be back at a Jisc event, 10 years ago I used to chair a Jisc Committee on research. Jisc have done some great work in supporting research – this conference is more on teaching and learning but they still play an important role in research.
Research is increasingly moving towards data intensive research, and in fact about 10 years back the UK had the eScience initiative which put us ahead for being part of this. Much of science is now Data-Intensive. The Higgs Boson researchers, your Bio Informatics folks are intense data users, with petabytes of data. But we have a long tail of science and many use off the shelf tools with discreet data sets. But in those data intensive space we see not only scale but variety, combining of data from different sources and specialists. We also have data from sensors. And we have volume but also velocity of data accumulating.
Probably one of the easiest ways to start is to talk about the Sloane Digital Sky Survey, which kicked off in 1992 and “finished” in 2008. That project took surveys of more than 1/4 of the night sky, producing 200 GB of data per night. It was two surveys in one – images and spectra. There were nearly 2m astronomical objects captured. But astronomers haven’t the time to look at all of that…. And that data had “absolutely no commercial value” so the data was published openly on the web and was published before the research was really done.
This project led Jim Grey from that project to consider an idea like eScience (but not). And I’d talk about this as the Fourth Paradigm. We started in science with Experimental Science, describing natural phenomena; then we had Theoretical Science – Newton’s Law, Maxwell’s Law etc; Computational Sciene emerged in the last few decades accommodating simulation of complex phenomena. Now we have Data-Intensive Science. That started at the beginning of the 2000s and is about scientists dealing with data from instruments, networks, sensors…
So, for instance… Genomics and personalised medicine. You can use genetic markers (e.g. SNPs) to understand causes of disease. My old team at Microsoft Research looked at SNPs working on Wellcome Trust data for second complex diseases… And analysis with state of the art machine learning enabled some valuable insights into causes and patterns of disease.
Another example… I live in Seattle, which is on the edge of an earthquake zone. There is an electro-optical cable on the sea bed which captures data from a network of sensors etc. The issue is no longer too little data, but so many sources that it is hard to know what to analyse and explore in more detail. And researchers like John Delaney do work on oceans, volcanos, and the ecosystem that exist around that. That data is transforming ocean sciences.
Latestly, and this is where I work, we have CEDA: the Centre for Environmental Data Analysis, which pools data from different sources. And there we have the JASMIN infrastructure, which puts petabyte data storage next to petabyte flop processing.
If you want to know more, this book, The Fourth Paradigm: Data-Intensive Scientific Discovery, is (as far as I know) the only book by Microsoft published under a Creative Commons license.
So, that’s data intensive science. But we are moving to a more open world too…
My colleague Jim Grey works with the National Institute of Health in the US. They mandated that you should deposit the data from your research in the US National Library of Medicine. When that was a recommendation they had about 25% of data, when it became law it went up to nearer 60-70%. Now they withold the next grant on condition of data deposit and that is driving deposit more and more each month. So data is increasingly being openly shared.
Now there are problems… Amgen identified 53 landmark publications in bioinformatics and tried to reproduce them. Only 7 could be replicated with similar findings, the rest could not. That doesn’t mean it’s wrong, just that it can’t be reproduced. So that’s kind of an existential crisis.
Now, in February 2013 there was a US White House Memo on increased public access to the results of federally-funded research. This was a directive requiring that federally funded research had to be publicly available, and that the data also has to be publicly available.
Meanwhile the EPSRC has high expectations for Data Preservation. There the requirement is that research data is securely preserved for at least 10 years after it has been shared. Also the Digital Curation Centre, established 2004, was set up to give guidance and advice, and they are key to some of Jisc’s work in research data management, and will remain important.
There is also an issue with the maintenance of data links. We need sustainable links which endure. Research from early 2000s, according to a specific piece of research, was in 44% of cases already unreachable due to broken links. And we see tools like ORCID helping you find specific work, identified consistently and clearly.
Last year I presented at IDCC and Jim Frew, who was presenting there, talked about progress in data curation in the last 10 years. He said the biggest change is funding agency mandate, there are better curated databases and metadata now – but not sure that the quality fraction is increasing. Frews laws of metadata: First law: scientists don’t write metadta; Second Law: scientsts can be forced to write bad metadata! I would say that physicists are a bit ahead of the game here… But still much to do.
So, I hear a lot about libraries. I personally think they should be central to the university. But as Dean of engineering my students hardly went there (other than to study and have coffee), so what is the function of the library? I’m going to talk about their role a bit… I think they need to be central to provision to teaching and research as much as possible. In research I believe the library should be ensuring they keep a copy of the research output (another Jisc example, OpenDOAR).
So, what are we lacking? Is it true that the UK is world leading? Having spent 10 years in the US… Well when I went to Caltech for my PhD, they barely knew about the US, let alone Oxford where I had previously been. So we have to be cautious of statements of world leading.
So, in the US the NSF Task Force on “Campus Bridging” (2011) have been establishing a vision to ensure there is speed in the network as if the data and the processing were next door to each other. This is Science DMZ architecture is a network design pattern, improving the baseline end-to-end performance through ongoing global adoption. That connects universities across the States, but I haven’t seen that done here. There has been an attempt to implement Science DMZ approach, but there is a need for universities to support that too.
Also in the US there is a Pacific Research Platform which is developing infrastructure to achieve data transfer speeds comparable to being located in the same place.
So, moving into the final part of my talk… What is a data scientist? Well there are at least three skills that make up at that role (not that all data scientists have to have all of these) but we need scientist career paths, for data scientists not just PhDs and PostDocs… The final vision, like the National Library of Medicine and PubMed Central (the version adopted in the UK), is that of Jim Gray who has the vision for All Scientific Data Online, that allows reuse of data, and use across disciplines and increase scientific velocity. I see Jisc doing good work there, but I am concerned about that end to end aspect.
Over the last two days we have been looking at the power of digital. I have been in awe of the innovation and passion I’ve seen over the last two days. Digital is disruptive, but education hasn’t seen that disruption in quite the same way. Only 15% of businesses turn to UK universities for support and training in digital skills. There are huge opportunities – are you ready? It’s imperative to understand the opportunities for digital transformation across the organisation in all aspects of the organisation in teaching, learning, research but also operational aspects. With the right leadership we can be great, perhaps the best in the world. Now either we can grab that opportunity or it will be lost. So, how do we win it:
- Digital Learning
- Digital Content
World class research can’t flourish without the Janet Network – we believe it’s already world class but also investment to get to that full vision (as is the case in the US). We are investing in higher bandwidth, in reimagining the network. But there are opportunities to make the best of data, skills, and to ensure use and reuse across disciplines and boundaries.
Digital Learning is transforming everything we do… Have you heard of Colin Hegarty, a maths teacher at Preston Manor school in London. He is in the running for a million dollar prize for his YouTube videos teaching maths to secondary school kids. The sector needs to develop this kind of process so it is not limited to pockets of best practice.
Our learning spaces have to be fit for purpose… Learning spaces must blur boundaries between learning, working, and leisure. Connectivity and wifi is keen – that’s especially a challenge in the workplace. So, can we make Eduroam ubiquitous outside academia, so it’s there in coffee shops, in apprentice employers.
We are at risk in not investing enough in learning and skills. We are paying special attention to FE, for FELTAG, but also have to focus on HE and Research. But back to Apprenticeships. Digital Learning is key to delivering the ambitious apprenticeships the government wants to establish by 2020. Apprentices can feel quite lonely on placement, so we’ve been working to develop a social network for apprentices to help them feel supported and connected.
Turning to data we need to engage with learning analytics – the UK can’t afford to overlook this. We are setting up – a world first – a UK National Learning Analytics service and framework for HE and FE. Wouldn’t it be great if all organisations in HE and FE took part in this, so that we can understand pathways and their role especially for widening participation.
Finally securing access to the best Digital Content is key. You will have heard from my colleagues this week on negotiations around journals, and working with SCONUL on the UK National Digital Library. We are on a real cusp of something here… The future of Digital is in your hands. Digital changes lives. Technology enables students like Alex (featured in a video yesterday) find it life changing for what it enables him to do and participate in. We have huge opportunity and we have to make use of it.
And with that we finish with another view of the new Jisc video, More Power to You.
Chairing this session is Catherine Grout, head of change – research, Jisc.
Catherine: My role is to deliver innovative and smooth services for research data. Before our speakers I’m just going to give you a brief overview of the importance of metrics. Using metrics in an appropriate and responsible way, underpins the research process, grant applications, resource allocation, research assessment, research impact etc.
Advances in technology enable us to use other types of data – sometimes under the heading of altmetrics. But there are potential pitfalls. The HEFCE Metric tide report: http://bit.ly/hefce_metrictide/ was really important in setting out a future where traditional peer review could sit alongside new metrics. It made recommendations about how metrics which might be deployed more effectively. But there are many challenges and pitfalls along the way…
Stephen Curry, professor of structural biology, Imperial College
I was part of the steering group for that HEFCE Metric tide report, which I’d encourage you all to read if you haven’t already. That report certainly reinforces that metrics are here to stay. Quantitative measures are part of all we do, including research.
We did, in working on the report, questioned even using the word metrics. Metric suggests you know what you are measuring. I personally prefer indicator. But “metric” was the brief given to us by the minister.
If we do need to use metrics they must not stand alone. Metrics are informative but must sit alongside peer review. And institutions need to be transparent about use. That means being transparent and properly engaging staff in metrics. They must clearly state principles for assessment. They must not delegate measures of excellence to league tables or journals. And metrics must be put in place with staff. Data should be transparant – one of the biggest isues of league tables etc. And it must build on DORA and the Leiden Manifesto, about the principles for the responsible use of research metrics.
So, this was a great document that I was proud to be part of this. But words are just words. And we have a deep cultural issue with metrics. Despite the REF not intending to force researchers towards high impact journals, our researchers certainly feel that pressure. There are difficulties to be resolved. And the best way to do this is to clobber people with the evidence to make your case. The impact factor number is something simple to mask something complex. Data on Nature Materials citation performance shows how variable impact really is. 50% of citations came from a very small number of journals – it’s a long tail… And that happens across journals. ALL journals publish papers with high citation counts, ALL journals publish papers with low citation counts. Citation distribution data should be shared for all publications – I’m very pleased to see the Royal Society doing this, others are following, all should.
We also need to understand that the correlation between JIF (Journal Impact Factor) and article citations for individual scientists are poor (Seglen 1997, BMJ, 314, 498-502). That paper is 20 years old but it isn’t known and we have to ensure that is known. We don’t want to reinforce the unintended and bad effects of metrics. This isn’t the fault of those who publish metrics’ fault but we have to reimagine what we do in academia.
We should be working towards high quality research that is highly reproducible, not just novel or shaped for high JIF journals. I’d like to see open publishing and efficient publishing at all stages of the process. And I’d like to see mentoring and citizenship recognised – e.g. contributions to peer review. I take Ron Vale’s cue that
“as stewards of our profession, academic scientists have a collective responsibility to consider how to disseminate knowledge through publications and how to advance (careers)…”
Cameron Neylon, professor at the Centre for Culture and Technology, Curtin University, Western Australia
I guess we should check our metrics for this session in the form of social media sharing… Or maybe even email sent (a negative measure).
Now, I’m going to disagree somewhat with my learned colleague here. We know that metrics are very flawed. The data we have are largely meaningless.. They don’t tell us what we say we need to know about, or indeed what we do know about. As scientists we are supposed to be testing what is known. So two words for you… Excellence and Quality… Try and define them in your head without using either of them… I think you’ll find that difficult.
Everyone wants to be Excellent… But at the root of this is not just numbers that are inaccurate or unhelpful, but it’s also about a system of research and research funders who don’t know what we’re talking about. In Ewan’s talk earlier he referenced Justice Potter’s comment of not knowing what pornography is, but he knows it when he sees it. And Excellence and Quality are assessed in a similar way. Researchers either know Excellence or Quality when they see it, or can point to those who can identify that. That’s ok in a small community. But that’s a problem and loses meaning when we try to communicate that more widely to funders, to policy makers etc. Applying those ideas to all research is just nonsense… But we’ve internalised this very deeply. Stephen said this is a cultural issue…. A very deep cultural issue…
So, again, think… Can you rank two universities? Two researchers? Two papers? Two grants? You probably said “no” at first… But actually for grants it must be true as we do that. Imagine a world where that is not true. Where proposals are purely about allocating funding to all, not competing in these ways. The idea of ranking is so deeply felt but actually when you pick that apart it just isn’t true. We believe it because we think we have limited resources… I’m not even sure that’s true… The assumption that we only fund the best best is political but deeply embedded.
Lets go back to Justice Potter… He said something that was actually a bit different from the well known quote. He says “It is not possible to explicitly define something as obscene, as there are too many things that might be obscene” but that “for a certain set of agreed principles advanced, I can assess that this doesn’t meet any of them”. So he actually said, he can see when he doesn’t see it.
Now in research we are great at knowing what should not be funded, but we are terrible at knowing what should be funded. We have different priorities. We have to think in terms of qualities, not quality… But that is challenging… That’s about changing research funding, the shape of careers… And that’s pretty impossible. But what can we do?
Well we can move – using my new humanities view on the world – from people to peoples, from quality to qualities… Pluralising changes how we think about the sentences we use – try it! It changes the way we think, The stories we tell ourselves, our narratives, the way we think… that changes… So changing the words is the start to change the culture… Supported by evidence, supported by analytics.
And we have to ask ourselves again and again what we are measuring as performance. The idea of being good researchers and good scholars.
Semantometrics – Dr Petr Knoth, research fellow, Knowledge Media Institute (KMi), The Open University
I would like to talk about an idea we had at the Open University about two years ago, which came out of frustration at current research metrics. We wanted to gamify this a bit. We wanted to find a way to do this in a way that wouldn’t change their work. The metrics should adapt for different behaviours in which we work. One way to do this is to use the full text but I’ll get to that in a moment…
At the moment we have many metrics, but the main advantages of these is that they are very explicit and easy to calculate… But there is insufficient evidence that they measure quality at all. They all developed in an axiomatic way… No one queries the way they are calculated and we need to… We need to move to more data driven model.
So, I just want to show you the rejection rates for peer reviewed journals (n=570) with impact factors. This shows no correlation between rejection and impact factor. And one of the challenges of peer review is that in ranking papers the reviewer cannot read absolutely everything – as Cameron said it’s easy to reject the poor quality work but much harder to understand good quality work.
Metrics are also difficult because there is a time delay. The REF takes publications only from the last 5 years. Citation based metrics are almost impossible to use well for, say, the work published in the last year. Altmetrics are useful but they are measuring popularity, for understanding interest, but is that appropriate for decisions on career progression?
The problem with all these metrics is that they measure interactions (in different ways), without qualifying how important the interactions are. They are very limited. So the idea of Semantometrics is to judge the full work – we are asking them to be good communicators rather than good researchers. So, we can take full text and use NLP to detect good research practices followed, we can detect paper types, we can analyse citation contexts – tracking the progression of facts. We can detect the sentiment of citations. We can normalise by the size of the community who will access the research.
So we have developed an approach for Semantometrics… Each publication builds on prior knowledge and by analysing text we can measure the link to the discussions in the past and those going forward. The higher the distance from previous work, the more impactful the publication. This can be measured, and is natural – it doesn’t distort science. You can find out more at our website semantometrics.org.
To finish I want to talk about what I think we need to do with research metrics. We need to take a data driven approach, where we see how we can test impact metrics on data sets. In data retrival there is the concept of the “golden set” or “ground truth” and we need that to take this approach. But there are also human judgements to be made here. And then there are many factors to take into account – financial impact, etc. as well.
On a related note we did work on the WSDM Cup which was work on new research metrics based on the Microsoft Academic Graph (>120m papers). There is no full text there but the judging of that prize was online through Bing search engine, and through human judgement. This type of model for advancing research metrics seems like a good approach.
So, to summarise, full-text is necessary for research evaluation. Semantometrics are a new class of methods. We are studying on method to generate new metrics, but there isn’t only one solution so we need to explore further.
Q1 – Neil Jacobs, Jisc) Really interesting to hear the different perspectives today… So how do we have this conversation, do we have shared vocabulary even?
A1 – Cameron) It’s the same class of problem as any interdisciplinary approach. It’s actually worse than you suggested: we have some common vocabulary but we mean different things by it! We need deep questions: why do we use public funds for research? What do we expect to get out of that? And how do we assess our performance against that to enable us to do better in the future? We need to have the fights and create new ways to do this… In my head I hear my inner-researcher say “this is interesting, we need to do more research” and that doesn’t seem right either.
A1 – Stephen) Cameron’s point on quality over quantity is well made. It’s so hard to compare work across disciplines… You can’t choose but actually we have to… Right now our society chooses to spend a lot of money on biomedical research, reflecting society’s values. There is a wider public conversation to be had about what we do fund. That conversation has to include historians, it has to involve the public… Part of that process has to include numerical evaluations, and I think that’s ok as long as it’s only part of the criteria.
A1 – Petr) In terms of shared vocabulary, I particularly think thats about impact, rigour, etc. We don’t know what we value at the moment. We have to understand that researchers – especially early career researchers – don’t always choose their work, they are often assigned to it or taking the available role. And they are rewarded based on that. Maybe we should reward quality for their actual work instead. We have to break impact down into what we actually value, and how we will use them for specific purposes.
A1 – Cameron) Stephen, those of us in this space though that the Metric Tide report was excellent, but presumably you had that conversation on vocabularies…
A1 – Stephen) We did and there was disagreement. In natural sciences there was some sympathy for citation counts, but in the humanities you can be looking at an artistic performance where citation count simply isn’t relevant. In Australia they used metrics for sciences, not for other things… But we saw that two-tier approach that can value one side of research more than another as being quite dangerous. The impact element of the REF has been interesting first time around, and good for the sector… But we could see no consistent pattern that could be pulled out as metrics to use consistently in the future. But it’s good that there are so many potential types of impact that can come out from research.
Q2 – Catherine, Jisc) In that Metrics Tide report the findings are quite wide ranging. How do you feel about the next steps? What are the most important recommendations with highest priority to take forward now?
A2 – Stephen) We have 20 recommendations in total. Some of them are cultural. We did aim to draw a line in the sand about where we are and where we want to be for best practice. Some are straightforward and technical – data formats, encouraging all to sign up to ORCHID etc. But we didn’t see an easy alternative to REF peer review panels. Some of the other recommendations around transparency and conversations within universities, those can start right away. At Imperial these conversations have already begun – triggered by the unfortunate suicide of Professor Graham. We have talked about what should be incentivised, where we wish to see contribution. So we are having those conversations and want to take those forward, and I hope other universities do that too.
A2 – Cameron) I think the report did all it could in terms of recognising and pushing those cultural shifts. But an aspect that is needed is the data structures and infrastructure, how things have been calculated, and ensuring that data is available for access for the full range of scholarly criticism and engagement – to apply the same standards to our metrics as we should do for any other scholarly data set.
A2 – Petr) I think that this shows that impact is not just economical, but has real impact on people’s lives. The other thing is transparency. I have experience of comparing just citations. We did work comparing Google and Microsoft citation data and they were totally different, so the transparency is the critical bit. If people are judged on data that is incomplete that is hugely problematic. And if you do research on data sets, you are often not cited (the data set is instead) and that also needs to be taken into account. The ability to go back and recalculate values is important. So I support the infrastructure issue too.
Q3 – Catherine, Jisc) I wanted to ask about “snowball metrics”, which weren’t recommended in the report. I was wondering what your view of snowball metrics are, and it’s place in this sort of area. If there is scope for that.
A3 – Stephen) I don’t know a lot about snowball metrics. We had a submission from John Green, who heads up that approach for a collaboration of universities and Elsevier. We didn’t want to back any single approach but this one has a degree of openness, so healthy to that degree. But it is a project pushing the line that just adding up the numbers tell you all you need to, to shape the strategy of the university etc. And this is where I really encourage universities to have those sorts of discussion of what this means.
A3 – Cameron) There is some underlying issues here. It comes back to public infrastructures for information too. The good thing is that it is a set of recipes – analytical recipes for normalising citations to the income of the university, etc. I have to be honest that the measures available there aren’t that useful for strategic decisions. I’m sure the snowball people would suggest that I create those recipe. What concerns me is that that is a collaboration between universities and a single data provider, so it relies on a specific data structure (Scopus). It is different than Thomson Reuters. And this is no use of public data yet. We need that data to be publicly owned and available and we need to be able to scrutinise that.
A3 – Petr) I’m not familiar with that project as much, but I would say that when I compare data from Scopus and Mendeley – different data sets from the same supplier – they do not match up or agree. And that is one supplier. So what happens when we compare data from different sources?
And with that, and a closing thank you from Catherine, Digifest is a wrap for 2016. Thanks to all at Jisc for organising and to all the lovely speakers, fellow delegates, etc. that I had the pleasure of seeing/meeting/catching up with this week!