Today and tomorrow I am in Birmingham for Jisc Digifest 2016 which I’ll be liveblogging here. I’m particularly hear wearing my Jisc’s 50 most influential higher education (HE) professionals using social media hat, helping to share the event with the wider sector who aren’t able to be at the ICC.
As usual, this is a liveblog so all corrections, additions, comments, etc. are very welcome.
Plenaries: the power of digital for change
Dr Paul Feldman, chief executive, Jisc
Good morning and warm welcome from me and the whole Jisc team to Digifest 16. A warm welcome to those, like me, here for the first time. Digifest is all about the power of digital in education. That video of More Power to You is all about a subtext coming from Jisc over the next few months about people and technologies.
Now I’ve been in post only a few months and in that time I’ve been finding out about how you are using digital. And Digifest 16 is all about seeing the power of digital in practice. You, as well as others not able to be here today, and our online audience will do just that. Some of those articles we shared ahead of today have already had 800 views, and we want to carry that conversation on after today.
Before I hand over to our Jisc Chair I also want to thank our sponsors and partners for Digifest: Pervasive, Danny Boyle, ITR, Optix, Saville AV.
And with that, I will hand over to David Maguire, Chair of Jisc, to talk a bit more about Jisc and what we do.
Professor David Maguire, chair, Jisc
Welcome from me, and from all members of the board to Jisc Digifest. I will be talking about the power of digital, and that you have to have vision as well as pragmatism.
So, Jisc, the body for further and higher education which represents all things digital. We represent shared digital infrastructure, services, advice and expertise. We work with around 1000 organisations around the country. We have a national network infrastructure with about 18 million users in any given year. That is well known. It is perhaps less well known that over 50% of all UK library spend on e-resources comes through Jisc. And we save the sector around £203M annually – about twice what Jisc actually spends to do that.
Jisc is of the sector, for the sector. We do three main things for you. We run shared digital infrastructure and services – including the Janey network but also things like learning analytics, research data management. We provide Sector Wide deals with IT vendors and commercial publishers – examples here include Microsoft 365, Amazon Web Services, Prevent web filtering. And we provide expertise and advice.
One of the challenges we face is the huge growth in interest – a six-fold growth in traffic in the use of Janet since 2010. That growth means we also need to invest in the infrastructure, to ensure that we update our infrastructure to keep it suitable to meet those needs.
I also want to talk about University digital challenges.
Right now there is lots going on – a Digital Wild West. We have BYOD (Bring Your Own Device) – easy to desire but hard to deliver well, Wikipedia Scholars – everybody can find out everything now, limited IP respect – and as custodians we have responsibilities to the IPO, to copyright holders, for things that happen on our network.
We see students moving faster than university policies/systems/practices/staff. The answer isn’t to slow students down, but to be ready to do that.
We have to keep up with demand – building industrial strength solutions. We’ve all heard of academics building something in a weekend, but it takes a huge amount of work to take an idea and turn it into a robust and flexible solution. That’s a challenge across MOOCs, VLEs, student systems, Learning Analytics.
Breadth vs Depth are also challenges for us to address. How do we cater to specialists and generalists at the same time?
Now, the education sector is rather a small technology sector for vendors to create appropriate robust solutions for. We need common information systems to reduce the cost of building systems that meet the sectors requirements.
So, Jisc is working on some ideas and solutions to address those challenges. Right now there is a Janet mid-term upgrade to ensure we can continue to meet that rising demand on the network. We are looking at Learning Analytics. We are also working on more technology and content agreements – to reduce the cost of delivering the same services to you. We are looking at Open Access – currently costing more to access content in Open Access journals than in old proprietary systems. We have FE area reviews. Big push on research data management in readiness for the next REF. And Paul and I are keen to ensure Jisc remains at the heart of Technology-enhanced learning.
Paul Feldman: On technology-enhanced learning there is no point doing that unless you think about the context and the spaces that that learning takes place in. Which takes us to our next speaker…
Professor Andrew Harrison, professor of practice at University of Wales Trinity St David and director, Spaces That Work Ltd
I’m delighted to be here to talk about the interaction of space and pedagoguey. I’m a psychologist by training but I’ve been working in education for the last 20 years. The internet has changes notions of place, time and space. What excites me particularly is I see more blurring between learning, and working, and leisure, blending in new ways.
Now, some people are predicting that these changes make physical campuses unsustainable – there is a famous quote from Peter Drucker on this. Now I don’t believe that but I think that traditional categories of space are becoming less meaningful as space becomes less specialised. I could give you a 30 minute talk on the importance of corridors! The meaning and function of spaces are being challenged. We are under more pressure to use spaces more effectively. And we are really bad at utilising spaces. The typical space usage in HE is 25-30% so we need to try and use space more intensively, and to make that space flexible for less specialised use. So we need some specialised learning spaces, but more generic learning spaces and also more informal learning spaces.
So, how can space support learning and teaching? Ideas about learning and teaching are changing, so what sort of space do we need to create to support interaction and active participation? How do we make spaces integrated, multidisciplinary? How do we support distributed learning that can take place anywhere, any time. And how does that physical space relate to our digital spaces? We need to create spaces that support the pedagaguey – thinking spaces, designing spaces, creative spaces, etc.
But, where I get really excited is your world. Where digital is not just equipped by technology but informed by it. Virtual and physical are not opposites – they are part of the same thing. Even when you are in a virtual space you are still situated. And we have to acknowledge that and respect the continuing usefulness of face to face experiences. My own work particularly looks at spaces that support blended learning. Universities regularly have to reshape campuses to support these types of activities – typically spaces are bigger, with better lighting, acoustics, technology. In HE shared learning spaces tend to be boring – they seem to be thought of as a neutral rather than telling stories, rather than being designed and that’s an integral part of the space.
Now this image (three images of classrooms) is here to scared University Estates teams. A traditional didactic learning set up means rows and rows of students – very efficient. More participative spaces maybe enable pair working. But a more active pedagoguey means a room that seats fewer people in clusters. As we move to this type of teaching and learning we may need fewer spaces, but larger spaces. And this shows the importance of estates and teaching staff working together to design learning spaces.
Some of my favourite examples of great spaces are the Melbourne University Learning Lab – a flexible, adaptable space, and the IED Blended learning classroom in Karachi. At the moment I’m working on the University of Wales Trinity Saint David, Swansea. We have an amazing opportunity there to entirely redevelop a quarter of the city. Part of that relates to understanding the UWTSD library offer, and a VLE that enables on campus and off campus experiences that are comparable. And Wales is leading the world in moving to a single VLE so that all universities and libraries will be connected.
When we talk abot the creatoin of learning-centred communities we can see different levels and types of learners as very different, but I’m excited to think about hybrid spaces blending schools and museums together. There are huge opportunities to explore what that mean.
To summarise I think the future learning experience is much more layered. It is about flexible activity zones to support learning, living and working. And it is about users choosing appropriate settings and technology for the tasks they want to achieve (a study by a US chair supplier years back found students sitting on the tables more than the chairs!). Needs for spaces change throughout the day. Those needs also change depending on the learners context, background, mode.
So, successful digital learning spaces are about:Space; Place; Process; Experience – including those moments before and after class, how do we create a student journey that celebrates learning and its place within wider society.
Paul: This takes us to our next speaker Donna Lanclos – it’s her third Digifest and she is here to provoke us, as I’m sure she will!
Professor Donna Lanclos, associate professor for anthropological research,UNC Charlotte
I was really interested in what Andrew said. We today, just like our students, are here benefiting from being face to face as well as the digital being available. That digital possibility doesn’t take away from that shared experience.
We do need to think about digital as a space. Society-wide we have to think about what does it mean to do the things that we do face to face, when things can be done online. What does it mean for teaching and learning when we can take those spaces around in our pocket. Teachers can worry about attention… But that misses the point, it’s about where people are, what they are doing. That concern about attention is the outcome of a flawed system of handing out tools and telling people to use it. Instead we have to understand what these different experiences means.
What does it mean for those without access to these digital places? When really important stuff happens in those spaces, but not everyone can have access to. It’s not the “digital divide”, it’s segregation. We have to deal with that. We have to move the conversation away from tools and practice. We have to talk about place and experience. Lets talk about people… They can be present, they can be online… they can be engaging multimodally. What does the existance of these digital spaces mean for engagement. What is happening in this room is more than you just sitting there. How does what you do on your life make you more here? That’s something we can leverage, and use. We have to not be afraid of that… Asking students to switch off devices is the modern equivelent of “look me in the eye young man” – and that never worked either!
The theme over the next few days is about leveraging the digital for change. And I’d ask you to question why change, for whom are we changing. When I run a workshop we often end up talking about tools – people are comfortable about talking about that. But at some point the conversation moves from the tools, to the people being engaged through those talks. And we don’t signal that, it just happens. It naturally falls out of that chat, and we quickly turn to ideas of presence and community. So, think about who you engage with in digital spaces… And that will vary hugely depending on your experience (as is the case for engagement in any space).
We have to think about, if we want to change the nature of engagement, then we need to think less about what you have to do, and more about with whom you have to speak. If you don’t want to change, then that is valid and you have to make that case. Likewise, if you want to change things, make that case. Ensure you have moments of reflection to think about that. More than that we need to think carefully about the roles of leaders to make space for that kind of reflection and change, and for there to be safety around the risks of change. Change happens because you are willing to take risks and see what comes next. Predicting the future locks us into something, blocks off other possibilities.
We were asked to to say what we thought you could do to get the most from Digifest. So, I recommend the mapping sessions, as I love those. But make sure you engage with human beings. I’d encourage you not to get out there to talk to people who hand you a tool and tell you how to do it, instead engage with those people asking you what you need to do, who you need to connect to. I would like these sorts of events to be about intention and purpose. You can go to any trade show, you, the online participants, this is the core of Digifest. I would frame this event in terms of the human experience that you can engage with, and not digital as a tool.
So, eventually technology will come into the conversation… But not starting with that gives you a much more interesting conversation to have.
Q1) About the flexible spaces, and the need to include technology… Actually that’s difficult in terms of challenges around power. Moveable flexible furniture makes power hard to manage.
A1 – Andrew) Power is an issue. Battery life helps, tablets help. We also have trolleys for device charging as well, and floor boxes can help. But not all sessions need technology… You can have technology zones around the edge, and flexible space in the middle.
A1 – Donna) Ideally you’d be supported to develop a pedagoguey that works whether or not the power comes up… I spoke with someone who hit a power issue and noted a colleague “taught anyway, without a powerpoint” as if that was extraordinary. We have to manage the untethered spaces… And reassure colleagues who are nervous of failure. No matter what happens you still teach and your students still learn.
Q2) I’m just having difficult visualising students you are talking about. You seem quite generic about different students you are talking about… Thinking about FE and HE I don’t hear cultural inclusion in any of your talks. We have such a variety of competencies and confidence… Can you give me a dimension of different types of students… You all seem to be talking about young students.
A2 – Andrew) I agree completely. The average age of students here is over 21, in New Zealand (where I’m from) it’s 25. The reality is that we have a much more complex set of students, expectations, skills that the institution needs to embrace.
A2 – Donna) My institution has a very diverse student body. Institutions have responsibility to have intentions around what they want their students to achieve. Of course they come in with a wide variety of preparations and experiences, but that shouldn’t mean we don’t have expectations for them. Funding can of course limit the degree to which we can target our work. We have a responsibility to teach the students that show up – not just providing technology support but also teaching and learning support. We should be less driven by student expectations coming in, and be more driven by our intention and ambitions.
A2 – David) I’m not aware of any technology that asks the user what age they are. Technology is neutral in this. And there are real opportunities for all kinds of students here.
Q2) We are seeing grants for disabled people coming in in September, and it would be really interesting to see how we can do more to assist them.
A2 – Paul) Absolutely and Jisc have a real role in promoting inclusion, including accessibility technologies. You’ll have seen in the video an example of inspiring use of technology to widen participation. One of the things that worries me about social inclusion. The first place I visited in this role was Sheffield University. I was incredibly impressed with the quality of technology kit that students had. One of the things we included in our submission to the Department of Culture, Media and Sport, is the need to ensure that we are socially inclusive and that it is not just well off middle class kids who have access to great kit.
Q3 – Dave Starkey, Portsmouth) You talked about collaborative learning spaces and that they take up more space. We have some flexible spaces at Portsmouth and they don’t seem to take up more space.
A3 – Andrew) Yes, on a student number basis they do take up more space – just to have desks, to have space between groups for group working. That’s less densely packed than rows of chairs with flip desks. The way Universities handle this is to timetable more intensively but having fewer bigger spaces. We are planning on 2.7m per student in teaching studios like this, rather than the typical 1.8m per student in traditional classrooms.
Q3) For us we do see some aoustic blend across groups, but that can be beneficial as they learn from each other…
A3 – Donna) We are seeing a huge demand for these types of rooms – asking to teach at 8.30 in the morning to get into the rooms. Making these active spaces available has huge impact. I mean, in what universe did we think densely packed spaces were a good ideas.
Q4 – Brian Kelly, UK Web Focus) Based on those presentations I’m not sure what this event is selling, the talk of a single infrastructure… 20 years ago we had a focus on national provision…
A4 – Paul) Yes, we have national provision to an extent, but we are here to help you do this stuff.
A4 – David) Yes, we provide infrastructure nationally. And there are some economies of scale. But we are very much about promoting best practice and opportunities. We don’t have an agenda here, other than what you as a community tell us.
A4 – Paul) Generally Jisc is moving away from big services. Janet is really important but generally we are focusing on best practice, on tools and expertise that you couldn’t afford as individual institutions, but which you can take and adapt and embed in your organisation. We want to know the spaces you want, the investments we can make to support you to teach your students, whether in HE or FE. Helping us understand what we can do to help you, for you to employ locally in your organisation, is what we want to understand.
Q5 – David White) To what extent do you think digital can make people better teachers or researchers?
A5 – Donna) It can’t.
A5 – Andrew) I think I agree. It can enable and enhance things.
A5 – Donna) Digital doesn’t do anything. It’s people that do things.
A5 – David) I basically agree but for some people digital can capture the imagination and motivate teachers and learners. It can in that sense make people better teachers. If we think the solution to all known problems is digital, that’s not the point. You still need good pedagogies, good learning objectives, etc.
A5 – Donna) I think technology can’t be seen as the solution, and we shouldn’t think of teaching and learning as a problem. It’s a process.
A5 – Paul) I think I would disagree to an extent. The student experience was so much about capturing information when I was a student. Now there is such availability of information that there is that space for discussion, for participation. You need great inspiring lecturers and teachers. But technology enables even less good lecturers and teachers to do a better for their students.
Q6) You’ve talked about the learning experience. But digital is transforming the research experience. There is such increasing availability of data. Digital is transforming the way we do research and that wasn’t reflected in those talks.
A6 – David) Absolutely. I touched on data access and research data management – where Jisc is hugely active. We are looking at informing the next REF and how we can play a role in that. Some of the things going on in Janet are focused on support for big data, for CERN, shared data centres for High Performance Computing, for the Crick centre, etc.
A6 – Andrew) From a space point of view research spaces are changing just as radically. The interdisciplinary drive is a big part of that too.
A6 – Donna) There are absolutely parallels between teaching and research staff. Again that issue of dealing with people through digital places to do the work they need to do with their research, but those motivations are still the same, even as technologies change.
Q7) The best practice you are advocating goes against the government’s practice to fit students in like sardines, to save money per head.
A7 – Paul) I’m not sure I’d agree that that is the agenda…
A7 – Andrew) All the universities I work with are trying to do more for less. But there is also a rebalancing of use of space… And reimagining or reinventing existing spaces to deal with larger numbers, to improve occupancy. But financially that is challenging too. The fee structures coming in does seem to have really changed the importance of the estate to attract good students and staff. Space is getting more attention at all levels.
A7 – Donna) I hate that particular government agenda. In fact I’m a bit “from the future” in that respect as we’ve had that in the US for longer. I would like to see more support and advocacy from Jisc for the sector for better teaching and research spaces and practices. There is a role for advocacey… So that collectively we don’t agree to do more with less, but to leverage shared agendas to push back on that. Or at least to call governments on their claims that they care about education.
A7 – David) It was ever thus. We have always asked for more. I would say that technology can be beneficial helper here, to reduce costs of delivery, to be more effective in what we do. Operating in the virtual world is more cost effective than a physical space. We can bring in wider audiences, and we can reach more people digitally.
A7 – Paul) My view, having come from the commercial world, is that the government is trying to apply the values of the commercial world on the education sector. But I would ask you to put pressure on your own organisational decision makers as they have a lot more power to make opportunities and to show leadership within that agenda.
Paul: And on that controversial question we are done here. So, go out and use our 30 minute break to engage with people!
Improving digital technology skills in FE: the CPD serviceAdvice and practical assistance – Sarah Dunne, senior co-design manager, Jisc; Clare Killen, consultant; Peter Chatterton, consultant; Georgia Hemings, co-design support officer, Jisc
After an introduction from Sarah, Claire is kicking off the session with feedback from students at college who are keen to make better and more effective use of technology. Teachers are looking to engage learners, to do that wherever they learn – whether classroom or home. But teachers are always short on time. For some traditional teaching modes are still their focus.
The sector is also facing challenges: FELTAG suggests the sector moves 10% of guided learning hours online – but who will create the content. There has to be motivation and support for staff in moving to a blended model. We also need to make space for elearning development, providing flexible training. In house access to training and support varies. Lots of content is available but there are challenges about making that work.
Peter: We are keen to hear your views, starting with the question: What are your biggest challenges in developing digital capabilities and opportunities?
Comments from the audience include: an abundance of strategies but not necessarily the time and resources to make that happen. And the challenge when things dont work all for 100% of the time – ensuring confidence and trust aren’t negatively impacted by that.
Peter: What about content?
Comment: Theres information out there… but you cant just take that and put it up on the VLE. So you have to make it c;ear what can be used, how to make that easy, and what you have to do to use this sort of content.
Sarah: Im going to talk about what we are planning to do, this is ore aspirational at this stage as this session is part of our planning process.
So, FELTAG is informing our work – it isn’t the sole driver but it is useful, particularly the findings on digital capabilities. Indeed Jisc has been doing work already in this area, underpinned by our framework for capabilities, which breaks this area into six key aspects.
So, to address some of these needs we will have a discover tool which enables you to assess your own digital capabilities, to understand which resources will be of most relevance, where there is scope to develop your skills. And this will helppeople access advice and support.
Second, we will have a Learn area, directing you to resources, with community ratings and reviews. This will be frames around specific digital capabilities and themes.
And we will have Build activities – an online activity builder app – a tool to assist with embedding digital approaches to learning and lesson planning. This will be later in the year, but will let you upload content, choose materials by level, etc.
And we are supporting Meet opportunities so that you can review and rate apps and learning resources, to develop your knowledge base and contribute resources, providing opportunities for collaboration and sharing of experience.
And finally, we are very conscious of the need to Find and Reuse a route directly through to learning objects and instruction on how to repurpose and reuse objects on various platforms – and we are currently working with organisations to identify those resources.
And with that Sarah hands over for questions, and Im switching sessions as the sound levels in Hall 3 are making it hard to hear this session – especially audience comments.
I have snuck into the Showcasing Digital Research Tools demo session as there are a number of interesting speakers lined up here. At the moment John Casey is talking about the Clipper project. As I’ve recently blogged a workshop on this project I recommend reading that post for the background.
John Casey is now doing a live demo – you can explore the demo site at: http://reachwill.co.uk/clipper2.1/clipper-editor/.
The Clipper tool uses APIs from major video platforms such as YouTube. I can search for a video, select it, and make it part of my project. I can choose to select a clip from that video – based on time markers. And I can title and annotate that clip. And because you access the player from the site these videos come from, you can use only videos you have appropriate access rights to. So, for instance, I’ve tried this with Jisc MediaHub and it works as playing a video in Clipper will direct you to login then view the content.
Giving researchers credit for their data – Neil Jeffries, Bodleian Libraries, Oxford University
This is a project aiming to encourage researchers to deposit their data in repositories, but also to get more value out of their data and other shared data. We have several partners in this work, including organisations focused on sharing methodologies rather than the research data itself, and those working with publishers.
The idea is that this tool is a “carrot” for data deposit. There is a “submit data” button in the repository – this means that repositories get more data deposits and better metadata. And the researcher gets an additional publication/citation possibility – and preservation of that data. Publishers working with this syetsm get more data paper submissions, etc. And we know that connecting that deposit to publishing can be a powerful motivator for researchers.
So, to make this happen we have various connectors built (or planned) to tools where data will be coming from. Within the repositories a deposit generates a page, QR code, links to data etc. And we have a “Data Paper Companion” space. When a researcher submits data we connect that to their Orchid ID, their data is viewable and explorable by journal, project, etc. For any data set supporting licenses, declaration of interests, metadata, etc. is shown on the page, along with a summary of data. As a user you can elect to download a sample or the full data. When you find a data paper (e.g. The Elton Archive) you can find the data associated with that, you can also find the information on publications etc.
As the publisher of that data you can also edit the record, add new associated data sets, etc. And, once everything is organised you can choose to submit your data paper to a journal such as F1000 Research. If you choose to do that your data and details are pulled through to their submission system, where you can make edits, add content, etc. but all of your data assets have been brought through for you making this quick and easy.
So, the idea is to encourage greater deposit of data, and the
We have various data sharing and publicatoin platforms… Mendeley, FigShare, DSpace repositories, etc. on board.
Q1) Is that Bodleian project live yet?
A1 – Neil) No, we aren’t scheduled to be done with phase 3 for another 6 months but we should have an update then. The idea that this is a route through to publishers though. We have made our source code available already, though we still have some work to do on connectors – Sword connectors will be build by the appropriate module owners though. And I know that Jisc is looking at a centrally provided service to enable this.
The Jisc project manager in the Pod also notes that there will be a showcase for this work, and you can follow #dataspring for further updates on all the projects.
Having had a chance to chat with the lovely folk at Guidebook (info, etc. on their website if you are curious) I’ve headed to a slightly different session, on open citation.
Verena: I’m here to introduce Drahomira who has been designing the open citation experiment, to test a new approach that evaluates the full text – the meaning of the citation. The idea is to overcome draw backs of conventional citation metrics, and takes advantage of the availability of full text.
This project was the first large scale analysis of this new type of metrics, based on over 1 million articles. Drahomira will say a bit more about the approach taken, and show a demonstrator website.
Drahomira: Thank you for the introduction. This experiment uses full text methods to understand research metrics – using Semantometrics.
So, what are Semantometrics? They are a new class of metrics for evaluating research. This is different from research metrics and altmetrics, both of which measure engagement. Whilst those counts have been widely used and adopted, despite criticism, but technology and the availability of full text make different metrics possible, that look at the full text rather than just the usage/engagement from outside sources.
So Semantometic contribution measures are based on the idea of measuring the progress of scholarly discussion. The hypothesis states that the added value of publication p can be estimated based on the semantic distance from the publications cited by p to the publications citing p. So this measure traces development and bridging of ideas and concepts (http://semantometrics.org/).
This work with Jisc was a comparative study with analysis carried out to investigate the properties of the contribution measure. The experiment were carries out on a dataset obtained by merging data from the Connecting Repositories (CORE), the Microsoft Academic Graph (MAG) and Mendeley. After merging the datasets there are 1.6 million publications (though 12 million starting data set).
So, I will now show you the demonstrator website – already online and public. We’ll also share our research around the project on the same site. What I’m going to show you is the visualisation made possible through semantometrics. So, we can, for instance, look at network diagrams showing nodes and networks across publications. And in this model the mode important paper is the one which bridges two different communities or areas of thought. We measure the distance of publications cited by a specific publication, and we look for the contribution value to a particular domain, and more broadly. We look at specifics of groups or clusters of publications, and the distribution between them.
So, papers in both sets may be dispersed… and that isn’t necessarily impactful. But a paper with a very narrow range of citations that opens ideas up to a much wider range of papers and communities may be very impactful.
I prepared some examples on some publications, with visualisations that put the paper at the core, then show linkages out to papers… And distance is semantic distance between ideas. Those visualisations show the links between papers, but also indicate the field itself – the diversity of the areas in which the publication sits.
I selected examples which generate interesting graphs… But there is more detail in the report, which is available on the website. Two of these graphs address contribution and citation count. These show a big difference… Very few papers have high citation counts but many papers have high contribution. We were looking at correlations between the datasets… We were interested in looking at readership – using Mendeley data. Citation count and readership have a high correlation – not surprising. On the average values we see that above a certain value of citations, publications receive always above average contribution scores. That confirms what we might imagine to be true – that impactful papers are cited more. But it also reflects that lower citation scores may represent smaller more specialist research communities.
Q1) Have you factored in negative citations – citing papers that are being critiqued or argued against?
A1) No, but that is citation sentiment and that is a research area that we know about and are interested in.
Q2) Do you factor in the age of a citation?
A2) No, not at the moment, but again something to consider.
Q3) An observation. I’m glad you’ve done this observation on an open data set, as Thomson Reuters impact scores for REF are hopeless, as they are closed and proprietary. Your work finally opens that up, and that’s great. There is some discussion on the REF, and the cost of running that. And discussion of whether there is a light touch REF – with more metrics and less human time. What impact could you see this work having in a lighter touch REF?
A3) One of our motivations here was to see how metrics could be use. A big advantage here for REF. Whilst there are issues – like negative citations etc. It can be hard to compare publications. But we need to better understand what exactly research metrics capture, whether metrics are stable – whether recently after publication is representative or not. We can develop new metrics that takes account of time. Lots of promise… But we really have to understand what the metrics tells you. On the openness I agree with you. What really helped us was that… Originally we missed the citation network so I have to say Microsoft really helped. But Mendeley is very managed by people, Microsoft is very noisy data.
Q3) At the moment we have to take publishers word for it..
A3) Sure, but we have to be aware of the downsides of public data sets.
Q4) I’m assuming this was a global corpus – how did you account for language as that can be so difficult to do with semantic processing and analysis?
A4) That would be really interesting. My colleague is an expert in this area and we hope to do more work on that.
Q5) What do you see as important next around the stability of the metrics?
A5) We are looking at the stability of the metrics at the moment. But we believe they should be more stable for citations, but contribution we think that that will change more over time. One of the other challenges here is how one handles uncited publications… The advantage of semantics is that the data is there from the moment of publication, so in terms of understanding contribution that’s immediately available. I think this can be used to distinguish key papers, and to understand distance between publications. We can place a value on each citation already.
Verena: We have arranged a workshop in March with domain experts, and a report will be coming out at the end of March. And we’ll tweet some of those links.
Jisc’s investment in digital content for humanities: understanding the impact on research outcomes – Paola Marchionni, head of digital resources for teaching, learning and research, Jisc; Peter Findlay, digital portfolio manager, Jisc; Professor Eric T Meyer, senior research fellow, Oxford Internet Institute; Dr Kathryn Eccles, research fellow, Oxford Internet Institute.
Paola: Welcome to this session. We will hear first from Eric Meyer at OxII at some work on Early English Books Online and House of Commons Parliamentary papers which really shows the impact of those resources. Then we’ll hear from my colleague Peter Findlay on the portfolio of services, and how things are changing going forward, and hwo we are looking for new ways of teaming up, and we’d love your feedback on that.
Eric Meyer: The project I’ll be talking about was funded by Jisc and ProQuest but I just wanted to start with a bit of background on our work. So in 2008 – but updated since – we created a toolset called TIDSR, supported by Jisc, to understand the impact that various digitised resources were having. Jisc had digitised a whole series of collections and wanted to understand the impact they were having. I won’t go into details but this includes quantitative and qualitative methods for assessing impact. This isn’t a downloadable thing, it’s instructions to do these things no matter how much experience you have (or don’t have) in doing this. You can see case studies there and explore those citing and using the toolsets.
So, back to the current study. The thing about measuring impact is that the size of the audience isn’t the only thing you want to look at, a small audience that is the right audience might be big impact. It is not easy to compare the impact of different resources for some of these reasons. In this work we looked at Early English Books Online and House of Commons Parliamentary papers, and we’ve looked at these collections before in different ways. These are quite different collections in lots of ways and we wanted to see what kind of impact they were both having on teaching, learning and research.
So, the first highlight may be obvious but is worth stating: the context of the use of digital resources is changing but these changes are incremental and have a long development cycle prior to the realisation of impact. But one of the interesting things about digital resources for humanities is that they seem quite familiar. So a scanned page is similar to physical page in many ways. It’s not like in the social sciences where an analysis, a network diagram, etc. might be something quite different. It’s also worth noting that EEBO enables studies that wouldn’t be available in the same ways before. So, if you were an undergraduate humanities student you might not be able to access special collections, to private materials at all – possibly not until well into a PhD – but now these digitised copies do give you that access.
We also saw that, whilst the serendipity of the previous space didn’t happen, new types of serendipity occur digitally. And having these resources available allow people to wander beyond their direct focus and explore related areas for new possibilities and directions to look in.
So, we also found that usage of both EEBO and HCPP has been increasing over the past decade. But HCPP has seen a less steep incline – and that is because it seems to have found it’s audience more quickly, whereas EEBO has only found it’s audience gradually. EEBO released full text relatively recently – that will be a factor – but has also been used more creatively, which we will come to later.
While researchers at top universities are most likely to use EEBO and HCPP, less research-intensive HE institutions also benefit from both collections. We knew that usage was high (particularly for EEBO) in research intensive organisations and we wanted to explore if that was just the “rich getting richer” or if those benefits were more widely spread. In fact the resources are used across HE and, to an extent, FE. One of the interesting aspects was that the ordering of (anonymised) institutions varied between the two services, this isn’t just about departments but also because of the type of usage.
One of our case studies is Open University and they are very high HCPP users but are not high EEBO users. And my colleague Katherine spoke to them and we found that HCPP materials were invested into other courses – around information literacy for instance – which made a significant difference to their use. We also saw usage from less expected subject areas of these collections, for instance literary heritage, conservation, preservation etc. courses also using materials in EEBO.
Researchers rely heavily on specific digital collections that they return to regularly, which is resulting in incremental changes in scholarly behaviour. Now Google was important but specific databases and collections was ranked much higher than any other way of finding resources. Users of HCPP and especially EEBO gave us lots of feedback on what the resource could and couldn’t do, and what they liked about them. Lots to learn from how you embed these tools in universities.
Resource use in the humanities is extremely diverse, and this makes providing access to needed resources and tools particularly challenging – we asked researchers to list some of these and there were so many resources there. The thing about EEBO is that it’s something that stakeholders in particular areas that have come to rely upon it. By contrast HCPP is an important secondary resource for many people who use it in a really different way.
The citation evidence that is available shows a growing literature that mentions using EEBO or HCPP, and these publications in turn are reasonably well-cited. Now we looked across lots of publications and citation data here, but these databases take a while to be updated. We see spikes of citations – outliers – but generally there has been an upwards direction of publications etc. But humanities publications have a long gestation period – it can be 8 years for history for example – but the number and growth look pretty good across both resources.
The number and range of disciplines that refer to EEB and HCPP is much more diverse than expected. We have visualisations here that help illustrate the spread. The ideas move beyond core humanities disciplines – for instance HCPP publications in medical and medical history areas for instance.
Researchers are more concerned with the content and functionality of the digital collections than in who provides access. That’s a challenge. The library is invisible for many students and researchers… They say they don’t use the library and then when you highlight subscription collections they aren’t aware these come from the library – they think it’s Google. So, that’s a problem as that isn’t transparent until users lose access, change organisation etc.
The UK is unusual for providing national-level access across institutions through Jisc’s national purchasing. Now we know that the UK punches above its weight in terms of academic impact. This obviously isn’t down just to this set up, but that national purchasing agreement and access to resources does contribute to the UK’s global prominence. And they have potential democratising effects – you may see some institutions, some FE institutions too using these resources less, but still using them. And there is opportunity to encourage greater use of resourcing in teaching.
Shifts to humanities data science and data-driven research are of growing interest to scholars, although there is still plenty of room for growth in this focus on digital humanities, particularly in teaching. For EEBO that usage increase really reflected that opening up of xml texts, the hack events and social media presences around that change which really encouraged use – projects such as Trading Consequences.
Digital collections have become fundamental to modern scholarship – for the summary and full report see: http://microsites.oii.ox.ac.uk/tidsr/case-study/2016-idc.
Please do take a look at the full report, give us your comments and questions. Do read the report and feedback.
Q1) We did a digitisation project of Newton’s notebooks and they were being used but the citations are citing the paper copies as if they’ve seen them physically – which they haven’t – rather than digitally, so how do you deal with that.
A1) That is really a big issue. There are scholars on both sides here… Some claim they wouldn’t cite the library they used for a book… And MLA’s advice to cite “online” not URLs isn’t helping any. We did a previous report of splashes and ripples suggested human readable, easy URI’s as mattering. But this idea of there being something dirty about digital is still there… There is less bias maybe but the confusion remains. Some resources give helpful suggested citations with URIs, but not all by any means.
Q2) How do you compare the kind of data mining impacts and the other direct impacts of resources? I was involved with the Trading Consequences project and I know those data mining projects use a lot of data and look quite different, but how does that compare with more direct impact.
A2) Direct and qualitative projects aren’t really comparable. So it’s about individual resources demonstrating what they can do. We did some work on a very niche resource a few years ago, with very low usage, but for teachers this resource on how dancers built a portfolio was invaluable. So it’s being able to assemble a bunch of different kinds of imapcts a resource can have, and demonstrate to funders etc.
Comment) That matters when looking at subscriptions and the value of those.
A2) We have built this toolkit and over the years people almost without exception come back and say how fun it is to use the toolkit, to find out more about their users, to think about how they use these things, to reflect their usage and interest. So this is an opportunity to reflect. The other quote I remember from years ago from a humanties scholar was that “this is the first time I’ve studied anyone who could talk back to me” as she was used to working on dead people, so she found this really exciting.
Comment) The other aspect of EEBO was, when we got the service, it saves time and money. This researcher was absolutely thrilled by it.
A2) The speed and volume of these things was the initial easy sell of these things, then we’ve tried to layer additional things beyond that.
Q3) We are looking at impact of our resources, are you still working on this?
A3) We have done lots of work before, hope to do more. One of the reasons I hired Kathryn back in 2007 was that she was a proper academic historian but she was new to this online world and her impact has been absolutely vital to this sort of work.
Q4) How about aggregated search points… Sometimes when staff and students search for resources they often get multiple materials, they find other ways in… How do you tae account of this.
A4) This is the stuff we tend to get from interviews. In a previous study we found that people were often arriving relatively deep in their website – coming through discovery tools – so we did work with them to help ensure users understood what to do next, to make their site more sticky by changing the page set up so you were signposted to the next issue, the context, the stuff to carry on exploring. We often think of people arriving at a website front door, but often they find a much less expected way in.
Q5) I work for a publisher like ProQuest and today someone asked me about the Return On Investment of our services… Is that something you have looked at?
A5) We’ve tended to shy away from that as you’d have to make so many assumptions. Maybe if we had an economist on board… We have looked at some to see how income related to impact but that’s the nearest to that idea.
Paola: The nearest thing that we have seen is to try to represent how much it would cost to visit physical resources, travel etc.. But of course if that was the requirement they might not access them at all.
A5) We also have services where two resources from across the world are compared side by side – that’s not something you can do any other way.
Q6) I wanted to ask a question about creative digital reads, by artistic rather than academic communities – particularly thinking of CC licensed and public domain resources. I work with the BL finding out how people use public domain collections in innovative ways. People sometimes thing that having the freedom to do things makes them concerned they might be doing something wrong… But we are really interested in creative use.
A6) You could compare images to see usage.
Q6) Peter Balnam(?) has been doing something like that.
A6) We do ask people in our surveys whether they have reused or repurposed resources… But there is lots of scope for that though – hence that EEBO hack event etc.
Q6) At British Libary Labs we expected lots of academic engagement and we have unexpectedly had a big response from artists and creative professionals.
A6) And i think that’s a thing you can think about… Hack events, Wikipedia editathons, etc. can also show real impact on the greater world.
Peter Findlay: Showing the impact of digitisation Jisc has funded over the years has always been a big challenge… When we had proposals in for this sort of work we did’t know what would happen… So this is all really exciting. We are now in a position where we can see this kind of impact but with the current changing public sector funding, the ability to do this has become a real challenge. The overarching question for us is about the future business models for digital resources.
The focus of institutions is also shifting. Even when value is demonstrated it can be hard to get that message across to decision makers in those institutions. And I’d like to explore with you how important it is to have access to these kinds of collections. These resources are part of people’s work every day… To make this happen again we have to work more closely together, in terms of what we do and in terms of how we fund it.
We’ve also been thinking about what kind of models we might contemplate. We’ve been thinking of a sort of Kick Starter for digital content – with Jisc as negotiator for collections. So, less about digitisation, more about collectively purchasing and finding mechanisms to select and identify content together so that they can be purchased. Not just a purchasing consortium, we are also interested in tools for analysis of content. So Jisc Historic Text is a platform for instance where we’d like to put more content.
A slight adjustment for that would be Jisc seeking core funding to kick that off. We could go to charities, foundations, etc. Essentially we are talking about us together purchasing content or, if you have it, distributing content. We have also been thinking of Jisc as publishers – for institutions together as a collective to enable reduction of costs, a bit like the open platform for the humanities ideas. AGain, this would focus on platform, development, and ongoing support through, say, some form of subscription (for want of a better word). We’d also need to think about cost recovery for any platfrom that is set up, so that it can be sustained
Our third model is Jisc becoming more a supporting institution for the development of tools around the analysis of content, lab activities, mechanisms for small initiatives that can be followed up afterwards.
We’ve been having some great discussions, I’m just nothing the feedback to the rest of the room.
Group 1: If digital collections were not available, nothing comparable would be available – they enable courses and work not otherwise available. For the BL where impact is hard to demonstrate in general, can be easier for some specific resources though. Impact of individual services is possible, and works – as per Eric and Katherine’s work. Humanities researchers often aren’t aware that resources cost money, they don’t think about costs often. Archives do get switched off and disappear. Legacy resources sometimes get ported to institutions who when they can no longer resources – opportunity there. There are resources available, and they can be marketed to students, but they aren’t always what is wanted. Cambridge commented that the impact stimulates funding. Preservation can be a motivation for sustainability – so others preserving content takes burden off institution. Crowd funding good but may mean small voices and needs may get crowded out. Concern from institutions about content on others’ platforms. Idea that institutions could support platforms… They digitise then share centrally would be one model – making things more effective for institutions, easier to promote, and brings platforms and content together, enabling publishers to put content on platforms too.
Group 2: We thought about current models… For my institution – we just had one or two of us from libraries. In a library, for us, buying a one-off is better than an ongoing subscription in hard economic times. That way you can keep it, rather than adding to yearly subscription burden. Pitching at the end of the financial year might be best, as that is when budget may be available. Over 90% of budgets year on year is committed to journals, databases, ebooks, we have very limited funds for new stuff. And we are keen for more materials to be online, for teaching and learning as well as research. We were quite keen on Kickstarter model… Mixed opinions on whether you need finance directors on board to make that work – although library directors have some autonomy. So, if you had a Kickstarter type website were libraries could show interest in new resources, but also offer a way to suggest ideas, capture gaps in provision etc. Also thought about ad hoc models… Pay per view being one option. Also talked about car leasing – lease then option to buy… Trying to borrow ideas from different sectors.
Group 3: Not a huge amount of library experience on our table either. Talked a bit about how we use wishlists (collections development request list) for new things to buy. So many new things appear and we always need to prioritise wishes. Jisc Colletions is crucial to a lot of what we do – the NESTE2 agreement for purchasing for example. We are also part of other consortiums for purchasing as well. We thought one way to think about material for digitisation might be to share wish lists, in an anonymised way to help deal with competitive drivers that can make collaboration more tricky. Also larger scale digitisation projects as a possibility here. Going back to wish lists those could also come out of a collective gap analysis, rather than looking at products already on the market. And demand is key to securing funding for any kind of digisation project, and we need to think of sustainable business models, and the ability for institutions to articulate what is important to us.
Peter: That was very interesting. Thank you very much for those insights, and we will continue to have those conversations with you. Thanks to all of our speakers and to ProQuest for co-funding this work.
The case for learning analytics – Phil Richards, chief innovation officer, Jisc; Michael Webb, director of technology and analytics, Jisc; Niall Sclater, learning analytics consultant
Phil: I’m chief innovation officer for Jisc and we are here to talk about the case for Learning Analytics… To start with I want to talk about what we mean by learning analytics. Google and Facebook track our actions, our interactions, etc,, using that to attract advertisers etc. as they are hugely revealing. Learning analytics is a bit like that, it’s about patterns and understanding data from our interactions with VLEs, library accesses, etc.
Michael: We really are talking about using the kind of big data that Phil was describing. We are looking, in our project, at areas such as improving retention but we also want to move towards adaptive learning.
Predictive learning analytics are statistical analysis of historical and current data derived from the learning process to create models that allow for predictions that can be used to improve learning outcomes. Models are developed by mining large datasets and seeking patterns of behaviour. That’s quite different from the “have they logged into the VLE in a week” type approach.
So, just a moment on retention. We have a huge number of students dropping out at various stages of their education and that recruitment and loss of students is expensive and problematic. 70% of students reporting a parent with HE qualifications achieved an upper degree against 64% if students reporting no parent with HE qualifications for instance. But we can address some of those issues.
I wanted to talk about some US examples that have inspired us. Marist College in the US, funded by the Bill & Melinda Gates Foundation, undertook work supported by Bill and Melinda Gates Foundation. They investigated how use of academic early alert systems impact on final course grades and content mastery. The outcome was interesting… Analysis showed positive impact on final course grades. But they also found that any intervention that alerted students they were at risk was effective.
The New York Institute of Technology also found a successful way to predict students at risk. And there has been the building of a Predictive Analysis model built for US universities which seems to have a high level of identification of at risk students. At Purdue University the signals project, based on performance, background, etc. was effective but that work has been critiqued. University of Maryland found that students being able to compare VLE activity with that of other students were 1.92 times more likely to be awarded grade C or higher compared to students who did not use it.
So, there is clear evidence of the usefulness of predictive models and their potential benefits for student outcome. And Jisc have a learning analytics project which has three core strands: Learning analytics architecture and service; Toolkit; Community – we have a mailing list and blog you can follow. The toolkit includes two main activities: the code of practice and the discovery phase. The Code of Practice was one of our most successful early pieces of work for this project. Failed learning analytics projects tended not to be down to technical issues but for ethical issues. And that code of practice has had great reception, including from the NUS.
We have done some research, which was in the Times Higher, that shows that students are happy to share data for learning analytics. We have a good reason for doing this, a clear code of practice and that means students will buy in.
So, what are we doing? Well we are building a national architecture, defining standards and models and implementing a core service. Why do that? Standards mean that models, visualisations and so on can be shared; lower cost per institutions through shared infrastructure; and this lowers the barrier to innovation, as there is consistency.
Our architecture is built around dashboards; but also alert and intervention system – we’ve defined an API that ensures interventions are captured and feed into the data store; we have student consent app – for how their data is used, preferences etc; and a student app. Then, at the centre we have a learning records warehouse – cloud based system built on open standards – and the learning analytics processor that sits on top of that. The kinds of data we are collecting includes self-declared student data; student information system; VLE; Library; and other things that may be useful (“???”).
To do this work we are partnering with the commercial sector, and our partners are Unicon (on open source stuff), Marist, Blackboard, Tribal, TherapyBox, HT2 (cloud solution provider). And that partnership has led to some really useful things already, including Blackboard making data more available.
So, the service includes dashboards – visual tools to allow lecturers, module leaders, senior staff and support staff to view. This includes student engagement, cohort comparisons, etc. Based on other commercial tools from Tribal and Marist. The student app is bespoke development by Therapy Box, and this interface is very much based around fitness apps. The first version will include overall engagement, comparisons to others including other high achieving students, self-declared data – including student-defined goals, consent management. We are inspired by gaming too – you get trophies for cool stuff!
The Service alert and intervention system, based on open source tools from Unicon/Marist (Student Success Plan) allows management of interactions around alerts.
The data collection falls into two types… Relatively static student record data, and the big ever changing activity data. We’ve taken slightly different approaches to those two data sets. So we have information on the student (ETL) based on HESA and FLR(?) in FE space and consistent with HEDIIP, and you can see our work on there on GitHub. For the activity data we are collecting via TinCan (xAPI) which lets you get quite detailed data. We’ve commissioned a Blackboard module, have supported a Moodle plugin etc.
Now the idea of an xAPI “recipe” is a shared way of describing activities. So the data from accessing a course is the same whether Moodle or Blackboard is used. So, same holds true for “borrows a book” etc.
We have had 72 expressions of interest from the sector. We have 26 organisations, across a diversity of organisation types are engaged in the activity. We have over 1 million records collected in real-time. We needed historic data for this project so we’ve also working on historical data collation from Moodle and Blackboard to enable those predictive models that require data to work on.
Across different stakeholders there are different priorities. For Russell group universities it may be about widening participation and support for students achieving 2.1 or better. For a teaching lead organisation it may be about focusing on interventions in teaching and learning, to improve retention.
Phil: Every year universities have to make around 7000 different measures reporting to HEDIIP. And this project can help aggregate that data, and to give back analytics to the individual institutions based on the architecture we have come up with. And this is the first project to create something like this which provides access to all the information needed for a HEDIIP return. One of the concerns about HEDIIP future reporting is that it may become more frequent… Currently that’s annual. If automated these returns could be quarterly or more regularly. Now learning analytics is a great reason to upload data more regularly for HESA and other agencies, and to benefit from learning analytics as part of that.
The way we’ve set this project up is very similar to the way UCAS has used Amazon Web Services. Until a few years back their website spiked dramatically on A-Level results day and the cloud scaling makes that possible without issues on their server.
Part of making this work is about keeping data private and carefully managed. They want to benchmark and compare. The way we have addressed this is by having users declare that they are happy to share their data if aggregated and anonymised into pools of, say, 10. But we need to have data in place to do that. We need to build up number of contributors.
Now you can look at this for interventions for individual students, or explore by cohort or background etc. Maybe there is potential for new metrics like this to feed into the new proposed TEF.
Some interesting potential in the medium term. Just to talk more about unified data definitions… Our basic standard for that more general data is the HESA model. And we’ve done some work with HESA on the national HE business intelligence service – a fully live production service that has been available from Autumn 2015.
The government is scrutinising subscription organisations like Jisc, like HESA, ever more so, and there are some real opportunities here. We took part in a HEFCE learning gain call in May 2015, which was around standardised tests, etc. and we have work to do there at the moment.
A quick move to genomics…
In Iceland everyone knows their ancestry and the Iceland government has gathered all the genomic data into deCODE and Iceland’s genetic data bank. This system uses reference data, undertakes analytics number crunching and outcomes include understanding the pathways and outcomes.
So, just to borrow that model… Maybe our learning analytics warehouse can be our DNA bank for higher e-learning. The background data would include demographics, early learning and employment outcomes. The analytics and number crunching, towards deeper understanding of elearning, metrics for engagement learning gain, personalised next generation e-learning.
In a recent report with pro Vice Chancellors said that HE was getting more global, more digital, more competitive. But none claimed the UK was taking a lead here. In universities we still use tools we have been using for decades, but the rest of social sciences have moved leaps and bounds ahead… Why not do this with our data?
So, Micheal talked earlier about personalised learning. So, right now we do capture data on how we learn, how your brain works, etc. And maybe sharing that earlier enables a truly personalised next generation elearning that helps you determine the pathways you need to take – for instance a student with low social capital wanting to study architecture might see what the different paths might be… That could really improve social mobility and close some gaps.
In the short term we’ve seen that interventions for not dropping out… seem to really help at risk students who are particularly likely to be widening participation students, which could really help bridge some of those gaps. Maybe this is the kind of work that can put the UK out there as leaders in this field.
I hope that’s given you a good case for why we are doing this work now. Where it might lead in 2 years, and where it might lead in 5 years.
Q1) Why has Jisc decided to do learning analytics from ground up, rather than work with an existing provider. And I was disappointed not to see UK examples in that mix – we have good examples, some better than US examples shown there.
A1 – Micheal) We aren’t building from ground up, we are combining existing tools and components. We are putting architecture together to make things work better.
A1 – Phil) We have put together an open architecture, but we have worked with providers… Those were selected through a public sector procurement process (as we are duty bound to do, at least until the referendum) and these are the companies that came out. And some companies have approached us wanting to take part, and we will open up the project to more commercial companies later this year. We want to help you avoid vendor lock in but to work with as many providers as possible. Why are we doing that? It’s what over 1000 people we spoke to in the scoping process ranked most highly.
A1 – Michael) Why US examples – just wanted to use some different examples and I’ve shown the UK ones before – you’ll find them on the website.
Q2) I work at a learning analytics start ups, really great to hear what Jisc are doing, and great to hear about that focus on widening participation. I’m really interested in what the big barriers are: is it cultural, ethical, technical?
A2 – Micheal) It’s a mix of all those things. Some can be solved relatively easily – getting data in and out. Student records systems still tricky but will get easier. Senior staff buy in really matters, a key part of our discovery phase was getting buy in and understanding their system. The pattern is that there is no pattern…
Q3) A follow up question… You spoke about Russell Group universities and the possibility of a positive effect on widening participation, can you say more about that?
A3) We ran a scoping process and one of the use cases presented by this type of organisation was specifically about widening participation and also narrowing that gap between those achieving 2.2 versus 2.1.
Q4) You mentioned models elsewhere being mappable to here… library data and VLE data. What about other types of explicit engagement like citations etc.
A4 – Micheal) Yes, want to do that. But actually assessment data is an important early start there.
A4 – Phil) Some commercial companies aren’t interested in shared or common metrics but we saw evidence in the States that it can work, and enable benchmarking. We think that has real value and that that doesn’t preclude commercial vendors from also providing more granular and bespoke solutions.
And, with that, day one at Jisc is done. I’ll be tweeting to #digifest16 for the remainder of the evening for the dinner etc. I will be back on the blog again tomorrow.