Jul 092014
 

Today I am at the Student Social Media Showcase (#SSMS2014) and the Mixed Methodologies Seminar, both precursors to the European Conference on Social Media (#ECSM2014) which I will be at until Friday. I’ll try to liveblog most of the conference days but today I’ll be posting notes as this is a loosely structured day. The Showcase, being Storified here, brings together both students and academic delegates of the conference and, for the student social media showcase, over 100 local school children as well as local businesses and apprenticeship schemes operating in Sussex. Both the conference and today’s event’s have been organised by the Brighton Business School, at University of Brighton.

This morning, while the kids have been experimenting in the creativity suite, I have met the organiser of ECSM2015 (which will be in Portugal), and we have been hearing about the DV8 Sussex Apprenticeship scheme which has been placing students, aged 16 to 23, in businesses from very small cafes to big social media agencies, on specific digital media and social media apprenticeships. They spend four days a week at their employer, and one day a week at college taking a number of social media, digital media, and marketing modules. It sounds like a really interesting scheme and the two students we met this morning seemed like great representatives of the scheme – they will be running hands on experiments in running mini campaigns for the students.

Introductions

Asher, one of the main organisers, is talking about social media and how central it is in business and marketing, and the business school’s recognition of the centrality of social media in our day to day lives. Today the focus is on what social media means for us, for the kids in the audience, and for jobs. And Asher is also talking about some work on “what is it students get out of studying?”, we think that the most important thing is learning how to learn… if we give you a seminar on Snapchat, it will be out of date in 6 months time, so the important thing to learn is how to research this stuff, how to learn about it, and how to think about what social media can do in business, in media, in the arts.  And as you look at the displays around the building you will see work by students that demonstrates that.

Sue: When we knew we would be hosting this event we went out looking for partners from the local community. We knew that the research conference would bring in people from across the world, but we also wanted to pull in local graduates and near graduates, but also local employers, and schools. We want to see how this all works, and we plan to do it again and again every year. We should have lots of spontaneous conversations… talk to anyone, see what they do, what they use… And there will be stuff every hour in this theatre – and we have five students you can talk to right away…

Tom English: I will be talking about Snapchat and ASOS, and how Asos could use Snapchat to sell their clothes

Cecilia: I’ll talk about Zara and how they use Facebook, Twitter and Pinterest to communicate with customers

Abiola Oduwasi: I’ll be talking about how people prepare to present themselves for the jobs market – graduates and recruiters

Sean Fitzsimons: promoting your writing and journalism through social media

Alice Britton: I did a project on how Bagelman, a local business, used social media for their business

Showcase

Running throughout the venue today are screens showing digital media presentations from students. Some nice case studies that I’ve already seen included a presentation on beauty bloggers and brands’ use of sponsored posts – where the blogger receives direct or indirect benefit from the brand for writing about them. Some examples were shown and some research suggesting that consumers find reviews useful no matter whether or not they have been paid for was quoted – an interesting finding in the blurry authenticity space that is social media. More on that in Lu, Chang and Chang 2014.

Brighton Fuse Project: Why Social Media needs all your skills – Dr Jonathan Sapsed

It will be good to talk to you today about this project, the Brighton Fuse Project, a research project looking at new media, digital media, and creative industries in Brighton. There is real clustering of these industries in Brighton – you see it in Shoreditch, in Bristol and Bath to some extent, Salford, etc. There is no one big company in Brighton drawing people in – unlike the BBC in Salford – so we wanted to see what was drawing them to Brighton, what attracts them. And we also saw that these companies need people like you (the teens in the audience), and all your skills.

This was a £1.5 million project with University of Brighton, University of Sussex, Natioanl Centre for Universities and Skills, BBC Academy, etc. involved. And Ed Vaizey welcomed this report and it’s findings on the Brighton CDIT. We’ve had a lot of interest because we looked at how the creative industries and skills really intersect with business. And we’ve also seen a huge investment made in Brighton to encourage these industries, to improve infrastructure and the quality office space for these high growth creative businesses. These sorts of things can be exposed through this kind of research, and you can then talk about how to address this.

So what is “fusion”? Well the combination of creative design skills and digital technology skills, the mix of artists, programmers, and business skills. One of our participants from Plug-in Media talked about how important the relationship between creativity and tech is. And we’ve known that idea, that concept of fused content, is important for a long tie for converging platforms – games, tv, mobile, online, etc.  But we didn’t know the extent to which this fusion was needed in sectors like social media. So lots of these digital media companies who have been running since the 1990s are increasingly adding design skills, social media skills, it’s about working out what the company desires, what they will want next, how a campaign can engage people more, to sell more. So you need those sensibilities of the analytical, segments, and patterns of search but also the creative skills and sensibilities for this space.

We looked at entrepreneurs… those who did their first degree in Arts and Humanities or Design are about 48% of the entrepreneurs. That was a bit of a surprise. And those with more degrees, with PhDs, their businesses often were doing even better. And whilst STEM and Computing folks were also doing well, it was equally as well as those from Arts and Humanities backgrounds.

But we also found that some firms are more fused than others. Some – about a third – are specialist so only really employ developers, or only really employ designers. About a third have some mix, and then we have the “super fused” who are dependent on having a tightly integrated mix of these skills. In terms of what types of companies are represented here… the Digital Agencies are more likely to be super fused, as are design services. And the least fused were arts organisations – but that’s probably a good thing, they need to be specialists in my opinion. On the whole fused businesses correlated positively with innovation and turnover growth. The super fused firms grow three times faster than unfused companies. That mix is very important.

So, looking at business models, the firm iCrossing, probably the second biggest digital agency in terms of employees in Brighton, do lots of work as “creative technologists” for various big firms, including Rolls Royce. Now they have a small customer base, they are happy with sales levels, but they want their brand to be more popular…  [brief break as kids leave] So Rolls Royce is an example of a company not looking at sales as a measure. But they had 14 measures of engagement in social media – really playing into the geeky side of what they do, the craftsmanship is shared via YouTube videos and shares of those… so it’s about good creative skills, how to make that interesting and enticing engagement, that is needed. So those 14 measures also get used for triggering payments to iCrossing. Each time they meet a target there, they get paid. So iCrossing employs programmers, journalists, copywriters, graphic designers, tim makers. They are looking for “Creative Technologies” job roles.

And an iCrossing campaign – which I can show now the kids are gone – was for Ann Summers and around paid search (YouTube: Ann Summers: Sexy Paid Search). So this was about using high interest news related web searches that hijack that news story by triggering related ads – for the budget, the BA Strike in particular – and got a good reception and impact for clients – click throughs, media coverage, a huge boost in profile etc. So for that client they have that client on a retainer – giving space for creative ideas, something thought of on the fly. That’s a particularly useful space for experimentation, for lateral thinking, for trying stuff out that is clever rather than high tech, trendy stuff perhaps. Counter intuitive stuff.

We found high levels of innovation in the cluster… and we used the types of innovation used in the European Innovation Survey… usually they find 60-65% innovation but for this cluster in Brighton  99% innovation. And more innovation in super fused companies. And 37% of firms allow time for personal projects – and that allows space for unexpected products and services for the firms.

Fusion is linked to innovation but… it’s not new to the world technology, traditional R&D, or protected by patents. Instead it’s service-oriented, continuously attending to user-experience and design. The value is hard to capture, in spire of £231m revenues across the 500 companies we looked at.

In terms of location… these organisations work for some local firms 40% ish of the companies do local, often business to business work for each other. A good 56% work for clients in london. And about a quarter work for international clients. And these firms are relatively young… the average respondent is 41.7 years old, two thirds of respondents are in their 30s and 7.8% in their 20s. And there are real cross overs of backgrounds… some have STEM backgrounds (22.89%) but many are from Arts and Humanities, Design, Business Management or Economics… but some have, say, stage management degrees… and they bring that creative background to bear on their work.

And the people working in these companies… only 8.4% always lived in Brighton. Many moved to Brighton for the lifestyle (e.g. one of the most successful web company CEO’s cited Britain’s only Vegetarian Shoe Shop as a reason he moved to Brighton!), many for personal reasons. Rarely do they move to find a job, for professional reasons… we think that is starting to change… there’s a kind of second wave here… many of these companies started in the 90s and they need people like you guys to be part of that next wave… And Ian Elwick, Founder-Manager of Brighton Media Centre and The Werks cite the support, the peer communities, these physical co-working spaces, those types of aspects as being important to these communities [we are now watching video – findable on the AHRC website along with the report – on these types of spaces, how they foster knowledge sharing and “being a good corporate citizen in the modern world”].

There are a lot of different styles of network events… there are cheese and wine events… but those are not so much about help, collaboration, contracting in a business sense… and those engaging in those benefit in material terms… So, a good example. Black Rock Studio, a big developer which was acquired by Disney. They did so well for 10 years they were brought by Disney… something happened… probably a failure of marketing for two big games… closed in 2011… made all of their 279 staff redundant… but a whole group of “black pebbles”, companies started by former employees, set up… and they create apps, small games, smaller scale stuff… some work for hire… some brought out by big Shoreditch company… they meet up, they help each other out, they use social networks online and offline, supportive culture there that is so important to clusters. Though fusion tends to be weak at community level, strong at a business and project level.

But it’s not all perfect news… some risks and barriers facing these companies. Fused firms face skills barriers, they find it hard to find the right skilled candidates. Easy in Brighton to recruit good design hirees, but paid search, product managers, etc. are not skills easily found. Sometimes they have to hire more technical roles through London. That limits growth. They find it hard to find the right people with the right skills… and larger firms perceive artistic community as a barrier… perhaps too laid back, too bohemian according to some. The recession and skills barriers were the main issues facing these firms at the time of the report.

But a key conclusion for us is that arts and humanities is key to interdisciplinary interaction and innovation and economic growth… but the HE system can be suite set again interdisciplinarity, often fields of study are quite separate and that’s not a good fit for creating these fused individuals. And this is a really organic cluster in Brighton, it’s hard to create that sort of effect artificially… policy makers often want to support a wide geographic range of locations but we think they should fund succeeding clusters more, to stimulate growth there…. to let that growth be organic…

Q&A

Q: You didn’t mention Brighton SEO… are you aware of any other conferences or similar happening that cement Brighton as a digital hub…

A: There are lots of those but tend to be very segmented and just known to that sector. In September Reasons to be Creative… and another which Warren Ellis is involved in, Deconstruct,… lots of these things… Twitter is the place to look for these things… a lot more smaller meet ups, in pubs, etc. and a great way to meet and make connections and find jobs, etc. That stuff leads to pub chat… I know one guy, now a senior manager for Electronic Arts in Montreal, who got the leads that led to that job through a pub chat…

Q: If you were designing a module or similar what would you include to address gaps… stuff to support such clusters in future…

A: We’ve talked a lot about this… but a lot of the message that comes from businesses themselves is that comfort with technical and creative sides is essential. And knowing how to manage a project, to be organised, to show leadership, also key. And we’ve thought about ways to best deliver that… practitioners say that graduates aren’t industry ready… and you ask them to help and to get involved in course design… and they are too busy to help… But the bureaucracy of developing courses, and the existence of disciplinary silos, can be the enemy of those sorts of skills…

Asher: if you are a graduate and you have experience of creative writing but never done SEO… or vice versa… what are the first steps to being part of this fused economy?

A: A lot of these skills are very much self-taught… a lot of people learn in that way. A lot of people hire someone they know with those skills and pay them for a morning to teach them on an ad hoc basis – as courses often exist that help with that. And they learn through others…

Information Visualization for Knowledge Discovery: Big Insights from Big Data – Ben Shneicerman, Professor of Computer Science at University of Maryland

One of the fun things here I think is the breadth of types of people involved in these spaces, as we heard before in Jonathan’s talk. Steve Jobs used to talk about his work being at the intersection of technology and the liberal arts. I am based at the Human Computer Interaction Laboratory, an interdisciplinary research community of Computer Science, Information Studies, but also Psychology, Sociology, Education, Journalism, and the wonderful Maryland Instute of Technologies for Humanities. Now many of you may know me from the book Designing the User Interface. Now the stuff you will be talking about at this conference was a real driver for the most recent update, in 2010, to that text. More than 5bn people have mobile phones now and they are changing the world, the way that we interact around health, around community. We have mobile, desktop, web, cloud. We have diverse users, diverse applications… so many opportunities to explore the world around us…

Now today I am going to talk about “Big Data”. In 2012 a release from Obama, announcing a Big Data initiative and talking about visualisation, talks about developing scalable algorithms for processing imperfect data in distributed data stores, and creating effective human-computer interaction tools. So we need to be teaching the key skills of visual reasoning, which we don’t usually teach… In 1999 we published a collection of papers on information visualisation. That area has now massively grown so no longer possible to capture in a book – the web gathers that whole world of papers that is emerging. But we do get some new directions… Jim Thomas and Kristin Cook wrote about the concept of Visual Analytics, Illuminating the Path, in 2004 (online for free). And in Europe Daniel Kein wrote on visual analytics (also available for free).

Now… one of our graduates set up an information visualisation company called Spotfire, growing a business out of their research work. For instance a visualisation showing Retinol’s role in embryos in vision – a rare example of a single image acting as an important research finding. That’s a rare occasion… but that tool became well known for genomic, biomedical, oil and gas discovery, etc. So…. increasingly visual tools are being used… we see a move to large display walls (10M to 100M pixels) helping productivity… Bloomsburg uses arrays of 8 screens with very fixed windows having huge value… we see radiology workstations with multiple displays to see a brain scan… some with 16 displays showing last weeks as well as this week’s scans… these sorts of workspaces are becoming common – multiple people sharing, collaborating, around multiple screens.

We are also seeing small screens (1M pixels and less) having a real impact… mobile screens with data such as Google’s expansive transportation interfaces through their maps, and historical data on that… There is a huge amount of data, our job as designers is to organise that, to understand data needed to make decisions…

So, the information visualisation mantra (and I once wrote this a dozen times in a paper – now cited over 27k times!):

  • Overview – the full range of items
  • Zoom and Filter – let the user do that, find what they want…
  • Details-on-demand – let the user drill into the data

The most compelling part here is the centrality of the human user. It’s not just about the algorithm…

And if we think about the last 50 years of Scientific visualisation in 1D Linear (Document Lens; SeeSoft, Info Mural), 2D Map (GIS, ArcView, PageMaker; Medica Imagery) and 3D World (CAD, Medical, Molecules, Architecture) forms… and they have a great future. And we now have the new area of Information visualisation… often about muti-variable data (Spotfire, Tableau, Qliktech, Visual Insight), Temporal (LifeLines, TimeSearcher, Palantir, DataMontage); Tree (Cone/Cam/Hyperbolic/SpaceTree/Treemap); Network (Pajek, UCINext, NodeXL, Gephi, Tom Sawyer). Loads of blogs here that are worth a read: Flowing Data; Perceptual Ledge; Etc.

So, let me go to the first demo… traditionally we often look at temporal data… for instance Stock Market Data. So… overview first… so looking at a year… February has a lot of uncertainty. Now you (an audience member) mentioned a “spike”… is that a spike upwards? Or downwards? We have the wrong language for visual reasoning yet! Now we can zoom into this data… look through this data…. seek patterns… Information visualisation allows you to see new patterns, new changes, to ask new questions. So with this [demo] visualisation you can create a pattern and look for that in your data set… but people were interested in how one might do the opposite – make a pattern and explore by inverses of that pattern… that’s thought patterns you can’t explore on paper and you can do it rapidly, and readjust them on a screen… You can try out and test hypotheses easily with these tools – and you can try this out, look for “TimeSearcher”. TimeSearcher was designed to do time series for stocks, wealth, genes, and to work with large data sets and allow the user to really shape interactions.

Now another tool we built was LifeLines, an attempt to create a visualisation for Patient Histories – with the overview acting as routes into that medical history, to understand changes, medications, interactions… And one of the nice things I like is that visualisations can also show you what isn’t there… harder to do algorithmically… but you can see gaps that might be concerns, questions, it’s a starting point…. we thought one patient was good, but a million patients would be better… so we worked with some data from the Pediatric Trauma Centre in Washington DC and using a tool we built called EventFlow (also free to download). The hospital (via video recordings then transcribed) record initial checks – airway, breath sounds, distol and central pulse in the first few minutes… and then you get longer for the secondary checks… Looking over a large set of data (216 patients) you can get a sense of how quickly secondary checks occurred… And you can spot anomalies in how staff conducted checks – not dangerous perhaps but not the hospitals protocol…. And you can see all the ways that these patients have been seen, how they vary… the most common variance was starting the disability check before secondary checks… there are some repetitions… some took ages to get their checks done.

So talking about Treemaps… that was our work… for instance SmartMoney Stock Data… looking at a terrible day you see a single blip of good activity – a real clear contrast… often you see patterns that are more subtle… but that visual training happens when data is spatially fixed, when you can spot change…

Treemap: Newsmap (work by Marcos Weskamp) looks at global news items and the number of online articles on a given topic… you can compare countries’ coverage directly… again, a free to use/explore visualisation.

And we did some work with the Hive Group on tree maps for Nutritional Analysis. SpotFire added tree maps in 2007, Tableau now has it. the New York times have used tree maps now. And a German researcher developed the idea of Voronoi tree maps – they look cool and organic it can be hard to read. There is a design aesthetic aspect here, these look cool but are hard to compare size of spaces.

Manual Lima has a great site called VisualComplexity.com with thousands of network visualisations…

And the work we did was in a tool called Node XL, it’s free to download and use, and it’s a network overview for discovery and exploration in Exel… designed to show interactions and connections between people… So for instance can be used to see voting in the US Senate… And you can use NodeXL to directly import from Facebook, Twitter, YouTube etc… feel free to create another importer tool… So one of our first experiments was for #WIN09 Conference back in 2009… and you could see from the 80 people in the room a kind of split between two groups of people – computer scientists and sociologists – and in the tweets you saw that clearly shown… just one cross over in a graduate student!

And that sort of connecting and cross over issue is even more compelling in political discourse… So we did this for the #GOP tweets… you could see a very cohesive densely connected group of republians. A less connected group of democrats. And a few cross over people… but they talk within their group but very little interaction between them. Cross over only via Politico. Media consumed between these groups otherwise really diverged…

But, this work kinda works…. but not a great way to visualise… using grapes for inspiration we tried to restructure around smaller clusters, separations, etc. in a more clear to view way…. for instance used in looking at #SOTU (State of the Union Address).

And… a researcher called Scott Dempwolf who looks at Innovation Networks… he took data on companies, patents, grants from government agencies… 26k edges, 11k nodes…. so he has created a beautiful visualisation for Pensylvania Innovation Networks… but hard to read…. so we tried to break this down a bit…. found a major pair of nodes who hold a lot of patents…. And you see real cluster of some of the big players in innovation…. Westinghouse Electric and the Navy being key drivers here…. So drilling down you see the big players…

We asked Scott to show us something on Maryland…. he created a visualisation for our lab…. again looking at connections and gaps… we can also look at innovation in Chicago to see how we see clusters here… You begin to see the finer grained structure more clearly when you have a visual way into the data…

Recently we published this on the Pew website – you can see Node XL Gallery for more of this sort of data – looked at Twitter network structures: polarised crowds; Tight crowds; Brand clusters; broadcast network; community clusters; and support networks… for those doing customer support via Twitter…

So, you can read more. You can find out about our Social Media Research Group. And we also want to talk about not only business but also other spheres in which these tools can help, for instance the UN Millennium Development Goals… Some progress towards their goals… Bill Gates is helping with next goals… The Gates Foundation is a big user of Node XL… in that presentation earlier we saw visualisations via Bar Charts but understanding interactions is key here.

Q&A

Q: I’m sure over the next few days we’ll see a lot of papers with statistical analysis… what would your advice be for business and finance academics to get papers more visual, and get published…

A: A good question. You do see Science and Nature moving to printed visualisations… they are static…we have a long way to go to make those interactive… by contrast the web and blogs are much more interactive and visual… and increasingly you see that supplemental stuff – video or interactive website – online. Science encourages you to have a website, data if possible, and visualisation tools with your papers. Actually  there is an annual competition around visualisation run by Science and partners…

Q: This is on errors and potential for misrepresentation… with many of these tools there is so much potential to accidentally misrepresent the data…

A: You are right of course… statistics can lie, data can lie, and visualisations can lie… you can use colour, labelling, etc. in misleading ways. But for any visualisation I think an intelligent understanding can reduce that impact. But the majority of datasets I get into my office have errors that the person whose data set it is didn’t know about it…. I was looking at emergency room admissions data recently… 8 patients in that data were 999 years old… those kinds of errors are widely found in data, or a patient admitted 14 times, but discharged only twice… And you have people using flawed data to predict sales but miss one month when their sale is on! Statistics without visualisations risk never spotting that error… visualisation provides a sort of microscope, telescope… new ways to explore and understand our data. And you need a new sort of literacy, that concept of visual reasoning. And the tools have made that possible…

Q: You talked about a lack of vocabulary… what should we be using?

A: We have a tool, not quite as polished as a shape finder, but the question is can you make a measure of the spikiness of each spike? In books you see standards about what is and is not a spike. During a discussion a student suggested something brilliant… using the angles within the spike to find sharp spikes, and also areas of fall and rise. So we have started to explore this sort of stuff… but of course volatility can be a measure… but there are interesting shapes that we ca use and explore here… you have concepts like “value line”, sizes of plateau. It’s a rich space we’ve only just started to explore in the shape finder.

Q: In terms of the methodology to create these models… I am interested in customer journeys between social media channels, capturing those touch points between platforms…

A: You have some systems, like Klout, that gives you numeric data… but we are interested in networks here…. IBM did a project with their internal networks of these things, of connections in discussion. My colleague did work with emails, to see cohesiveness of discussions… but we are only 5 or 6 or 7 years into this social media world… but it’s definitely an opportunity to do good… And again there is an effort from the National Cancer Institute to use social media to make health related opportunities, for smoking cessation, obesity reduction, etc…. to get changes through use of social media… And you see media networks evolve. Jenny Priess and I wrote a paper called “From Reader to Leader”… On Wikipedia only 1/10th of 1% ever make an effort… and only 11,000 admins…. so we need to understand the dynamics of that… how one goes through that path, what the motivations, rewards, recognition, to encourage people along that path… The sciences of the natural world have been successful for 400 years but I think the science of the made world, of social structures, etc. is the science of the next 100 years.

Q: You mentioned bar charts etc. in my presentation earlier. We have looked at new ways to present this data… info graphics etc… there are a lot for quantitative data but fewer for qualitative data…

A: Well one step back…. it’s not about visualising your data…. it’s about your goal, your question, what are you trying to answer… in your data there was clearly more there… a simple taste of what’s possible… the network structure of these community might be interesting…. so it might be a geographic relationship… but you need to know the questions first, and use that to decide what you need, what you will find in the data, how you make new opportunities happen.

 

Mixed Methodologies Seminar – Professor Dan Remenyi 

Dan Remenyi is introducing himself as an itinerant academic, who teaches research methods at various universities and also supervises PhD students.

When I completed my PhD, rather late in life, I felt the most interesting part was the research methodologies but I felt like I needed to learn more in that area, and had a lot to learn. I have supervised a lot of PhDs now and most actually use “mixed methods” but, a bit like “reflection”, you needed to do this stuff… you have to do that… these days you can’t just do it, you actually have to write about, to describe that stuff. If you use the phrase “mixed methods” about your research – and I’m going to counsel you not to necessarily do that – you have to be able to say why you did that, what that means, what the implications are…

So today we will talk about what Mixed Methods really is, and how you talk about it… You should all have had the slides in advance… I took those slides and put them into Wordle… you can see I’ll be talking about Data, about Mixed Methods, and about Synthesis… Now… as I progress down this road of talking about research methodology I’ve learned that it is so important to understand the vocabulary of the research world, how to use them appropriately…. Some are easy perhaps but some are much more tricky. You should know these… I suggest you create your own glossary where you really pin down your own understanding of these words… You need to know what they mean, you need to be able to defend your work.

Now, lets talk Mixed Methods… Well this is an expression, some call it a misnomenclature – it really doesn’t explain what it does (a bit like Life Insurance, of Jumbo Shrimp, some often refer to “military intelligence” as the same type of misnomer!). Why? Well there is almost no way that methods can be mixed. What we mean is using both qualitative and quantitative data to make a convincing argument… In the previous talk the speaker talked about charts, visualisations, and that the research question is absolutely key. And that’s the case in methods… but think slightly wider than that… in actual fact when we do research the research enables us to understand better the research question, and come up with possible answers for it…

So what is usually meant by Mixed methods is that combination of qualitative and quantitative data in research. In your research you need to be contributing to the academy, both in terms of the findings and the theoretical aspects of the field. And you have to convincingly make your case. There is still a lot of confusion about Mixed Methods. Researchers sometimes lose sight f the fact that evidence, of whatever sort, is a constituent of the argument which underpins the findings. The challenging part is bringing these different dimensions of the argument into a convincing whole.

At it’s heart Mixed Methods is a research design issue. You can adjust that plan as you go along, academia is essentially about self-improvement… your plan will always emerge and involve as you go along. A research design might start with what data you require to answer the question, then think about how you will collect it. How will you analyse it? How will you use it to establish some findings? And increasingly you are expected to interpret those findings, to talk about what the implications of your research is.

So the term Mixed Methods is being used in two senses…

  • – There is an emerging school of thought, or community of practice, that argue for the use of mixed methods research design.
  • – There is the research practice which has been in place for decades which have called upon researchers to use different methods at different times, stages, phases in their research. Indeed it is hard to use an entirely quantitative approach in research.

Now, not all researchers welcome the concept of Mixed Methods… some think you have to be world class and that you cannot be world class quantitively or qualitatively…. the aspiration is to be world class but I think you can be extremely competent at both. But the philosophical argument is trickier… the ontological argument is that you can either be a realist – positivist, quantitative type road – or a relativist and that that takes you down the more constructivist, analytical route. In reality we are often a combination of both in reality…

Now the key person in this area, he has made it his own, is Creswell. He says you cannot tell your story unless you can put together the numbers behind your research and to tell the stories behind those numbers. He says that numbers never speak for themselves… you have to be able to see the numbers and the facts in context. Paulos (1998) talks about statistics as being uninterpretable without context, background, their origins then they cannot be properly understood…

An example here… stats on home runs in the US Baseball league show increasing numbers of home runs… what’s happening? More matches? More training? More reporting of games? Changes in recording measures? More rewards for better players? Stand out players like Babe Ruth? But a more important reason… they banned cheating! Generally Baseball was played in the afternoons… and the light got dimmer… flood lights weren’t great… pitchers started messing with the ball, spitting on it, rubbing it in the dirt… and the batter could see the ball…. How will you know that just looking at numbers? You won’t, you need some other form of research to understand that data. (For more on these stats Dan recommends Bill Bryson’s book A Short History of Nearly Everything – a great book for PhD students to read as, essentially, a history of science. And his book One Summer: 1927 include those statistics… in that book the most important thing is Charles Lindberg flying the Atlantic….)

Now, there is another phrase you need to be aware of and that is “Multiple Methods”… If you are using multiple methods in the qualitative arena then some say you are using Multiple Methods, that Mixed Methods is exclusively for the combination of Qualitative and Quantitative Methods. You also hear Combined Methods, Hybrid Methods, and (from an audience member) Multi-Level Methods.

A few really important distinctions… At the highest level research can either be Theoretical – this is based on secondary data, data that has been previously been published, and already-established ideas and you create something new from those existing ideas. Empirical Research is about the collecting of data. Now data is a hugely contested term, there is a surprising lack of papers on data… when I questioned what data was in a statistics department they thought I was mad but data is a really tricky term, I’ll come back to this.

Now, in theoretical research is highly linked to empirical research, but always relating that back to theory, and using existing empirical data.

And then we have the two major paradigms of Positivist which is about the qualitative world, numbers (mostly), the process is deductive so there are hypotheses that you are attempting to reject (you try to reject it, if you don’t you accept it pro tem), it’s interpretation with a “little i”. And we have Interpretivist approaches… an inductive process, uses a wide range of data… and it’s about taking that data and from it attempting to form a hypothesis from that. Now the vast majority of research is deductive, a faster process. An inductive approach can take longer and require much more data… Now… Mixed Methods sits between these, straddling both positivist and interprevist perspectives. And following a side chat on mathematical methods, mathematics fits not quite anywhere into these research paradigms… The concept of Ocham’s Razor is useful here: the explanation that the idea that is simplest is best… In general we can never say we have proved something… the only thing that is certain is that we know what we don’t know… But we can say “the evidence suggest”, or “it appears from the evidence”… that can be said… much harder to say that “the evidence shows this is true”.

Now… a comment on Qualitative and Quantitative research and how they differ…

In Quant: You articulate the research question, you collect evidence, you process evidence (questionnaire) – only after you have collected data, and you produce findings…

In Qual a learning loop is involved: you articulate the research question, you collect evidence (interview), you understand the question as you process the evidence and you really have a loop, you learn as you go, and you do produce your findings.

There are alternative approaches too… Action research often takes an iterative approach for instance.

Of course Mixed Methods can be used in theoretical work… you might collect data to support a theoretical perspective. And Mixed Methods are particularly useful in interdisciplinary work. And it can also be useful in applied research, where there are blurred boundaries between topics…

So we have 12 steps in research design:

Setting the course

  • 1. Field of study exploration and conceptualisation
  • 2. Literature review
  • 3. Research question
  • 4. Research design

Moving the project forward

  • 5. Data acquisition …………………… when is triangulation relevant?
  • 6. Data management
  • 7. Data analysis
  • 8. Presentation of findings

Completion Issue

  • 9. Theory development
  • 10. Research question resolution
  • 11. Implications for practice
  • 12. Limitation and future research

Each step informs the next step, although the research process is not a water fall based project

Remember that to do competent academic research we not only have to understanding our data and analysis of that but we also have to understand all of the arguments in the body of knowledge, and we have to be able to articulate that. And that has to feed into the research design.

There are different ways to approach Mixed Methods research…. One way is to start with qualitative data as a way to reach understanding, and to design a quantitative instrument (e.g. a questionnaire) that is then deployed and leads to findings… It’s a big deal to create a questionnaire from scratch! And in this approach each step is distinct. You take two steps… one step followed by another… the mixing is very minimal…

But there is no reason not to take a different approach… You use an established research instrument to gather data, then you conclude that stage, and you take a qualitative approach next, in order to reach your findings. That’s a perfectly respectable Mixed Methods approach.

Now you can also take what they call a “supportive mixed methods” design… here you have overlap between types of research, you can benefit from understanding the data of one type in your work collecting data of another type. Now I like metaphor… so take the buttress (flying and not)…. someone pointed out to me that the way that Cathedrals are built is fundamentally unstable… will push the walls out… and that’s why buttresses, and flying buttresses came about. And I like to think of scientific discovery as not always standing on it’s own without data from a variety of different sources. Multiple sources of validation are always welcome… they act like buttresses… (and now we have a side chat in which Dan makes  the point that doctoral students should not touch longitudinal studies… “that’s a different methodological world”).

You should know that academic research gives you a great deal of flexibility in what you do. It is based on peer review – your papers will be seen by at least two people reviewing it – but there is a lot of flexibility as to how you do it. Paul Feyerbiant wrote a famous book, a difficult book, called “Against Method”. And in that book he says the only universally accepted academic research methods, and that is “anything goes”! It doesn’t mean you can be sloppy… it means no one can tell you how you must do your research, or what you cannot do… you can do it your way as long as you can convincingly argue your case, and show that you are contributing to the academic body. As long as you can argue that your methods got you to the right answer, you have to be able to argue your methods, to justify them… I had someone come up for examination who had done 35 interviewers… a particularly tough examiner who said he needed more… but how many do you need? Well you need as many as need before you reach the point of data saturation… you have to be able to justify the number that is acceptable. As it happened this guy went out and found a whole load of papers showing that 35 could be a valid number… this is part of why you have to understood the literature… you have to have read everything that can be read about your topic… And the other thing about academic research is that you have a lot of flexibility but you have to use the language consistently, and to understand the meaning of those words… we had a chat before about what it means to be longitudinal… it means an extended period of time… is that 3 months? 3 years? 3 weeks? For anthropologists they conduct ethnography, they talk about a lived experience… how many of us in the business or management world truly have a live experience… Ethnography is, as a word, taking liberties there… but we can talk about being “ethnographically informed”, by the same token we could talk about “a longitudinal type study”. Teet was talking about interviews over a few months as being not a snapshot… but argued appropriately you could use some of that language of longitudinal language… Because, as we’ve said, we have to be clear of making a clear and justifiable case for your choice of methods… We have so many methods but you have to be clever about how you put your argument together…

So… back to a third model for Mixed Methods… this is a parallel or converging Mixed Model… Where you undertake quantitative and qualitative research in parallel… now I have gone light on talking about “triangulation” here… some people love that term, some hate it… to be precise the word is borrowed from land surveyors who use various tools to map particular features, measuring from different angles… social scientists have borrowed that term to talk about different perspectives… now when I did my research 25 years ago I was told triangulation was a way to resolve conflicts and contradictions in the data… that is nonsense… by being able to look at things through different perspectives, different lens, different data, different people… you get a richer understanding of the question, of the issues involved. Now some say the term “triangulation” is too positivist, that something like corroboration is better…. I don’t really mind… more perspectives is usually better. BUT…. it is tempting to believe that the more panoramic the view, the better… and that may often be the case, but is not always true….  Sometimes putting all this extremely rich view into a cohesive whole can be really problematic… Research does not seek complexity for it’s own sake… If you have a credible answer to the research question from one or two data sources then the job is probably done… Answering the research question is the paramount issue.

So in this third approach, the parallel or converging mixed method design… we will get two sets of data, from two different sources, and bring them together into an argument… and we will draw on both sets of data to draw our conclusions… There’s no other sense in which we want to mix it… Now in the literature you will see some discussion of putting numbers into words and vice versa but I am not convinced by that. Some critical issues… were the two different data collection strategies driven by the same research question? If not, then why to? Was the same research logic used for both – i.e. inductive or deductive? And are the results commensurable? They don’t have to be but you will have to argue your case well, you have to change your argument and explain any contradictory results. And again, you have to answer the research question.

Now, reflection is central to research. It has always been necessary. But it’s now really important to be able to discuss it… Reflection may be defined as a process of questioning the range of activities and thinking which have been performed by the researcher in order to surface any inadequacies or bias which may be present in research. And why you have come to the conclusions you have come to.

Reflexivity – and the piece in MIS Quarterly is worth reading – is about seeing the interrelationships between the sets of assumptions, biases and perspectives that underpin the different facets of the research undertaken. So you might ask yourself what assumptions are at play when you start your research? All research starts with assumptions that there will be an answer to the question, that that question is worth answering, and that the process of answering that research question will change you, will develop you to a higher level in the case of a doctorate for instance. Reflexivity is about understanding that, of understanding biases… nobody likes to feel that they are biased… but you can’t get away from the facts what you are… so I’m a white, British, elderly, academic… all of those mean expectations and values… I might work against those but there are always some residues there… You also want to ask yourself what values of yours affect your research? So all of us have the shared values that knowledge is important for instance, we want to learn more. As someone in academia you also have to believe there is some value in sharing, that’s part of being an academic… you could explore all of that much further of course… but that’s what we mean by reflexivity.

Some mixed methods researchers talk about integrating the qualitative and the quantitative data so that an overarching analysis can be performed… so about how and when you mix the data… now I argue that we are really talking about synthesising the arguments. And the test of an argument is whether it convinces… There are various types of evidence which include data, authority and logical inference… So in academia argument is used to support theoretical conjectures. The way we learn is influenced by the Greeks… Socrates, regarded as close to a tramp, walking around picking arguments, who developed the idea of the dialectic… and that is how academia works… you articulate a thesis… you float an idea, then someone does the “ah, but…”, they correct the idea or take the antithesis… and then you put those together, you synthesise them, and create a new idea… and that re-articulation of thesis starts a new cycle… that’s an ancient concept that still underpins academia.

Now, Teet earlier mentioned a model like an Advanced Mixed Methods Design, something which may result in a case study, experiment or action research project. But what actually determines the method? This can be influenced by your background… an engineer may not want to work in qualitative research, a humanist may not want to undertake complex equations… So it may be about the scale of the work required, the skills that you have and, in the case of doctoral students it may also be about the influence of the supervisor or culture of the institution.

And with that, we are done.

 

 July 9, 2014  Posted by at 11:15 am Events Attended, LiveBlogs Tagged with: , , ,  No Responses »
May 142014
 

Today I am at the University of Edinburgh Digital Humanities and Social SciencesDigital Scholarship Day of Ideas 2014 which is taking place at the Edinburgh Centre for Carbon Innovation, High Street Yards, Edinburgh. This year’s event takes, as it’s specialist focus, “data”. These notes have been taken live so my usual disclaimers apply and comments, questions and corrections are, as ever, very much welcomed.

Introduction: Prof Dorothy Miell, Head of College of Humanities and Social Science

I’m really pleased to welcome everybody here today. This is our third Digital Scholarship Day of Ideas and they are an opportunity to bring in interesting outside speakers, but also for all of us interested in this area to come together, to network and build relationships, and to take work forward. Again today we have a mixture of international and local speakers, and this year we are keeping us all in one room so we can all hear from those speakers. I am really glad to see such a popular take up for the day, and mixing from across the college and Information Services.

Digital HSS, which organised this event, is work that Sian Bayne leads and there are a series of events throughout the year in that strand, as well as these events.

Today we are going to be talking about the idea of data, particularly what data means for scholars in the humanities, how can we understand the term Big Data that we hear in the Social Sciences, and how can we use these concepts in our own work.

Sian Bayne, Associate Dean (digital scholars) is introducing our first speaker. Annette describes herself as an “itinerant researcher”. Annette’s work focuses on internet and qualitative research methods, and the ethical aspects of internet research. I think she has a real talent for great paper titles. One of my favourites is “Undermining Data” – which today’s talk is partially based on – but I also loved that she had a paper entitled “Fieldwork in Social Media: What would Manonovsky do?”. Anyway, I am delighted to welcome Professor Annette Markham.

Can we get beyond ‘data’? Questioning the dominance of a core term in scientific inquiry – Prof Annette MarkhamDepartment of Informatics, Umeå University, Sweden; Department of Aesthetics & Communication, Aarhus University, Denmark; School of Communication, Loyola University, Chicago (session chair: Dr Sian Bayne)

As Sian mentioned I have spent a lot of time… I was a professor for ten years before I quit in 2007 and pushed myself across other disciplines, to push forward some philosophical work on methods. For the last 5 years or so I’ve been thinking about innovative and creative ways to think of methods to resonate better with the complex and complexity of modern life. I work with STS – Science and Technology – scholars in Denmark, Informatics scholars, Machine learning Scolars in Boston, Language scholars in Helsinki… So a real range across the disciplines.

The work today is around methods work I’ve done with colleagues over the last few years, much is captured in a special issue of First Monday: Vol 18, No 10: Making Data – Big Data and Beyond Special Issue. And this I’m doing from a post humanist, STS, non positivist sort of perspective, thinking about the way in which data can be used to to indicate that we share an understanding when actually, we are understanding the same information in very different ways. For some data can be an easy term, consistent with your world view… a word that you understand in your own method of inquiry. Data and data sets might be familiar parts of your work. We all come from somewhere, we all do research… what I say may not be new, or may be totally new… it may resonate… or not at all… but I want this to be a provocation, to make you question and think about data and our methods.

So, why me, well mainly I guess because I know about methods… so this entire talk is part of a bigger project where I look at method, at forms of inquiry… but looking at method directly isn’t quite right, but looking at it from the side, from the corner of your eye… And to look at method is to look at the conditions in which we undertake inquiry in the 21st century. For many of us inquiry is shaped by funding, and funding priviledges that which produces evidence, which can be archived. For many qualitative researchers this is unthinkable… a coffee stain on field notes might have meaning for you as an ethnographer but how can that have meaning for anyone else? How can that be archivable or sharable or minebale.

And I think we also have to think about what it is that we do when we do inquiry, when we do research… to get rid of some of the baggage of inquiry – like collecting data, analysing and then writing up as there are many forms of inquiry that don’t fit that linear approach. Another way to think of this is to think of frames, of how we frame our research. As an American Scholar trained in the Chicago School of Sociology is that I cannot help but cite Erving Goffman. They both tell us to focus on something, and to ignore other things… So if I show you a picture of a frame here…. If I say Mona Lisa you might think of that painting. If I tell you to look outside of the frame you might envision the wall, or the gallery, or what sits outside that frame. And if you change the frame it changes what you see, what you focus on… so if I show you a frame diagram of a sphere and say that is a frame, a frame for research what do you see? (some comment they see the globe, they see 3D techniques, they see movement). The frame tells us to think about certain phenomenon…. to also not think about others… if I say Mona Lisa now… we think of very different things… Similarly an atomic structure type image works as a very different type of frame – no inside or outside but all interconnected node… But it’s almost impossible to easily frame, again, Mona Lisa…

So, another frame – a not-quite-closed drawn circle – and this is to say that frames don’t tell you a lot about what they do… and Goffman and others say that frames work best when they are almost invisible…. like maps (except say the McArthur Corrective Map). So, by repositioning a map, or by standing in an elevator the wrong way and talking to people – as Harold Garfield had his students do – we have a frame that helps us look differently at what we do. “Data” can make us think we look at the same map, when we are not… Data may not be understood as a shortcut term of a metanym, it could be taken rather as preexisting aspects of the phenomenon – have been filtered and created through a process, and organised in some way. Not the meaning I want for my work but not good or bad…

So I want to come back to “How are our research sensibilities being framed?”. In order to understand inquiry we have to understand three other things. (1) How do we frame culture and experience in the 21st Century; (2) How do we frame objects and processes of inquiry; (3) How do we frame “what counts” as proper and legitimate inquiry?

For me (1), as someone focused on internet studies, I think about how our research context has shifted, and how has our global society shifted, since the internet. It’s networked for instance. But also interesting to note how this frame has shifted considerably since the early days of the internet… So taking an image from the Atlas of CyberSpace – an image suggesting the internet as a tunnel. But city scapes were also common ways to understand the world. MIT suggested different ways to understand a computer interface. This is about what happened, the interests in the early days of the internet in the 90s. That playfulness and radical ideas change as commerce becomes a standard part of the internet. Skipping forward to Facebook for instance… interfaces are easy to understand, friendly, almost all social media looks the same, almost all websites look the same… and Google is a real model for this as their interface has always been so clean…

But I think the significant issue here about socio-technical research and understanding has been shaped by these internet interfaces we encounter on a daily basis.

For me frame (2) hasn’t changed that much… two slides…. this to me represents any phenomenon or study – a whole series of different networks of nodes connected to the centre. There is no obvious starting point. Not clear what belongs in the centre – a person, an event, a device – and there are all these entanglements charecterising these relationships. And yet our methods were designed for and work best in the traditional anthropological fieldwork conditions… And the process is still very linear in how we understand it – albeit with iterative cycles – but it’s still presented that way. And that matters as it priviledges the neat and tidy inquiry over the messy inquiry, the inquiry without clear conclusions… so how we frame inquiry hasn’t changed much in terms of inquiry methods.

Finally, and briefly, (3) my provocation is: I think we’ve gone backwards… you can go back to the 60s or earlier and look at feminist scholars and their total reunderstanding of scientific method, and situated research. But as budgets tighten, as research is funded under more conservative conditions this stuff that isn’t well understood isn’t as popular… so we’ve seen a return to evidence based methods, to clear conclusions, to scientific process. Particularly in media coverage of research. It’s still a dominent theme…

So… What is data?

I don’t want to be glib here. The word “data” is awefully easy to toss around. It is. In every day life this term is a metanym for lots of stuff, highly specific but unspecified stuff. It is arguably quite a powerfully rhetorical term. As Daniel Rosenburg says the use of the term data has really shifted over the last few hundred years. It appeared in the 1760s or so. Many of those associated with the word only had it appear in translations posthumously. It is derived from Latin and, in the 1760s, it was about conditions that exist before arguement. Then as something that exists before analysis. And in that context data has no theoretical baggage. It cannot be questions. It always exists… has an incontrovertible it-ness. A “fact” can be proven false. But false data is still “data”. Over time and usage “data” has come to represent the entirity of what the researcher seeks and needs in pursuit of the goal of inquiry. To consider the word in my non-positivist stance, I see data as “what is data within the more general idea of inquiry”. In the mid 1980s I was taught not to use that word, we collect materials, we collect artefacts as ethnographers… and we construct… data… see even I used it there, so hard not to. It has been operationalised as discreet and uncontrovertible.

Big data has brought critical responses out, they are timely and subtle responses… and boyd and Crawford (2011) came up with six provocations for big data. And Nancy Baym (2013) also talks about all social media metrics being a nonrepresentative partial sample. And that there is an inherant ambiguity that arises from decontextualising a moment of clicking from a stream of activity and turning it into a stand alone data point. Bruno LaTour talked about this too, in talking about soil from the Amazon, of removing something form it’s context.

And this idea disturbs me, particularly when understanding social life as representated in technology. Even outside the western world, even if we don’t use technology, as Sonia Livingstone notes, we are all implicated in technology in our everyday life. So, I want to show you a very common metaphor for everyday life in the 21st century – a Samsung Galaxy SII ad. I love this ad – it’s low hanging fruit for rhetorical critique! It flattens everything – your hopes and dreams offered at equal value to services or products you might buy… and flatterns as equal in not infitesimal bits that swirl around, can be transmitted, transformed, controlled – as long as we purchase that particular phone. An interesting depiction of life as data – and humans and their data as new. It’s not unusual and not a problem as we don’t buy into it as a notion, uncritically.

This ad troubles me more. This is Global Pulse, an NGO, a sub committee of UN, that distributes data on prices in the developing world. It follows the story of a woman affected by price shifts. So this ad… it has a lot of persuasive power and I want to be careful about this arguement that I make to conclude…

I really like what we get from many big data analyses. I have nothing against big data or computational analysis. Some of the work you hear about today is extroadinary, powerful… I won’t make an arguement about data, about data to solve certain problems. I want to talk about what Kate Crawford talks about as “big data fundamentalism”. I wouldn’t go that far… but algorithms can be powerful but not all human experience can be reduced to data points. And not everything can be framed by big data. Data can be hugely valuable but it’s important to trouble what is included and what is missed by big data. That advert implies data can be understood as it happens. Data is always filtered, transformed, framed… from that you draw conclusions. Data operates within the larger framework for inquiry. We have to remember that we have strong and robust models for inquiry that do not focus on data as the core of inquiry. Data might be important – it should be the chorus not the main player on the stage. The focus of non-positivist research is upon collecting the messy stuff….

And I wanted to show a visualisation, created in Gephi, by one of my colleagues who looked at Arab Spring coverage in media and social media in Sweden… In doing this as he shifts the algorithm he is manipulating data, changing how the data appears to us, changing variables to make his case… most of the algorithms of Gephi create neat round visualisations. Alex Galloway critiques this by saying that some forms may not be representable, and this tool does not accommodate that, or encourages us to think that all networks can be visualised in that way. These visualisations and network analyses are about algorithms… So I sort of want to leave it there, to say that data functions very powerfully as a term… and that from a methodoly perspective it creates a very particular frame that warrants concern, particularly when the dominant context tells us that data is the way to do inquiry.

Q&A

Q: I enjoyed that but I find you more pessimistic than I would be. That last visualization shows how different understandings of that network as possible. It’s easy to create a strawman like this but I’ve been reading papers where videos are included in papers… the audience can all think about different interpretations. We can click on a data point, to see that interview, to see that complex account of that point. There are many more opportunities to create richer entanglements of data… we should emphasize those, emphasize that complexity rather than hide the complexity of how that data is created.

A: Thanks for finishing my talk for me! If we consider the generative aspects of inquiry then we can use the tools to be transparent about the playfulness of interrogation, by offering multiple interpretations… I talk about a process of Borrow / Play / Move / Interrogate / Generate. So I was a bit pessimistic – that Global Pulse ad always depresses me. But I agree!

Q: I was taken by your argument that human experience cannot be reduced to a single data point… what else can it be reduced to… it implies an alternative to data… so what might that be?

A: I think that question is not one that I would ask. To me that is not the most important question. For me it’s about how we might make social change – how might I create interventions, how might I represent someone’s story. I’m not saying that there is an alternative… but that discussion of data in general puts us in that sort of terrain… and what is more interesting or important is to consider why we do research in the first place, why do we want to look for a particular phenomenon… to not let data overwhelm any other arguments.

Q: I think your talk noted that big data focuses on how people are similar and what similarities there are, whilst ethnography tend to be about difference. That makes those data tracking that cover most people particularly depressing. Is that the distinction though?

A: I think I would see it as simplification versus complexity… how do we envision inquiry in ways that try to explode the phenomenon into even a more complex set of entanglements and connections. It may be about differences but doesn’t have to be… its about what emerges from a more generative process… it’s an interesting reading though, I wouldn’t disagree.

Q: I wanted to share a story with you of finishing my PhD, a study of social workers when I was a social worker. I had an interview for a research post at the Scottish Government and one of the panel asked me “and how did you analyze your data” and I had never thought of my interviews and discussions as data… and since then I’ve been in academia in 20 years but actually I’ve had to put that idea, that people are not data, aside to progress my career – holding onto the concept but learning to talk the talk…

A: I can relate to that. You hear that a lot, struggling to find the vocabulary to make your work credible and understandable to other people. With my students I help them see that the vocabulary of science is there, and has been dominant… and to help them use other terms to replace the terms they use in the inquiry, in their method… these terms of mine (Borrow / play / move / interrogate / generate) to get them thinking another way, to make them look at their work in a different way from that dominant method. These become a way that people can talk about the same thing but with less weighty vocabulary, or terms that do not carry that baggage. So that’s one way I try to do that…

Crowd-sourced data coding for the social sciences: Massive non-expert coding of political texts – Prof Ken BenoitProfessor of Quantitative Social Research Methods, London School of Economics and Political Science (session chair: Prof John McInnes)

Professor John McInnes is introducing our next speaker, Professor Ken Benoit. Ken not only talks about big data but has the computational skills to work with it.

I will be showing you something very practical…. I had an idea that I’d do something live… so it could be an Epic Fail!

So I took the UKIP European Election Manifesto… converted to plain text in my text editor. Made every sentence one line… put into spreadsheet… Then I’m using CrowdFlower with some text questions… So I’ll leave that to run…

So back to my talk… the goal is to measure unobservable quantities… we want to understand ideology – the “left-right” policy positions… we have theories of how people vote, that they vote to parties most proximate to their own positions. For political scientists this is a huge issue. We might also want to measure corruption, cultural values, power… but today I’m going to focus on those policy positions.

A lot of political science data is “created” by experts… a lot of it is, frankly, made up. A lot of it is about hand-coded text units – you take a text, you unitise it…. e.g. immigration policy statements… (Comparative Manifesto Project, Policy Agenda Project). Another way is Solicited Expert Opinion (Benoit and Laver, Chapel Hill, etc) – I worked with Laver for years looking at understanding of policies of each party. It’s expensive work, takes an expert an hour to fill out a form… real headache… We have expert-completed checklists (Polity, Comparative Parliamentary Democracy Dataset, Freedom House, etc.). And there are Coded International events (KEDS, Penn State Event Data). And we have inductively scaled quantities (factor analysis such as “Billy Joe Jimbon Factoral analysis).

So what are some of the problems of coding using “experts”. Who are experts anyway? Difficult to find coders who are suitably qualified. It’s hard to find them AND hard to train them… most of the experts coding texts tend to be PhD students who find it a pleasing thing to do whilst avoiding finishing their thesis. There can be knowledge effects since no text is ever anonymous to an expert coder with country knowledge. Human coders are unreliable – their codings of the same text unit will vary wildly. And even single coding is relatively costly and time-consuming. So only one coder codes each text. Even when you pay the experts, they are still doing you a favour!

So I will talk about an alternative solution to this problem, and that problem is about classifying text units. So the idea is to observe a political party’s policy position by content analysis of it’s texts. And party manifestos are most common texts. The idea behind content analysis is breaking text into small units and then using human judgement to apply pre-defined codes. e.g. coding something as right wing policy. And usually that is done for LOTS of sentences by only ONE coder.

Tomorrow I’ll be in Berlin… the biggest (only?) game in town is the Comparative Manifesto Project (CMP). This is a huge project with 3500 party manifestos from 55 countries from 1945-2010 though still going. Human coders are trained and have PhDs. They break manifestos into sentences, human judgement to apply pre-defined codes. Each sentence assigned to one of 56 policy categories. Category percentages of the total text are used to measure policy. And each manifesto is seen by just one coder, and coded by just one coder.

So… what could we do… crowd-sourcing involves outsourcing a task by distributing it to an unspecific group, usually in parts… based idea of this, versus expert coding is that it reduces the expertise of each of the coders, but increase the number of coders. Distribute texts for coding partially and randomly. Increase the number of coders per sentence. Treat different coders as exchangable – and anonimous, and we don’t care if sitting in internet cafe in Estonia in their underwear, or whether they engage on a day off from a bank…

The coding scheme here is to have a more simplified coding scheme. We applied it to 18 of the “big 3″ British party manifestos from 1987 to 2010. So a sentence can be coded as Economic, Social or neither… under either of the first two categories there are further options (anti, neutral or pro) from “Very left” to “Very right”, or “Very liberal” to “Very conservative”. And there is a 10 question test to show correct codings, to guide the coder and to keep them on track.

So, to get this started we wanted a comparison we understood. We wanted to compare crowd coding to expert coding. So my colleague and I, and some graduate students, coded a total of 123,000 sentences between us… With between 4 and 6 coders per manifesto and using the same system to be deployed to the crowd. This was  a benchmark for the crowd sourcing end of things. This took ages to do… we did that…. that’s a lot of expert coding… and in practice you wouldn’t get this happening… For the crowdsourced codings we got almost twice as many codings…

We used an IRT type scaling model to estimate position. We didn’t want to just take averages here… we used a multi nomial method here. We treat each sentence as an item, to which the manifesto is responding, and the left or rightness (etc) as a quality they exhibit. Despite that complexity we found that a mean of means approach led to very similar results. We are trying to simplify that multi nomial method… but now the results…

Comparing expert codings to expert surveys on economic and social positions look pretty good.. good correlation for economic particularly a thing that we’d expect – and we see.

We tested to see how best to serve up results… we tried the sentences in order and out of order. Found .98 correlation so order doesn’t matter…

For the crowd sourcing we used Crowdflower, a front end to many crowd-sourcing platforms, not just Mechanical Turk. Uses a quality monitoring system so that you have to maintain an 80% “trust” score to be rejected. Trust maintained through “gold questions” carefully selected and generated by experts…

So, we can go back to the live experiement… it’s 96% complete!

So, looking at results in two dimensions… if Liberal Democrats were actually Liberal would be right of economics and left of social… but actually they are more left on economics. Conservatives on the right socially but getting nearer the left in some cases… but it’s not about the analysis so much as the comparison with the benchmark…

When we look at expert codings versus crowd coders… well the points are all over the place but we see correlations of 0.96 for economic, 0.92 for social dimensions. So in both cases there isn’t total agreement – we have either have a small crowd of experts or a bigger crowd of non experts. Its always an average but just a matter of scale…

So, how many coders do we need? No need for 20 codes for a sentence if it’s clearly not about immigration policy… we did massively over sample, then drew sub sets there for standard error… we saw that estimates from our errors the uncertainty starts to collapse… The rate of collapse for experts is substantially steeper… for aggregate of these two processes you need five times more non-expert coders than experts. But you can run good codings with five coders…

So we did some tests for immigration policy… used 2010 British manifestos, knowing that there were two expert surveys on this dimension (but no CMP measures). Only coded immigration or not, and if immigration is positive or not. Cost about $300. Ran again, same cost, extremely similar results…

Doing this we had 0.96 correlation with Benoit 2010 expert survey. .94 correlation with Chapel Hill Survey. And between the two runs correlation of around 0.94. Would have been higher… the experts differed between the immigration policies of Labour and Conservative… were not obvious positions in the text… but they had positions that experts knew about…

So, who are these people? Who are these crowd coders? They are from all over the world… the top countries were USA, Britain, India and Estonia. One person coded over 10,000 sentences! Crazy person loves coding! The mean trust score rarely drops below 0.8 as you’ll be booted off if it does… You don’t pay or get data from those that fail. Where are these jobs being sourced? We tried Mechanical Turk… we’ve used Crowd Flower… there are huge numbers of these sites – a student looked at about 40 of these sites… but trust scores are great no matter how these people are sourced… Techniques are not all ideal… but they don’t stay in the system if trust score changes. No relationship between coder quality and platform…

Conclusions here. Non experts produce valid results, just need a few more of them. Experts have variance, have noise, so experts are just another version of a crowd with higher expertise (lower variance). Repeat experiments prove that the method is reliable (and replicable). Some places require your work to be replicatable… is data plus script a good way to do that? Here you really can… You can replicate everything here. You can redo in February what you did in December… with the right text you can reproduce the result. Why does this appeal? Well it’s cheap, it’s flexible. Great for PhD students who lack expert access. And you can work independently from big organisations that have their own agenda for a study. You can try an idea, run again, tweak, see what works… Can go back again… And this works for any data production job that is easily distributed into simple tasks… sign up for Mechanical Turk, be a worker, see what it’s like to actually do this… for instance for transcriptions of audio tapes… it’s noisy…. a common job is that they upload 5 second clips and you transcribe that… gives you pretty good human transcription that timestamps weaves back together. Better than computer method…

So, we are 100% finished with our UKIP crowdsourcing experiment… Interestingly 40 negative, 48 positive… needs further analysis…

Q&A

Q: In terms of checking coders do the right thing – do you check them at the beginning or do you check during the process of codings?

A: Here I cheated a bit… used 126 gold questions from another experiment. You have to give a reason for each question about why it’s there – if the person doesn’t get it right then they get text to explain why that is the case… Very clear unambiguous questions here. But when you deploy a job you can monitor how participants responded or if they contested it… In a previous experiment we had so many contested responses that I actually looked again and removed it…

Q: A very interesting talk… I am a computer scientist and I am interested in whether now you have that huge gold data set you have thought about using machine learning.

A: Yes, we won’t let that go to waste. The crowd data too…

Q: I am impressed but have two questions… you look at every sentence of every manifesto… they are funny things as not every sentence is about the thing you are searching for – how do you deal with that? And a lot of what is in manifestos are sort of dog whistle things – with subtexts that the reader will pick up, how do you deal with that in crowdsourcing?

A: You get contextual sentences around the one you are coding, that helps indicate the relevance of that sentence, it’s context. In terms of the dog whistle question… people think that but manifestos are not designed to be subtle. They actually tend to be very plain, very clear. It’s rare for that subtlety to be present. Want truly outrageous immigration policy look at the BNP manifesto… every single area is about immigration, not subtle at all.

Q: I’m a linguist, I find it very interesting… and a question about tasks appropriate to crowdsourcing. Those that can be broken down into small tasks, and that your participants can relate to their daily life. I am doing work on musical interpretation… I need experts because I can’t see how to do that in language, in a way that is interpretable to non experts…

A: You can’t give something that’s complex… I couldn’t do your task… you can’t assume who your crowd is, we have very little information… we didn’t ask about language but they wouldn’t retain that trust score without some good English language skills. But workers have a trust score across projects so anything they can’t do they avoid as losing that score is too costly… You could simplify the task with some sort of task that can test corect or incorrect interpretation… but we keep the task simple.

Q: A very interesting talk, I have a quick question about how you set the right price for these tasks… how do you do that? People come from different areas and different contexts.

A: Good question. We paid 2 US cents per sentence. We tried at 5 cents and it was done very fast but quality wasn’t better. A job at 1 cent didn’t happen fast at all. So it’s about timings and pricing of other jobs.

Q: Could you say something about the ethics of this kind of method… you are not giving much consideration to the production of these texts, so I wondered if you could talk about the ethics of this work and responsibilities as researchers.

A: Well I didn’t ruin any rainforests, or ruined any summers. These people have signed up for terms and conditions. They are responsible for taxation in their jurisdiction. Our agreement with Crowdflower gives them responsibility. And it’s voluntary. Hopefully no sweatshops for this… I’m receptive to the idea of what ethical concerns could be… but couldn’t see anything inherently wrong about the notion of crowdsourcing that would be a concern. Did run past ethics committee at LSE. Didn’t directly contact people, completing tasks on the internet through third party supplier.

Q: You were showing public domain documents… but for research documents not in the public domain how would security be handled…

A: Generally transcriptions are private… but segments are usually 3 or 5 segments… like reading a document from the shredder basket… the system have that data but workers do not have access to that system

Q: But the system does have that so you need trust in the platform…

A: Yes.

Comment from floor: companies like Crowdflower have convinced companies to give them data – doctors notes etc. they have had to work on making sure they can assure customers about privacy of data… as a researcher when you go in you can consider what is being done in that business market in comparison

Q: Have you compared volunteer coders to paid coders? I am thinking particularly about ethical side of things and motivations, particularly given how in political tasks participants often have their own agendas. Might be interesting to do.

A: Volunteer crowdsourcing? Yes, it would be interesting to compare that…

Reading Data: Experiments in the Generative Humanities – Dr Lisa Otty, Lecturer in English Literature and Digital Humanities, University of Edinburgh (session chair: Dr Tom Mole)

Dr Tom Mole is introducing our next speaker, Dr Lisa Otty whose interests are in the relationship betweeen reading, writing and the technologies of transcription. And she will be talking about her work on Reading Poetry, and the process of what happens when we read a poem.

Now to be  a literature scholar speaking at an event like this I have to acknowledge that data is not a term typically used in our field. When you think about what we are used to reading texts are often books, poems… but a text is not neccassarily a traditional material but may also be another linguistic unit, something more complex. Taking the Open Archival Information Systems (CCSDS 2002) describes data as “a reinterpretable representation of information in a formalized manner suitable for communication, interpretatio, or processing”. Interpretation being crucial there. When we look at texts like books or poems those are “cooked” – edited, curated, finished. Data is too often not seen as that.

Johanna Drucker – in Humanities Approaches to Graphical Display (DHQ 5.1 2011) talks about data as Taken Not Given, Constructed from the Phenomological World. Data passes itself off as a priori conditions, as if same as phenomena observed, collapsing the critical gap between the data collection and observation.

Some of these arguements gel with some of the arguements around close versus distance reading. And I think it can therefore be more productive to see data as a generative process…

Between 2009-2012 I was involved in the research project Poetry Beyond Text (University of Glasgow, and University of Kent). This was a collaborative project so inevitably some of my reflections and insights are also collaborative and I would like to acknowledge my colleagues work here. The project was looking at interpretation of poetry, and particular visual forms of poetry such as artist boks. What these works share is that they are deeply resistent to being shared as just information.

For example Eugen Gomringer’s (1954) “silencio” is an example of how the space is more resonant than the words around it… So how do we interpret these texts? And how do our processes for interpretation effect our understanding. One method, popular in psychology, is eye tracking… a physical way of registering what you are doing. We combined eye-tracking with self-reporting. Eye Tracking takes advantage of the movements of a small area of the retina. So a map of concentration sees those little jumps, those movements around the page. But it’s an odd process to be part of – you wear a head brace with a camera focused on your eye. You get a great deal of data from the process. Where more concentration that usually indicates trickiness or challenge or interest in that section – particularly likely for challenging parts of text. From this data you can generate visualisations from this data. (We are watching a video of eye tracking process for poetry).

Doing this we found a lot of patterns. We saw that people did focus and understand space, but only when that space has significance in the process. In poems where space is more conceptual than nemetic. But interestingly people who recorded high confusion also reported liking them much more… With experiments with post linear poems the cross-linear connections. All people start with a linear reading patterns before visual reading. And that reflects the colour strip test – psychology test that shows that visual information trumps linguistic information… so visual readings and habitual reading processes are hard to overcome. We are programmed to read in a certain way… our habits are only broken by obstacles or glitches in the text we are reading…

Now talking about this project if I talk about findings I am back in that traditional research methods… and that would be misleading. We were a cross disciplinary team and so I am particularly interested in focusing on that process, on how we worked on that. The eye tracking data generates huge amounts of numerical data… we faced real challenges in understanding how to understand, to read this data… a useful reminder of the fact that data’s apparent neutrality has real repurcussions. Its one thing to make data open, another to enable people to work with it.

To my colleagues in psychology didn’t understand our interest in visualisations of numerical eye tracking data, it is an abstraction… and you have to understand the software to understand how that abstraction works. Psychologists like to interpret the data through the numerical data. They see visualisations, graphs etc. as having a rhetorical rather than analytical function. Our team were interested in that rhetorical function. We were humanists running an experiment – the framework was of hypotheses, of labs, of subjects… but the team came from creative practice background so this sense of experiment was also in play. In it’s broadest terms experiments are about seeing something in process and see how they behave, for scientists about testing hypotheses in this way, creative experiements rather different… For humanist analysis of these texts you have to deal with a huge number of variables, very much a contrast to traditional psychology experiements. For creative experiments there is a long tradition of work in surrealism, dadaism, etc. that poetry can unleash and disrupt our traditional reading of texts… they are deliberately breaking our habits. The reader of the literary form is a potentially revolutionasible(?) subject.

In Literary scholarship and humanities the process of reading is social, contextualised process. In psychology reading is a biomedical process, my colleagues in this field collapse the human and machine. In a recent article by Lutz Koepnick asked Can Computers Read? (2014) and discussed the different possible understandings of what reading is for.. our ideological framework of reading means to us… computational reading is less about what computers are, more about how we invest in them and envision them.

One of the things that came out of our project was the connections between poetry and psychology, and the connections to creative experiments.

To finish I want to talk about some examples of experiments around reading and what reading can mean.

The readers project – John Cayley and Daniel Howe (2009 – ) their work explores imaginative critiques of reading. Cayley is a literary scholar and has been working in digital production for some time. The readers project features “programmed autonomous entities”. Each reader moves through a text at different speeds and in different ways. So for each part of the experiment projections are used, and they are often shown with books, a deliberate choice. A number of interfaces are available. But these readers move according to machine reading rather than biomechanical reading. Cayley terms this an exploration of vectors of reading… directions in which reading might take of. It explores and engaged with new creative understandings of reading. This seems to be seen by Cayley in avant garde context. Emphasis on constructed nature of the work.

“because the project’s readers move within and are thus composed by the words within which they move, they also, effectively, write. They generate trxts and the traces of their writings are offered to th eproject’s human readers as such, as writing, as literary art.” (Cayley, The Readers Project website).

As someone engaging with these pieces the experience is of reading with, more than processing or consuming or analysing.

Tower – by Simon Biggs and Mark Shovman (2011), working at Hive, uses knowledge of natural language processing to build visualisations. When the interactor speaks their words spiral around them. And other texts are also present – the project is inspired by the Tower of Babel and builds up and up. Shovman’s previous work at Hive was on geometric structure. Biggs hope is that participants “will be enabled to reflect upon the inter-relations of the things that they are experiencing and their own contingency as part of that set of things.”

Michelle Kendrick talks about hybrids, that hybrid of human and machine interaction, the centrality of human investment in computer reading.

When I talk about this work I am overwhelmed by the rhetorical significance of words like “experiment” and the dominance of scientific research methods – the first interpretation of this work is often wrongly around seeing the work as applying scientific methods to literary interpretation.  But instead this work is about interpretation and exploring methods of understanding and interpretation.

Q&A

Q: You talked about different disciplines coming together. Do you think there is a need for humanities researchers to understand data and computational methods?

A: I think we would all benefit from a better understanding of data and analysis, particularly as we move more and more into using digital tools. I’m not sure if that needs to be in the curriculum but it’s certainly important.

Q: One of the interesting things about reading is the idea of it being a process of encoding and decoding… but the code shifts continously… and a challenge in experimental reading or interpretation is that literature is always experimental to some extent because the code always changes.

A: I think the idea of reading as always being experimental… I think that experimental writing is about disruption… less about process but more about creating challenge.

Q: I was very struck in what you were presenting there in the Poetry Beyond Text project about the importance of spatiality and space… so I was wondering about explicit spatial understandings – the eye tracking being a form of spatial understanding…

A: We were looking at the way that people had been interpreting those texts in the past, in the ways people had looked at that poetry in the past… they had talked about the structural work of the poets themselves… and we wanted to look beyond that…We wanted to find out people’s responses to some of these processes, and what the relationship was between that experience and those critical views of those texts.

Q: Did you do any work on different kinds of readers – expert readers or people who had studied these works?

A: It was quite a small group but we looked at the same people over time and we did see development over time. We worked mainly with students in literature or art and most hadn’t encountered this type of concrete poetry before but were well experienced with reading.

Q: I wanted to ask you about the ways in which we are trained to read… there are apps showing images of texts very very quickly, are we developing skills to read quickly rather than more fully and understand the text.

A: There was a process of rapid image showing to the eye (RSVP was the acronym) – to allow you to absorb more quickly but in actual fact that was quite uncomfortable. We do see digital texts playing with those notions. I don’t think we will move away from slow reading but we are seeing more of these rapid reading processes and technologies.

Chair: Kinetic Text project works in some of these ways, about focusing eye movement…

A: The text can also manipulate eye movement and therefore your reading and understanding of the text. Very interesting in that respect.

Algorithm Data and Interpretation – Dr Stephen Ramsay, Associate Professor of English at the University of Nebraska; Fellow at the Center for Digital Research in the Humanities (session chair: Prof James Loxley)

James Loxley is introducing our next speaker, Dr Stephen Ramsay.

I want to say that my mother is from Ireland, a little place west of here, and she said that if she had ever been to University it would have been to University of Edinburgh which she felt was the best in the world.

Now I was planning to teach a technical talk – I teach computer science in an English faculty. But instead I’m going to talk about data. So I’m going to start with the 1965 blackout of New York. At the time it was about disaster, groping in the dark, a city stranded. But then 9 months later they ran stories on the growth in birth rates, a sharp rise across hospitals across the state. All recording above average numbers of births. Although one report noted that Jewish hospitals did not see an increase. Sociologists talked about the blackout as in some way responsible… three years later a sociologist published a terse statement showing no increase in births after the Great Blackout. This work looked at average gestation period and noting that births would have been higher from June through to August, not just in August… but he found that 1966 was not unusual or remarkable. Black Out Babies were a myth…

You could read this tale as a cautionary one about the misuse of data. But I think this can be read another way… the New York Times piece said something about human nature – people turning to each other when power out is a sad reflection on the place of television in our life, but a hopeful narrative for humanity. And citing birth rates and data and using scientific language adds to that. And the comments about Jewish people shows prejudice. But at the same time that subsequent analysis frames the public as prone to fantasy, as uninformed, with the scholar overcoming this…

The idea of “lies, damn lies, and statistics” encourages us to always look for falsehood hiding behind truth… so we think of what stories we are being told, and what story we want to tell. It’s simple advice that is hard to do. I want to give a different spin on this. I think that data is narrative automatic. the way we use data is instructive – we talk about lists, numbers… Pride and Prejusice does not seem to be a data set unless we convert it. It gains narrative in transformation. The data can be shown to show and mean things – like stories, stories waiting to be told… data doesn’t mean anything by itself, someone has to hear what it is saying…

What does data look like in its pre interpretive state? There is an internet site called “Found” – collecting random items such as notes, cards, love letters, shopping lists. Materials without their context. Abandoned artefacts. All can be found there. But the great glorious treasure of Found is it’s lists…

[small pause here for technical difficulty reasons]

These lists are just abandoned slips of paper… one for instance says:

beer

neat

dogfoot

domestic

stenga

another:

roach spray

flashlight

watermellon

The spareness and absence of context turns these data-like lists turns them, quickly into narrative… not all are funny… one reads:

go out for a walk with someone

speak with someone

watch tv

go out to cemetry to speak to mom

go to my room

Have you ever wanted to give your data a hug? Bram Stoker said in writing Dracula he just wanted to write something scary… his novel is far more interesting without him as the interpretations of others are fascinating and intriguing… Do facts matter in the humanities? In some areas… who painted a picture, when a treaty was signed… these are not contingent truth claims… surely we can say fact is a good word for those things that are not subject to debate. Scholars can debate whether a painting is by Rembrandt or his school, that debate is about establishing a fact. But facts still matter…

If we look at Rembrandt’s Night Watch the lighting of the girl equating to that of the captain is intriguing. If he said it meant nothing we’d probably ignore him… The signing of a treaty may be a fact but why it occured is much more interesting. Humanities are about that category 1 inquiry more than the category 2 fact inquiries. Often this is the critique of the humanities and the digital humanities, Jonathan Gotschil insists that the humanities should embrace scientific approaches and sense of optimism… And sees the sciences as doing a better job of this stuff but that “what makes literature special” should be retained… he doesn’t say what those things are. There are unsettled matters if one takes scientific approaches. Of course Gotschil’s nightmare is to understand data with the same criticality we apply to Bram Stoker, questioning it’s being and meaning… and I suggest we make that nightmare a reality!

[More technical issues… ]

What I wanted to show you was a list of English Novels [being read to us]… It is a list, from Hoover, organises novels in terms of breadth of the vocabulary in that list. I have shown this list to many people over the last few years, including many professors… they see Faulkner and Henry James at the top and approve of that and of Mark Twain…. and young adult novel writers at the bottom… but actually I read you the list in ascending order… Faulkner and James are at the bottom. Kipling and Lewis are at the top. And there it starts… richness is questioned… people want to point out how clearly correct the answer is, despite having given the wrong answer; some explain that the methodology is flawed or misreported… these are category 1 people being annoyed by category 2 reality…

But when we stop using it as a Gotcha it is a more provocative question… each of these titles contains a thousand, a hundred thousand thoughts and connections… it is what we do… as humanists we make those connections… we ask questions of the narrative we have created… part of our problem is a general discomfort with lettinng the computer telling us what is so… but if we stop doing that we might see peculiar mappings of books a cultural objects… it might show us a way to deeper understanding of reading itself… it raises any number of questions about the development of English style… and most of all it raises questions of our discursive paradigms.

That gives us narrative possibilities we could not see. We cannot think of text as 50k word blocks. The computer can ONLY apprehend the text in such terms. To understand the computer as finding facts is to miss the point. It is about creating triggers to ask questions, to look at the text in new ways. This is something I came across working on Virginia Woolf’s The Wave. The structure is so orderly… and without traditional cultural narrative. And they speak in very similar styles, sentence structures, image patterns… some see some difference between gender or solidarity… but overall it is about unity… this is the sort of problem that attracts text analysis scholars like myself. I ran algorithm clustering models looking for similaritudes unseen by scholars. On a lark we posted a simple question… “what are the words that the women in the novel use in common, that none of the men do?” and it turns out that there are 9 such words. Could see that as a narrative – like a Found list – and then we did it with men and found 120 words! Dramatic. So many words… Some critics found that disparity frightening… some think it backs up sexism of western cannon. Others see this as a chance to ask another questions… to try with other authors, novels, characters… if you think this way, perhaps you’ve caught the DH bug, I welcome you. But do we think we’ll find an answer to questions of gender and isolation? Do we want to answer those? The humanities want a world that is more complex, deeper than we thoughts. That process is a conversation…

In 2015 the Text project will release huge volumes of literature. Perseus contains most greek texts… there are huge new resouerces. almost all questions we ask of these corpuses have not been asked before… we can say they will transform the humanities but that may not be true… the limiting factor is whether we choose to remain humanists in the face of such abundance… perhaps we need to be programmers, tool builders, text engineers… many more of us need to invite the new texts – lists, ngrams, maps etc. – into our ongoing conversation. We are here to talk about philosophical issues of data and these issues are critical… but we have to be engaging with these questions…. Digital humanities means databases, mark up, watermelon…!

Q&A

Q: I am intrigued to think about how we design for the things we don’t know what we need to know…

A: Sure, imagining what we don’t know… you inevitably build your own questions into the tools… ironically an issue for scientific methods. The nice thing about computers is that they are fast, obedient and stupid. They will do anything we ask them to, even our own most stupid ideas, huge serendipity just baked into that! Its a problem but its amazing how the computer does that job for me, surprisingly.

Q: That was a brilliant fascinating talk. Part of the problem with digital humanities for literature right now is that it either tells us what we do know… or it tells us what we don’t know but then we worry that it’s wrong… The description of the richness list was part of that. I really liked your call for an ongoing discussion that includes computer generated data… but I don’t see how we get past the current description. If all literary criticism says something is so, and expects “yes, but…” I can see how computer generated data sits in that… but how can data be a participant in that conversation – beyond ruling something out, or concurring with expectations.

A: Excellent point and lets not downplay at all the first part of your question. I saw Franco Morelli give a talk about titles getting shorter for instance… who’d have thought?! But I think it has a lot to do with how we build our tools… I find it frustrating that we all use R, or tools designed for science or psychology… I want our schools to look more like the art-informed projects Lisa talked about. I think the humanities needs to do more like that, to generate the synergies. Tools that are more ludic.

Q: May be to be about perceived barriers being quite high. An earlier speaker talked about the role of repeatability. Ambiguity reading a poem is repeatible. if barriers to entry low enough for repitition and for others to play, to ask new questions, maybe that brings the data in as part of the conversation…

A: There are tools that let you play with the text more ludically. Voyant for instance. But we come with a lot of cultural baggage as humanists… there is a phenomenon that… no matter what they are talking about they give a literary critical reading of a text but when they show a graph we all think we are scientists… there is so much cultural baggage. We haven’t learned how to be humanistic users of these tools, or to create our own tool.

Q: A question and an observation… There is a school of thought in cognitive psychology that humans are infinitely able to retrofit any narrative to any circumstances whatsoever, and that is very much what was coming through your data… Many humanities departments have become pseudo social sciences departments… but if you don’t have a clear distinction between category 1 and category 2 they can end up doing their own thing…

A: I don’t want the humanities. I resist the social science type study of literature, the study of human record or of the human condition… when we are talking about… in my own work I move between being a literary critic and being an engineer… when it comes to writing software that method definition is wrong, it doesn’t work… when I am a literary critic it is about all those shades of grey, those complexities… but those different states both seem important in pursuit of that end goal… if studying flu outbreaks lets not be ludic… but for Bram Stroker then we should!

Q: In my own field of politics there was a particular set of work which gave statistical data a bad name… and I wonder in your field is the risk of the same is there…

A: In digital literary studies this is sometimes seen as a 25 year project to get literary profs into the digital field.. but I always say that that’s not true, there’ll always be things to be done. There was a book in the 70s that looked at slavery in an entirely quantitative way, it made the arguement no one wanted to hear, that slavery had been extremely lucrative. Economists said that it’s profitable. History fled from statistical methods for years after that… but they do all agree that that was profitable. And there is quantitative work there again/still. If I had to predict I’d say the same thing for digital literary studies does seem likely…

Q: I can’t resist one here… I was following a blog by Kirsch where you say that scholars should code and I wanted to ask about that…

A: OK, well Kirsch lumps me in with the positivists… I’m not quite in the devils party. But I teach programming and software engineering to humanists. Its extremely divisive… My views have softened over the years… for me programming is a magnificant intellectual excercise… knowing about it seems to help understand the world. But also if you want to do research in this area you need some technical skills. If that’s programming… well learn what you need whether thats GIS, 3D Graphics… if you want to build things you might need coding!

Big Data and the Co-Production of Social Scientific Knowledge – Prof Rob Procter, Professor of Social Informatics, University of Warwick (session chair: Prof Robin Williams)

Professor Robin Williams is now introducing Professor Rob Proctor, our next speaker, talking about his work around social informatics.

The eagle eyed amongst you will spot my change of title – but digital is infinitely rewritable! I am working in the overlap of sociology and computational tools and methods. So, the second thing I want to talk about is Sociology in the age of “big data”. I think what this demonstrates is the opportunities for sociology to respond in various different ways to this big data, and tools to interrogate that data. The evolving of tools and methods is a key thing to look at in the area. So that brings me to the Collaborative Online Social Media Observatory (COSMOS) and tools we are developing for understanding social media… and then I want to talk about Sociology beyond the academy – knowledge co-produced of social scientific knowledge. But there are other types of expertise being mobilised at the moment, in looking at the computational turns things are taking. Not always a comfortable thing for social scientists…

So firstly Social Informatics. So what is that? Well to me its the inter-disciplinary study of factors that shape adoption and use of ICTs. And what gets me excited is how these then move into real processes. And for me the emphasis on innovation as public, participatory process of experimentation and learning where meanings of technologies are collaboratively explored and co-produced. In social media you can argue that this is a large scale experiment in social learning… Of course as we witness growing scale of adoption more people experience those processes: how social media works, how they might adopt or use it… to me this is a fascinating area to study. And because it is public and involves social media it is very easy to see what’s going on… to some extent. And generally that data is accessible for social research purposes. It is not quite that simple but you can research without barriers of having to pay for data if you do it in a careful way.

So these developments have led me into social media as a prime area of my research. So firstly some work we did on the impact of Web 2.0 on scholarly communications – work with Robin Williams and James Stewart – many of us will be part of this, many of us tweet our research… but many of us are not clear of what that means, what the implications are. So we did some work, got some interesting demographic research… we also did interviews with people and got ideas of why they were, and why they were not adopting… Some very polarised. And in parallel we looked at how scholarly publishers incorporate social media tools into their work, in order to remain key players… they do lots of experiments and often that is focused on measuring impact and seeing the movement of their work to other audiences. Some try providing blogs on their content. But that is all with mixed success. A comment notes that it is easier to get comments on cricket reports than on research online… So it’s hard to understand and capture impact…

I’ll come back to that and about co-creation of knowledge. But first I want to talk about the riots in England in 2011. This was work in conjunction with the Guardian Newspaper. They had been given 2.5 million tweets directly by Twitter. They wanted to know if social media was particularly vulnerable for sharing false information, did that support calls for shutting down social media at times of crisis? So we looks at a number of different rumours known about and present in the corpus: zoo animals on the loose; london eye on fire; miss selfridge on fire; rioters attack a children’s hospital in Birmingham. I will talk about that latter example. But we wanted to ask about how people use and understand and interpret social media in these circumstances, how they engage with rumous…

So this is about sociology in the age of “big data”. It calls for interpretive methods but we can’t do that at scale easily… so we need computational methods to focus scarce human resources. We could crowdsource some of this but at this scale that would still be a challenge…

So firstly lets look at the work of Savage and Burrows (2007) talked about the “coming crisis of empirical sociology” because the best sociology, as they saw it, was conducted by private companies who have the greatest and most useful data sets which sociologists could not rival nor access. However we might be more confident about the continuing relevance of social sciences… social media provides a lot of born digital data… maybe this should be entitled the “social data deluge”. There is a lot of data available, much of it freely available. Meanwhile lots of policy initiatives to promote open data in government for/by anyone with a legitimate usage for it. Perhaps we can be more confident about the future of academic sociology…

But if you see the purpose this data is put to, its a more mixed picture… so we see analysis of social media for stock market prediction. But here correlation is mistaken for causality. Perhaps more interesting are protest movements – like occupy wallstreet – or use of social media during the Egyptian revolution… It is a tool for political change, a way for citizens to acquire more freedom and change? Is it a movement to organise themselves? Lots of discussion of these contexts. Methodologically its a challenge of quantity, and methods that combine social science understanding with social media tools enabling analysis of large scale data…

So back to that rumour from the riots and that rumour of a children’s hospital being attacked in Birmingham. This requires thorough work with the data, but focused where it counts.

So, what sparked this off was someone tweeting that the police were assembling in large numbers outside the hospital… therefore the hospital must be under threat. A reasonable inference.

So, methodologically we undertook computational methods for analysing tweets in an active area of research: sentiment analysis; topic analysis. We combine a relatively simple tool looking at information flows… and then looking at flow from “opinion leaders” to others (e.g. RTs). Once that information flow analysis has been done we can then take those relative sizes to analyse that data, size as proxy for importance… this structure, we argue, is relatively useful for focusing human effort. And then we used coding frames for conventional qualitative methods of content analysis to understand how Twitter was used – to inductively analyse information flow content to develop a “code frame” of topics; use code frame to categorise inofrmation flows (e.g. agreement, disagreement, etc.); and then we used visualisation around that analysis of information flows…

So here we see that original tweet… you see the rumour mushroom, versions appear… bounding circles reflect information flows… and individuals and their influence… Initially tweets agree/repeat… and we then start to see common sense reasoning: those working or nearby dispute the threat, others point out that the police station is next door to the hospital thus providing alternative understanding. People respond and do not just accept the rumor as true… So rumours do break quickly BUT they are not neccassarily more vulnerable as versions and challenges quickly appear to provide alternative likely truth. That process might be more rapid with authoritative sources – media or police in this case – adding their voice. But false information may persist longer, with potential risk to public safety – see follow on Pheme project.

But I wanted to talk about authoritative sources again. The police and media and how they use social media. The question is what were the police doing on twitter at that time? Well another interesting case here… riots in Manchester led to people creating new accounts to draw attention to public bodies like the police, as an auxillery service to raise awareness of what was going on. Quite an interesting use of social meidia where these see something like this arising.

So what these examples demonstrate is innovation as a co-production… lots of people collectively experimenting, trying out things, learning about what social media can and cannot do. So I think it’s a prime example for sociologists. And we see uses are emergent, people learn as they use… and it continues to change and people reinvent their own uses… And we all do this, we have our own uses and agenda shaping our interactions.

So this work led to development of tools for use by social scientists… COSMOS involved James S, Ewan K, etc. from Edinburgh… It would be an error to assume social media can tell us everything that takes place in the world – this data goes with crime data, demographic data, etc. The aim of COSMOS is to forge interdisciplinary working between social and computing scientists. To provide open, sustainable platform for interoperable social media analysis tools. And refine and evolve capabilities, provide service models compatible with needs of diverse user communities.

There are existing tools out there for social media analysis… but many are blackbox systems, its hard to understand that process that is taking place. So we want those blackbox processes to be opened up, they are complex but can be understood and explored…

So the Cosmos Tools let you view timelines, to look at rates and flows… to look for selection based on keywords and hashtags… and to view the networks of who is tweeting… and to compare data with demographic data.

Also some experimental tools around geographical tools for clustering. The way people use Twitter can show geographical patterns. Another factor is about topic modelling, topic clustering… identifying tweets on the same topic. This is where NLP and Ewan and his colleagues in Informatics has become important.

So current research looking at: Social media and civil society – social media as digital agora; “hate” speech and social media – understanding users, networks and information flows –  a learning challenge here about people not understanding impact and implications of their comments, perhaps a misunderstanding of social media… ; citizen social science – harnessing volunteer effort; social media and predictions – crime sensing, data integration and statistical modelling; suicide clusters and social media; humanitariansim 2.0 – care for the future; BBC World Service – tweeting the olympics. And we have a wide range of collaborators and community engagement.

Let me briefly talk about social media as digital agora… may sound implausible… many talk about social media as a force for change… opportunities to promote democracy… not just in less democratic countries, but also democratic countries where processes don’t seem to work as well… So we are looking at social media in communicative, in smaller communities. And also thinking about social resiliance in a day to day small scale way… problems which if not managed may become bigger issues. For that we have studied Twitter in several locations, collected data, interviewed participants… and built up a network of communications. What is interesting, for instance, is that non governmental group @c3sc seems to have big impact. We have to see how this all plays out… deserves longitudinal approach…

So, to conclude… let me talk about the lessons for academic sociology… and I think it’s about sociology beyond the academy and the role of wider players. Firstly data journalism – was interested in Steven’s 1965 press accounts of the black out earlier. Perhaps nowadays the way journalists are being trained might change that… journalists are increasingly data savvy. We see this through Fact Check, through RealityCheck blog… through sourcing from social media. So is citizen journalism, used to gather evidence of what is happening… tools like Ushahidi… and a sense of empowerment for these communities… reminds me of notion of sousveillance… and the possibility of greater accountability… And Citizen Journalism in the expenses scandal – guardian recruited people to look at the expense claims. The journalists couldn’t do that externally… so recruited others.

So, citizen social science… in various ways (see Harris 2012 “Oh man, the crowd is getting an F in social science”. And Ken Benoit’s work discussed earlier… we see more people coming into social science understanding…

So the boundaries of social science research production are becoming more porous, social scientific knowledge production is changing, potentially becoming more open. These developments create an opportunity to reinvigorate the project for a “public sociology” – as per Burawoy (2005) and his call “For a public sociology”. to make sociology accountable to more people, to organisations, to those in power. Ethically we need to ask what is needed and wanted, how the agenda is set, how to deliver more meaningful and useful social sciences to the public.

How can we do that? New modes of scholarly communications, technology, but it’s not enough… we’ve also been working with a company on a  possible programme for the BBC where social media is used to reflect on the week, a knowledge transfer concept. Also knowledge transfer in the Pheme project – for discriminating false and true information… all quite conventional… but we need other pathways to impact… with people as sensors and interpreters of social life, training and capacity building – in ways we have not done before, and something that has emerged in science and citizen science has been the notion of workshops, hackathons, getting people engaged in using mundane technologies for their own research (e.g. Public Lab), we need something similar for tools, social media, to extract data they want for their purposes for their agenda… to create more public sociology that people can do themselves. And we need to also have an open dialogue about research problems.

Q&A

Q: My question is about COSMOS and the riot rumours stuff… within COSMOS do you have space for formal input around ethics and law… you cut close to making people identifiable and locatable. And related to that… with police in those circles… may arouse suspicions about motives… for instance in Birmingham did police just monitor or did they tweet.

A: They did tweet but not on that rumour. It is an understandable concern that collaborations make powerful state actors more powerful… for us we want these technologies available for anyone to use them… not some exclusive arrangement, should be available to communities, third sector organisations… anyone who feels that social media may be important in their research

Q: I was more concerned about self-led vigilantes, those who might gang up on others…

A: A responsibility of civil society to be aware of those dangers, to have mechanisms to avoid harm. It does exist already… so if social media becomes instrument of that we have to respond and be aware – partly what hate speech project is about… Bigger learning problem is about conduct in social media space. And the probably issue that people don’t realise how conduct quickly becomes visible to much bigger group of others… and that relates to ethics… twitter is public domain space but when something is highlighted by others… we have to revisit the ethics issues time and again… for the study for the riots we did the usual clearance process… Like Ken we were told it was fine… but don’t make identifiable but that is nearly impossible in social media. Not an easy thing to resolve.

Q: I’m curious about changes in social media platforms and how that effects us… moves from facebook to twitter to snapchat to instagram… how does that become apparent, may be invisible, how do we track that..

A: There is a fundamental issue of sustainability of access to data from social media. Not too much of a problem to gather data if you design harvesting appropriately for their rate limits. In terms of other platforms, and people moving to them, and changes in modality and observability and accessibility of data… what social research needs is agreement with providers of data that, under certain conditions of access, that their data is available for research.. to make access for legitimate data easy. There are efforts to archive data – Library of Congress collects all tweets. Likely to allow access under license I think, to ensure access to platforms as use of platforms change…

Edinburgh Data Science initiative – Prof Dave Robertson, Head of School of Informatics

Sian Bayne quickly introducing Dave Robertson providing a coda to today’s session.

I’m just briefly going to talk about the Edinburgh Data Science Initiative. The ideas being data as the catalyst for change in multiple academic disciplines and business sectors.

So firstly the business side… big data can be very big and very fast… that can be off-putting in the humanities… And you don’t have to build something big to be part of this… I work in these areas but my models are small… and there is a stack you never see – economic and political side of this stuff.

And here’s the other one… this is about variety and velocity – a chart from IBM – looking at predictions of the volume of data and, more interestingly, the uncertainty of data… And the data sites in a few categories… Enterprise Data, loads of Social Media, and loads of Sensors (internet of things)… but uncertainty over aggregate data is getting hugely large… and that’s not in sphere of traditional engineering, or traditional business…

The next slide here is about architectures… this is topical… it’s IBM’s Watson system… this is the one that won Jeopardy… harvested loads of information and hypothesis generation… This stack starts with very computational stuff but the top layers look much more like humanities work and concepts…

Now technology and society interact. Often technology pushes on society. For instance if we look at Moore’s Law (memory in your computer doubles every year) mapped against the cost of mapping the human genome. It looks radically different, costs drop hugely in late 2000’s as a lot of effort is pushed in here. And that drop in cost to $1000 per unit… that is socially important… I could sequence my genome… maybe I don’t want to. You can sequence at population scales… machines generate a TB of data a week too – huge data being generated! And this works the other way around… sometimes technology gives you an inflection point and you have to keep up, sometimes society pushes back. A lot of time online is spent on social networks (allegedly 1/7)… now a unified channel for discovery and interaction… And the number of connected devices is zooming up…

So that’s the sort of thing that is pushing a lot of things… A lot of people have spoken to all the schools in the university… everyone reacts… you will find everyone recognising this… and you hear them saying “and it changes the way it makes me think about my research”. That’s so unusual to have such a common response…

Why this is important at Edinburgh… We have many interdisciplinary foundations at Edinburgh… All are relevant, no matter how data intensive, but we are well developed in interdisciplinary working…

And we have a whole data driven start up Ecosystem in Edinburgh… we have Silicon Walk (miicard, zonefox, etc.), Waverley Gate (Amazon, Microsoft), Appleton Tower (Informatics Ventures, feusd, Disney research, tigerface), Evo House (FlockEdu, Lucky Frame, etc), Quartermile (Skyscanner, IBM), Informatics, Techcube (FanDuel, Outplay, CloudSoft, etc.). A huge ecosystem here!

So, I’ll leave it there but input, feedback welcomed, just speak to myself and/or Kevin.

And that was it for the day…

Related resources:

 May 14, 2014  Posted by at 10:10 am Events Attended, LiveBlogs Tagged with: , , ,  1 Response »
Apr 102014
 

Welcome – Jeff Hayward

Jeff is noting how this is the twelfth annual elearning@ed event, and that’s a really notable length of time for an internal event like this.

There’s a real buzz about technology enhanced learning, eLearning, or whatever you want to call it. There seems to be a stepping up a gear and a real sense of fun and creativity here. Lots of rethinking of pedagogies, and of teaching and learning and the use of technology in this. And I think we’ve tended to do it that way around, and kept a solid idea of skills that students need as they go out into the world.

I also want to thank all of you working on MOOCs. And I wanted to thank all of you who are involved in the online masters programmes. I think we are quite unusual to have so many of these, so fully across the university. As the first phase of the DEI programme comes to an end we are well on our way to the 10,000 student target. Now that’s the good bit. But I know our students would like greater consistency in our use of technology, and technology a cross the programme.

Part of the stepping up a gear has been the advertising of senior posts in online learning. We include online learning in the job descriptions of senior staff in schools, of senior management, and that’s really significant. We are seeing schools who haven’t been involved in DEI yet, are coming forward now. We have ambitious plans for investment in digital education. And we have recently formed a new division, headed by Melissa Highton, within Information services to take this forward. And MVM teaching and learning technology team are joining IS so we will have a big experienced team taking this forward. So a lot of excitement and fun looking forward!

Keynote: Cargo cult teaching- the importance of authentic practice – Ross Galloway

Thinking about what authenticity might look like I felt there were three key areas to authenticity: authentic practice by instructors; authentic practice by students – particularly thinking about what students will go forward with in their careers; authentic practice in educational research.

I want to start with the notion of authentic practice by instructors. Here we have our classic 12th C picture of a lecture… And that’s pretty much looks like lectures now…. Very passive audience…

But there are alternatives as well. But there are movements around active learning, group learning. Problem based learning where students are more active, more engaged. So another picture of a lecture theatre here shows students working in groups, directing their attention across the row, not to the front.

Why should we care about how we teach? Here’s some compelling data from physics (hake, am. J. Phys. 1998) which shows evaluation of a number of introductory physics courses. It shows “gain” – the difference between pre and post testing, showing any improvement. Normalised gain of 0 means people have learned nothing. Normalised gain of 1 means they have learnt everything. Passive learning doesn’t show much over 0.3 gain. Active engagement varies but sits much higher in the 0.5 the 0.6 and above. So I will be bold and say that active learning is what works. The evidence is there. Surely we should be in a golden age for active learning then?

Surveys in the US did show 87% awareness of evidence based reformed approach. And almost half used them. Slightly lower in the UK… But still good… Except there is a catch….

For physics in the US more than a third of instructors who try these new approaches, subsequently discontinue (Henderson etc al 2012). In biology instructors report that they don’t see better results? What happens? Well we hear instructors saying “I tried it, it didn’t work!”.

So I want to talk a bit about Cargo Cults. During the war cargo planes were dropping materials and supplies. Locals also benefitted. But the war ended and the planes stopped. So locals missed that, they knew you needed watchtowers, then planes would come… But they didn’t. And this is a real phenomenon….

And that’s a bit like what has been happening with these new teaching approaches. So in physics 1/4 to 1/2 of instructors deviate significantly from established design of evidence-based teaching approaches (Henderson and dancy 2009). And a wide variation in actual classroom practices for the “same” approach (Taylor and finkelstein?). It’s like those wooden control towers… It looks like they’ve done the right thing but it’s not going to work the same way…

So an example. Peer instruction… You might pose a question. Let students think and vote, let students discuss amongst themselves, students revote, whole class discussion, confirm and summarise. That’s the evidence based approach.

But what happens in reality in some classes is people miss out the “students think and vote” so you never get that marker in the sand. Asking the question first means you have thought and committed. So you have to confront why there is disagreement. You want to engage and resolve conflict, reform existing conceptions. Skipping it means students votes come from a very different place. How many of those skipping that stage don’t even know why that step is there?

The other part which Is often missed out is the confirm and summarise stage. Students can get partway through learning, be developing ideas. But that confirmation and summarising is really important,it firms up what the correct approach is and why, is confirms what has been learned.

So, what to do? Well don’t blame the instructor! “Us versus them is not constructive” (dancy and Henderson 2010). Instructors are often not hugely aware of learning theory but that doesn’t mean they are unbelievers, that they aren’t open to change even if they do use traditional methods.

Avid don’t tell instructors what to do – an informed partnership works better (Henderson and dancy 2007). We are instructors because we are experts, we know what we are talking about. And there are key pragmatic reasons that some practices are hard to do – room layout can make a huge difference for instance.

So what do We do?

Classroom approach needs to correspond to the authentic practices of the educational reform. Implementation needs to be supported.

So that’s the polemic, now some examples. And starting with a failed experiment in reformed approaches. I can talk about this failure because it is mine!

So for undergraduate physics we have an assignment marking rubric. It works well, it’s supported by research. We look for techniques that experts use. For instance for mathematical execution and final answers we explicitly include “evidence of meaningful evaluation of answers”. So we want students to check over, assess, confirm things are correct. So, it makes sense. Super. Expert like.

What happens in practice? Students do this for equations like e=mc2. Where there is no point of doing that! Or they fail to do something in the equation and note a discrepancy. But don’t go back and recheck it. So they evaluated it but did not actually used it as a tool. They got so very close!

What’s the problem? From an expert perspective you do this stuff automatically… You work and correct as you go. But students see it as a hoop to leap through. It’s not useful or effective.

It’s not enough to encourage students to do what we do. The practice must be authentic within the context of the students activity – right there at that moment. It must be real. If it’s not it’s just that hoop to jump to. So I will take these things out of the play context, focus them in actual useful practices.

And that last section. Authentic practice in educational research. A happy successful example. Let’s go back to that peer instruction process. Wouldn’t it be nice to close the loop, to feed results into how we write questions. Why do this? Well voting responses highlight some concepts that are easily shifted – big gain. But sometimes we see something where the gain is very small pre and post discussion etc. how do we find out what happens here? Well we use smart pens. They give real insight into what is going on. These pens digitise and include microphones, captures pen strokes and audio recording in sync. I don’t listen in, that’s not fair… But I get to see process data. So this technology told us what went wrong in this question with terrible gain…

Firstly it was a negative question so confusing doubles. And there were lots of confusions about the symbols using in the question – which isn’t important. The concepts are the key focus or should be. And the question saw students focusing on irrelevant features. And symbols activate formula-based approach. These are superficial but divert students from talking about the core concept. I learned a lot here. I do walk the rom but students can feel inhibited so this technology really helps.

So we rewrote the question. We added an image to set up the idea of what was taking place. It’s no longer negative. And we took out symbols. But numbers still had to be here. And we retested this approach. And we went from gain of 0.09 to 0.51. That’s a great result. We did this for a number of other questions, revising question based on insights. Some saw modest improvements, some substantial improvements.

The smart pen technology is highlY effective for observing student process. It was embedded in a really authentic experience, the real classroom setting, real problems students were solving. A really authentic experience.

So, authenticity. Are we being authentic as instructors? Are students playing at being students or can we make their experience authentic and real and relevant to them. And how can we ensure when we look at educational research it’s relevant and authentic to us, to our teaching context.

Q&A

Q: thank you so much Ross. I was thinking about what you said about failure. You admitted something didn’t work. You talked about constraints for instructors… How can I encourage instructors to try something that might not actually work, that might be a failure, to engage and enjoy that experience regardless
A: I think there is no easy answer to that. But, welts all relative. Even when reformed practices don’t work well they are usually better than what went before. Didactic passive lectures work alarmingly poorly. Students often learn from books, from friends, from the libraries but take little from the classroom in that form. My practice could certainly be better but it’s an iterative approach. Even if you try it only once or twice a semester, to learn from something appropriate and authentic to their context. Incorporate small pieces and build from three,

Q: what data do you get from the smart pens?
A: you can see an animated PDF of line moving and audio effectively, it’s a proprietary format. Students transcribed, or looked for keywords, coded independently to make sure similar. That’s tricky actually. That takes some time to do but if I see two thirds hung up on symbols, that’s probably significant.

Q: I like to throw the messiness at my students, all the ambiguities. So if you clean up that question are you removing those?
A: I do like those ambiguities but I include those a bit differently. I have my students read ahead. We focus on fundamental concepts as most peoples fundamental concept of the universe is actually different from physics and deceptively difficult to shift. We do embrace ambiguity but not in lectures, in workshops where we have four or six tutors around and we ask big real world ambiguous questions. So questions like “how many street lights are there in edinburgh” – a question to think about what they can see, what they can estimate… The technically gifted students hate this. At school they are rewarded for the right answers. Physicists get employed because they have the skills to think “well I can see 12 lights from here so if I think about how many there might be across the city based on that…” to think around the question, not to have a single right answer.

Q: I am trying to take the same approach online. But we are having difficulty with students response to this approach. These are a mixed mature student group from ten different countries and significant portion ask “where is the lecture?!” Have you had this?
A: we have had some responses like that. We ask students in teed back surveys about these formats versus other classes, about what works and what doesn’t. Overwhelmingly they embrace it and write very nuanced responses. 85-90% like it! a few are neutral! and a tiny but very civically core don’t like it. For them though it’s about explaining why, the evidence base for this approach. I don’t see it much in my class but in the literature there are reports that students think it’s instructors being lazy. Not true of course, it is more work and you absolutely have to be on top of your game. But we do have the issue of students being very conservative. Every year we have a few students who only want to study like school. Don’t want to do coursework, workshops etc. exceedingly risky for them. So you have to convince the students to engage here. One thing I do in lectures is ask students to speak to someone who doesn’t agree with them – little happens, then I say “and if you can’t find anyone I will talk to you” – and that does the trick!

ePortfolios, ACJ and reflections – David Pier CMVM

Our programme, ChM are surgical programmes devolved with the royal college of surgeons. So we will be talking about process we use at the milestone between specialist training and practice. W try to get our surgical trainees, who have been in raining for many years, to go from “how could you treat this condition” and instead to weigh up evidence based approach to “how will you treat this patient?”. These guys have a lot of core knowledge, but we are looking at the application of this knowledge.

And these guys have lots of surgical retaining but they may not have had any research training, to assess that evidence. They will have some skills but we have this academic skills module looking at evidence based practice in surgery, finding the evidence, assessing ones own practice and implementing change, critical appraisal, non-technical skills. We really want them to think how they as a leader can impact how things happens. We teach some of these skills through information, particularly through discussion boards but we also wanted to ensure there was assessment. And that assessment had to be e,needed in their day to day work, be real vent, and be based on wreak every day cases. So we came to the conclusion that we wanted to use a reflective eportfolio – which would be very well aligned to the types of portfolios used professionally. So these could see something in theatre, reflect upon it. We wanted to be as reflective as possible. Students can upload thoughts to VLE into a private area. We had some requirements but the main thing was to get thoughts down as they happen, to put as much in as possible.

What we hoped was that they would capture lots of events. They witness a huge number of events, they are in surgery every day. And they may see a different or new technique or practice or experience. Then we wanted to reflect on this event they had recorded. Then to look for the evidence, see how that relates. And hopefully that will lead to a set of objectives… Which may just be about doing more reading in the area…. And then there may be some follow up, some reflection back to those objectives. This is a two or three semester assessment. And we hoped that actually this could go further… There might be a gap in the evidence… Perhaps you design your own scientific study for instance… Could potentially use this inn the research project at the end of the programme. It has happened to some extent but not quite all the way through.

So for the reflective eportfolio we wanted to make it organic but there had to be some sort of structure. So we gave five categories to use – can be swapped around – but gives structure. These are: quality improvement and patient care, research and experimental design, teaching skills, self-learning, and ?)

So we asked students to go and do this. Some took it up but some were very skeptical. The personal tutor system has helped a lot here. When you tell them that you are already doing all of these processes – this is just a structure to use – that does click with most of them.

We obviously have to mark this, and we have six marking categories and they don’t match one to one. Self disclosure, critical analysis, evidence based analysis, learning objectives, teaching and learning, research principles. These are based on gmc guidance, experience from Undergraduate portfolios, form the evidence. We mark across all of these.

So, in the first hear that we did this we had some initial confusion. This was around the time the Adaptive Comparative Judgement (ACJ) tool appeared – a tool for comparing a pair of pieces of work (and selecting which is better). and we thought this would be brilliant for peer assessment, to leave comments, to find how their work fits with others, to understand the best one. The rank order is the only measure that comes out of the system. There was as slight hiccough. We wanted to students to mark all the examples. And we wanted tutors to mark them all as well to compare. But it didn’t quite work – both sets of marks were combined together. But this did help students look at lots of examples without having to deal with marking criteria. Students mostly found it useful (ranked it ok to very useful). About 70% of students tried it out. Those students did seem to do better in summative assessment. Now they are self selecting so that might be a bit biased, but it does seem to help.

The second time around we made some changes, we used exemplars from the first year, we did some peer assessment and asked students to assign a grade, and let them know our grade. And where a notable gap – where students don’t quite get it – between those we have been able to offer extra support. In the past we have had students concerned or not understanding where their marks sit. But that peer assessment has helped a lot so they can see how they sit against others. Students have questioned marks to tutors also mark and often that is very similar, meaning students are more accepting of the peer marks. We did reduce the narratives required… Students hadn’t quite understood that. And gave a word count. Posts got wordy… So having to reedit strange themed the writing and kept posts concise.

Largely students enjoy this. Even those that don’t love it accept that they need to do this as a consultant and do find it valuable. And many are now using in their own clinical practice building on this experience.

And just to end some acknowledgements and thanks to Paula Smith, helen Cameron, Ewan Harrison.

Q&A

Q: how do you manage to keep it authentic from students point of view. Student reflections from students point of view, and personal take. Does looking at each other’s work mean mastering the art of producing outcomes that perform well but are not authentic.
A: interesting question. But there is a right way to do this. We want students to take this and becomes. Better student. Maybe they do a presentation, and see audience not engaging, so want to make it better. Students commenting May question the evidence, suggest ways to do that… All of the experiences they have are relevant. You just try to focus them on particular key issues and find places for improvement. The experiences a re what they are finding day to day. Very authentic in that sense. But there is a system to help them go through, to see progression.

Digital vs “real” – Lindy Richardson, ECA
I am from edinburgh college art so we’re are teaching students about design. I wanted to talk about the difference between learning about design from real experience, or learning through a screen. My students would love to do everything through the screen!

I want to give you three examples we have given to our students to balance the experience of real design and the virtual elements of design,

So the first project was called THE CAST. We wanted to make sure that students engaged with real materials. They could bring smart phones, cameras, but also sketchbooks and materials to record the experience. So we went on a bus trip to a derelict modernist building. Full of beautiful tactile experiences. One of the problems with technology is that we don’t have that haptic technology yet!

So this building is concrete, it has burnt wood, it has graffiti, it has moss growing in it. It’s fantastic. And we asked them to record the experience, and not just with their camera – so taking sketches, rubbing, touching things, smelling things, noting what they might do with materials back in the studio. And then we mixed students up – different courses and differnt disciplines, all I’m workshops… This confused them a bit! They are printing in the fabric studio, they are making fabric formed concrete, working with hot glass, and engaging with and touching and examining these experiments. Sharing all this stuff too – they all want to prove they did this so they update Facebook or twitter. But they were physically together and talking and collaborating. It was so exciting. And it was wonderful for the staff – to have people in our department who had no idea what then are doing but up for experimenting. So they made 3d bags out of concrete. The materials were informing the design. The materials led here, and non soecialists pushing innovation through challenging preconceived expectations. Can dividing the tasks to experts inhibit what takes place?

So we have images here… Glass burning into the textile. Playful experiments scarring the concrete. And that’s brilliant. The expert wouldn’t have thought of that!

But let’s bring this back… Students have to be ready for the real world. So we get them in the studio designing repeating wallpaper… Create handrawn motifs, full scale designs in repeat manually using photocopies and drawing. Then photoshop workshops in repeats. Then work on colour separation.

What did students learn here? They learned how to create a half drop by hand which really helped me to understand the process before learning how to do it in photoshop – where the repeats can get very square.

Another learned how to make a half drop repeat by hand and it was less manageable than by digital means… Their finished wallpaper may not have looked as strong but the evidence was very strong…

And another found both new and found both helpful – and evidenced it well.

The thing here is that we have all these students with different learning styles. And it’s so important to understand the colour separation process, and what can go wrong. Ding that be hand makes a huge difference.

So we get students to manually and digitally create prints. Photoshop can really lack fluidity, but with experience of the manual process the digital pieces can end up more fluid…

Our students are amazing with their thumbs! They are skilled in some ways to we are loosing traditional skills as well. I am very conscious that I go to teach a technique. They will youtube it, try it once, and then never again. We don’t perfect, or get the nuances. W miss out because of that screen.

So… The 45 bus route project. Students had to travel the whole route and to work as a group… We’ve had a chat about group work already. It can cause a lot of friction… Students try to get out of it with Facebook, Instagram. Snapchat, email, blogging, phoning… But face to face interaction championed in the end. One made bread, one made jam. They met to make and do and prepare presentation. The blog is brilliant, I’ll make sure that’s shared with Wilma to pass on!

Now we’ve all been sitting paying attention to the front… I want a little hands on authentic experience, to chat with each other as they do this…. Hopefully you can chat to your neighbours. So I will teach you all to finger knit! My attic is full of fabric and wool and things! So I have wound up a ball of wool for each of you. Take one and we’ll all try it….


E is for experience – rob thomas

I think I am the case study here! I am relatively new to the full time academic world… When I first came to the university is was introduced to this term “eLearning” and I’m not entirely sure I understand what eLearning actually means yet. E, I think, is for experience. Everything else I think adds values to that experience, and adds value to that experience. In my trying and career I never went to unversity. I did my first degree with the open university in a pre digital age, everything was handwritten including the feedback. Very positive. But the highlight for me was the week long summer school where you had an opportunity to reconnect with reality, with yourself… And got a chance to play with things. It was an incredibly important moment. And I think it’s something that could be missed in eLearning, a critical element that cannot take place electronically. (Rob notes that he’s still attached to his finger knitting. It’s adding some swagger to his style though! )

So… Looking at a milk carton that has beef cows on it… It wasn’t authentic… It hadn’t been checked. There was an issue of credibility here… And looking at the notorious “bingo!” Poster after the budget there was a significant error of judgement there. A real issue of authenticity…

From my experience outside of academia much or learning, training and assessing is a person mediated process. We don’t learn from digital materials.

Whether organisational learning, individual in structure or evaluation, the central mode is the direct experience of those involved. Organisational learning can be a process… But organisations often don’t learn, they end up repeating mistakes in a loop.

Digital tools aid communication and information, to most learning is through doing. They are means to process and manage information. The learning is about the physical or the behavioural doing.

Life is authenticated by the self. Experience is self authenticating. If wea re caught up in rights and wrongs and assessment, we continue to believe in what someone tells us. There is a disconnect… When that person leaves academia that person needs to be able to understand their own authentic experience. It’s a very sensitive idea… Authenticity is spatial and temporal here. Things change.

I mostly teach online… We try to make it organic, to make them think of the content to an extent. Ideally when a course has been taught, the materials would self destruct. Don’t keep the traces (other than for the external examiner). We should be forced to rethink all over again. Just because we think we have authenticated something once, doesn’t mean it’s still useful one year on…. Can be so far from meaningfulness and relevant. Bit of a hangover from powerpoints and lecture notes. Should be forced to start again.

Out in the real world there is work experience. Some universities and colleges do sandwich courses (just 9.5% in 2002-3, 7.2% in 2009-10 of full time cohorts did this) And the Wilson Review found there were huge advantages. But it’s hard work to provide that experience, for industry, for employers. So it’s hard to do but gang disconnect is really important. So we have to to think about how we can better prepare students in vocational courses to be empowered to understand the workplace, the subject, and to learn about organisations. And those soft bits, how to work with people. Group work for an hour is fine but working with a diverse group for a year is a very different beast in terms of what you learn, how you relate content and how you project yourself in a team environment.

I think the general approach that didactic lectures is dead. Like online material can just kill it. And there is also an opportunity for students to step up and express an opinion, and encourage students to do this. In the real world that is how people learn. This is the medium by which information is shared, or dissected, or understood. The flipped classroom is effectively the way we learn in industry. Here we are implying this is a new concept, been going on for centuries in the outside world…

So individuals learn from colleagues, from mistakes. I think we need to allow people to make mistakes. To move away from grading and negative impacts of mistakes. To include training. To experience the outcomes of organisational learning. Organisations learn from consequences…

However lessons learned belong to the organisation but are held by the individual… And individuals leave… Which is why organisations end up repeating mistakes…

Blended learning tends to be the model for industry. Grown organically. Organisations have employed blended learning for some significant time.

The traditional project management cycle are: problem, design, implement, monitor, evaluate, adjust and go back to the problem. A cycle. But you can add in “innovate” between monitor and evaluate. That can feed up to a “new problem”, and some “abandon” at the point of evaluation – a cycle within the cycle essentially. Industry tends to abandon unproductive activity. And abandon unuseful problems or ideas when no longer valued. A speedy way to work.

A couple of examples… Logical framework (log frame): projects have objectives, they have means of verification, imdicators, assumptions, outputs and lessons. Those assumptions could be things we. Know but students don’t. Or assumptions students make that need to be addressed. And the important part here is how we learn from those lessons. So many discoveries that can be used.

When I joined the university last year the were performance and development reviews… A system that evaluates and makes you accountable… This seemed the norm for me so I didn’t understand some colleagues reticence. This is a key human resource management tool, it’s for your benefit to exploit…

So I will leave you with a T.S. Elliot quote “where is the wisdom we have lost in knowledge? Where is the knowledge we have lost in information”.

Q&A

Q: why is abandonment different from adjustment
A: adjustment is you still working towards the same goal. Abandonment means a whole new goal is persued!

Networked Scholars and Authentic Influence? – Bonnie Stewart, University of Prince Edward Island
It is a big honour to be at the university of edinburgh, particularly because the university’s reputation for digital education is well known in my field. But also because my name is Stewart so it’s lovely to be in Scotland!

Now there is a question mark in my title to raise this as a question. How many in the room use twitter? (Quite a lot of us do). Now ther at cliches that circulate. But there are ways to use twitter beyond celebrity. My work looks at twitter and scholars. And I think twitter is a space for networked scholarship for us as scholars. With pedagogies… But what does it mean to be authentic in these networks, particularly in the scholarly ways.

The question of influence is a complex equation. Traditionally in institutional worlds is that teeny group that understands your work in that instiution, and we have outsiders, external markers, or the journal you publish in… Where you went to school, the last grant you had… And all of a sudden things like twitter come into that mix. Both concepts centre around reputation, albeit in very different ways. When you are present and active in spaces like twitter you are creating an identity position.

Willinsky 2010 says that in an academic world scholars are taught to understand reputation with some subtlty and depth.

Now, what am I doing here? In another country? Well I’m a graduate student. I have twenty years experience as an educator, early experience with MOOCs. But my twitter profile is probably a major reason I am here. This is a parallel identity really…

So I want to talk about authenticity in networked scholarship and how you perceive it in the world you live in. These people on screen are people I work with, whose books I read, who read my blog… They are the public sphere in which I speak and build my reputation, and I am part of how they build theirs.

Online networks enable different forms of identity legitimacy, and authenticity. There is the scholarly world, and the what people ate for lunch on twitter world. I study the overlap, the place where higher education is changing.

The fire hydrant is a great metaphor for information. There is abundance. We have moved from paper texts to a world of persistent, replicatable, abundance of information. How many of you teach? (Many of us), how many of you let students have devices out in class? (A lot of us). That’s happily higher than I sometimes expect. Our students can have Wikipedia at hand with more information than we could ever have.

And we have a real changing educational culture. Public and institutional values have been changing. Public values have moved to a more market value or vision of the university. It’s a messy mix and intersection of open and closed systems, of knowledge security and knowledge abundance…
And there is increasing pressure to go online – to engage with the terrible MOOC monster!

Within this networks are one way in which the channels of abundance can be managed. It is hard to try to take everything in from the fire hydrant of information. If we don’t have ways to structure and understand that information we will quickly be overwhelmed. Traditionally we had gatekeepers to knowledge based around institutions. They remain useful but many are not within those spaces, many are not allowed to speak in those spaces. So many use these open online channels.

Networks are not just online or offline. Not binaries here. If you have families you will have complex and different relationships with each individuals. Networks operate in the same way. We already have networks and literalise for dealing with them. T our institutions do not have ore existing literalise to deal with them. See yesterday’s LSE blog headline for instance – about the lack of reward structures within the institutions for public engagement. And my work looks at this as a matter of literacies.

So if we. See the marketoonist.com social network adoption cartoon for something on this,networks require time to understand what counts, what’s useful. In order to succeed in networks the price of admission is that you have to create a public identity. If you don’t have that centre to connect to, people cannot connect. That public identity can be confusing. It’s not about the tech or getting stuff online, its about building a different identity, creating those ways of being and of building relationships. Networked identities are multiplicitous and faceted.

I’m conducting a small ethnography right now. I have fourteen participants and eight exemplars who have agreed to let Bonnie show their profile to others to ask questions. And I did three months of participant observation on twitter and blogs and ten interviews about how they make sense of their networked participation.

The classic media story is about people reading more of your stuff – see pat thompson and Inger from Thesis whisperer’s recent paper – but it’s not just about dissemination.

I have three junior scholars or ohd students here, they don’t have big voices in their institution. They have used blogs and twitter to establish a presence, to share career and academic challenges.

If you see someone on twitter and they are quite formal and only talking about their work, and they are probably quite new. That’s not how we chat. We talk about other stuff – sushi and cake, and aren’t those nice boots. Ambient relationality between people. And twitter allows people to speak back to academia and to speak from the margins of academia. Whilst our academic policies are changing you can still be the only disabled person in your department , or the only queer person, or the only person of colour… An connecting to peers elsewhere has real value there…

Thomas Friedman wrote in the New York Times lamenting the lack off public scholars. He often does. People spoke back on twitter to point out that they are there, they just may not be of the lofty stature to get from attention… And twitter gives you a voice out, and from further afield…

But…

The is real concern that institutions will try to control and contain these activity. The university of Kansas put in a policy to contain what they say on twitter, regardless of academic freedom, regardless of tenure.

And Alice tiara, a peer of danah boyd, feels that the more quoted or well known she is, the less candid she can be…

And there is a big signal to noise filters. I get ten or fifteen articles each week that look amazing on twitter. But if I expand them all I could never finish my research!

And positioning fatigue can be time consuming, complex, and really problematic. And sometimes identity can seem to get in the way of each other…

And sometimes immersion can be required. Tweets from a conference talking about how it’s hard to “get” twitter without full immersion in the space, in those relationships.

In terms of social media being a signal that is fully immersed… Well it’s not happening and might not ever. And I’m trying to see how to signals can come in, so I’m looking at literacies for understanding academic networked publics. Institutions tend to be product focused, about mastery, bounded by time/space, hierarchical ties, plagiarism, authority in role – and everyone knows what that role signifies. In the public sphere of networks the individuals actual works factors much more than your role. For the institution the audience is the institution and the academy. In the public social sphere the audience can be the world…

So, authenticity. There is a lot that circulates in public networks around influence there is snake oil. We have yo be wary. And the word “authentic” can be dangerous in digital contexts. For many people authentic means real, and not on the computer. And even for those of us looking at what it means to do authentic digital world, the word authentic suggests a binary. So what is synthetic work online? What does it mean to do synthetic education? We all encounter the cultural narratives where teaching online is perceived as synthetic in and of itself. Keeping those naming a and binaries open is important.

One of the keys of being authentic online for me, is showing your work. Showing the logic of where you got, and how you got there, and citing as you go, and credits ideas, then you are more likely to be taken up as authentic. Even if you are blogging you may want traditional citations. You should, credit a conversation on twitter that triggered the thought. In networks we also need something to fasten onto. Transparency is key.

Metrics. The numbers that track your performance and participation. Tweets, followers, etc…. They are not meaningful. You can buy followers. Not common in education… I used to circulate in the blogging world and I’d meet bloggers who started six months previously with 60k followers… Signalled that’s what they were into.

Now I will show some exemplars. David White (@daveowhite) people looked at his profile and thought maybe it didn’t look exciting, but saw he was at Oxford, so they thought maybe they should follow him. But he tries to show more in his profile. He signals a joke in his bio – “the “o” is hitchcockian” (a North by northwest reference). This is playful. He’s doing visible identity work. As individuals we are really not used to doing visible identity work beyond the age of 24 really. You have to in networks. It takes courage to do that.

Looking at Audrey Watters profile! she does lots of identity work. She has a playful picture obscuring her face a bit. She links to her site, she calls herself an author – traditional credibility. And the big thing people note is her 21k followers. What that means, particularly when she follows fewer people… And note she follows almost 2000 people, she’s not broadcasting and following no one, she is networking. And she tweets a lot. About 21k tweets. She’s contributing. She has credibility… Now the numbers don’t tell us if she is entertaining or if it’s the quality of her work. I would guess both. But those number lend credibility.

Now Valerie Lopes at university of Toronto has a simple profile and bio, no background picture. Not massive follower count, fairly equal number of followers and people she follows. Her focus seems to be on her main role to she tweets a lot and has credible. Audrey doesn’t have an institution, she’s a freelance journalist. And dave uses twitter as a parallel research space to his institutional space.

If you see someone following a lot of people, tweeting a lot, pushing out stuff about products or projects…

At NLC 2014 Terry-Lynn talked about “technologies alone are not going to create mobile practices. Fluencies of navigating scale, negotiating openness, way finding, and curation”. The more you leave traces of your work and connections, the more you make sense of it and show credibility. Curation also matters. If you get overwhelmed by twitter, shut it down for a bit, come back to it and them manage it.

Maha Bali just published Bonds of difference: illusions of inclusion – hybrid pedagogy. She is in Egypt and I met her in a MOOC, a group that came together really in Facebook. It feels likea. Global conversation. T in this article she and an author from, I think, India say that yes, this is global and super but we really still need to think about the power relations here. Networks open up the power relations of the institutions. Hunt networks can continue power relations of sexism, racism, ableism. We always have to think about the power relations and who has a voice in networks.

I won’t make you use networks but we need to learn and keep learning to read networks, institutions need to keep learning, to understand what is authentic contribution, and what may not be.

Q&A

Q: I was interested in your slide with a sort of binary between the institution and the individual. And I’m thinking about staff having issues with negotiating that space, navigating having a public role in the networked sphere. And the balancing act for supporting students to balance that sphere. Do you have questions from your research to negotiate an area where we ourselves may lack expertise?
A: those institutional and network literacies are binaries in a way but they already interact. I think we still need institutional literacies, that’s really a space where we need reflection. A way to see what we already do and not leave those behind, but to learn new literacies as well. And in power terms institutions have a crucial role in keeping vibrant spaces for education. But we have to recognise that institutions adna be hide bound…. I’d like to think of ourselves as almost code switching – knowing when to call on the reserves and power of the institutions, and when to look out to the public sphere for what it does well. I would be concerned to see us only on one side of that screen. For my teaching – bachelors of ed students online, and offline but mostly blended. Sometimes my students are very institutionalised and concerned with moving beyond the social contract. Sometimes they need me to be fully literate in what the institution can provide to support. Sometimes they need me to be literate in networks and what that can offer them professionally. Depends so much on I divas students or groups of students and how they see their role.

Q: I’m sure many of us are guilty of saying what we do on our twitter profiles where we say what we do then adding “views are my own” for a mixed personal and professional account…
A: it can be difficult to have multiple accounts. Depends partly on an individuals role and status within the institution is. Avid what can be shared. I do not have my institutional role on my twitter profile. When I started on twitter I was on maternity leave. I have changed images but not text (much). I basically trade that freedom for the lack of credibility or affiliation. I’m ok with that trade off. Most do have their institution on their profile. If you do that… If you know you want to say things that are possibly unregularly and your institution may not want to hear… You might want to take it off. If you are having general conversations about eLearning that’s probably fine. If you want a presence to speak truth to power from a vulnerable position you have to remember that is persistent, that is replicatable that is public, you have to put your paycheck behind it. But this is why Kansas is so concerning, this idea of full overview of all accounts. If there is viewing of everything that public sphere becomes constrained. And we may see more pseudonymous accounts. And those aren’t anonymous, they are trustworthy and trusted identities… So you could be that person and build relationships without knowing your real name, recognising what you have to say may require protection.

Q: does @bonstewart resemble Bonnie Stewart and how does that work/change?
A: I think @bonstewart may be a better representation of me. In class my students don’t see a full rounded picture of me as a person, we are pushed for time. Following someone on twitter you can get bigger broader ideas of the person. For me what you see if what you get, especially online.


The student experience – Alex munyard

This year I have convened a short arts group or look at how we can maximise the open online resources online, within the university and also in a sort of global academic community. There are also questions about who will use this, just academics and students or more diverse audiences. So there is a real opportunity to become an open access leader.

So what is opening up lectures live? How does the open educational resources work? It means opening up lectures, slides, syllabi, the materials we share with students. Somewhat along the MIT open courseware model but more slickly, and justified on pedagogical grounds. There are good reasons to open up materials across the university – for revision, for learning, for developing ideas. If the university of edinburgh strives to be a global university they have to make moves to prove that. The idea that universities are global public goods, and I think this is something OER can be A hugely important pat of this. Students can record their own lectures but doing this across the university will assist those with students for whom English is a second language, to maintain quality. And the idea that you wouldn’t go to class if the video were available is just wrong, the evidence shows that students do go to class. But we need to use tech to stretch learning, not just using tech for its own sake. Talking out to a room of students isn’t pedagogically justified, just what we’ve done for centuries.

In terms of lecture recording… ELearning represents a real opportunity to make education more accessible, particularly for those with disability. Offer greater efficacy for tapping into students with diverse needs or interests. And a real benefit for international students. When more staff online and more resources available, there is a greater onus on staff to think about how they put material together. I guess to reiterate my core premise, technology should only be researched, invested in, when proven, and when there is pedagogic rationale. To it should in no way limit playfulness or creativity or risk taking.

So to conclude let’s innovate and embrace change. Predominant age old teaching methods need to change, we need to overcome traditional views of that.

Q&A

Q: at the risk of being slightly unfair… If we go down the oath of not only recording lectures, but also massively open course ware… Where is the value in paying your fees, coming to edinburgh, finding a flat… What’s the authentic experience there
A: I think the benefits of OER are multiple faceted. Firstly they would benefit in person students preparing to attend, selecting their course, and engaging in interdisciplinary work. And the wider audiences won’t become students here. MOOC takers are using resources not students at a edinburgh university. So that model can accommodate that idea.

Q: interesting initiative. Do you think full time students will be able to take advantage of having resources from other courses. They are time pressed so will they actually devote their time to that?
A: I think different students will have different focuses. A physics student might use mathematics resources to support that. Science students passionate about the arts may want to engage. But some will not want to engage in other subjects, but I think seeing and engaging with different theories of pedagogies could be very beneficial.

Q: my concern would be huge materials available without support… How would that work? Particularly if they felt they had expertise from doing or engaging with that material without support.
A: well I think not everyone in the University will take that course. It would need to be presented appropriately. But I think if we went down this route there would need to be massive support for staff to make this martial available. I don’t think that students would be graduating from every subject. But something that let’s students have a more rounded and holistic experience. If we make degrees better for employability… A holistic degree should equip students for the world. I’m not claiming that this agenda will make students in every subject… But that space for exploration can only enhance opportunity.

And now we have two videos from students from the MSc in a digital education

Authentic online learning – Ed Guzman and Anna Wood

Ed’s video:
https://vimeo.com/90484211

Anna’s video:
http://learningfrome-learning.blogspot.com/2014/04/authentic-online-learning.html

Technologies and collaborative learning – rubie rennie and students

Jingao
Technology offers lots of opportunities for scaffolding (gibbons) for students in the zone of proximal development (vygotsky 1978). So for instance collaborative opportunities include teaching language and vocabulary around animals by engaging with authentic virtual animals in second life.

And technologies, particularly web 2 (mak and coniam 2008) enable comment, feedback, peer support, and feedback that is rapid and regular, not just from the tutor and not just at the end of the course. For instance through reflective blogs.

And technologies promote collaborative learning by creating environments wher learners can change the social context cues (Ortega 1997) that may be problematic or inhibit the,. They can play with gender, even present themselves as animals.

Technology offers many opportunities for collaborative learning and when I become an English teacher in china I plan to make use of them!

Victoria
I have already been doing some online teaching in china? There are two ways this tends to be done. One is video recording lectures, replicating the lecture experience, not really eLearning really. The other way is using chat software or teaching software to teach students, whether commercial software or sns and chatting software. A word about the commercial software… There is potential value here for us to do this. There are huge quantities of people online and using online courses. But there are softwares we are already using… QQ, YY chat, wei Bo, sian UCAS, Wei chat, Skype. But there are disadvantages… Of seeing each other, not very secure… But these are very commonly used, very flexible, vary familiar, and in Chinese context we could easily use these software.

I want to talk about the main problems in my teaching process… The infrastructure needs to be considered. And we have to understand the students use of the internet. And the use of the technology should consider the context. Students used to technology can easily use it to achieve their gaols. For those less familiar with technology the class can be a much bigger challenge. Also worth noting that in china it is not usual to use email or blackboard to access information, and there is room to develop here. But we have some restrictions. We can’t use youtube or twitter or Facebook in China. Can’t share that. Can’t share experience of using them. Here we could share that experience and those learning materials with classmates, tutors, supervisors etc. so we need to think about that context carefully when we think about our students. Infrastructure is part of that too. There aren’t computers or internet access everywhere, sometimes videos could be downloaded and shared with a class, say.

Ivy

I’m going to talk about my online learning experience in china, at high school . But in china we cannot use computers or mobile devices in school. But one teacher did set up a course on non Kent Chinese literature via blogs, for us to access after school from home. We hadn’t studied this topic before and was really exciting. And it could help students to create their identities. School life can be very sessful and very separate from daily lives. But online courses can let us reshape our knowledge and identities through learning online.

After we entered university we had more opportunity to access online courses. But these courses are videos. Students can only watch the videos and not directly contact the tutors… And there are some people who came up with the idea of a video chatting course using taobao.com, the biggest shopping network online in china. So teachers sell courses in their taobao shop, and students can talk to teachers via Skype, FaceTime, or QQ. Helpful as some students in remote areas of china don’t have access to many learning resources. And moreover these courses are very flexible, students can negotiate with teachers. And it’s closest to face to face interactions.

sheng

I became a student when I was 6 years old, and I have been exposed to traditional Chinese pedagogic models for 17 years. So Herrington 2006 really struck a chord. We need something exciting and interesting and new to engage us. So I chose online learning as an additional course.

When I was an undergraduate student of English we did have an online learning space… It did have lots of functionality but we barely used it. So when I first got here I struggled with the ideal of communicating with staff via emails, how to use learn/blackboard. I think that online students are more likely to be a let To access good authentic materials to explore as language learning.

With online technologies tasks can be set to be authentic experience. Learning in second life enables students across the world to share in a learning event. It provides authentic opportunity for English learners to meet and learn. The authenticity it provides is bette than any other computer mediated communication tools. In techno life seminars, interviews, presentations can be simulated. Presenting in techno life is pretty similar to doing that in real life. And second life is in china, as are other technologies like this, but many teachers are not realising the benefits yet so when I go back to teach I hope to do that!

Q&A

Q: when you showed the screen of the learning environment I had a sudden shift in perspective about what our VLEs look like for our Chinese students. I can’t read the symbols at all and obviously my students do know the language. But it was a real shift in perspective was really interesting. Do you think the challenges can be overcome?
A: I think it’s good to explore tools like second life etc. but for learners especially at younger ages, we have to provide support and explain these environments to the students so that they can cope….

Q: do you think checking all the blogs that students might come up with will take more time for the tutor?
A: I think making a blog is a new experience for a student. There are disadvantages to using blogs but lots of advantages too. And I think that for me everyday I use ten or twenty minutes for a app that could be spent on a blog. It’s just a way to learn and I think we just compare disadvantages and advantages.

A: it is true that at the beginning of using something you haven’t used before it can take some time. When ruby first introduced us to second life I found it hard to find where our meeting was… So I asked coursemates. I thought this is a waste of time but as. Got more used to it it becomes more useful and valuable.

Making it real: authentic teachers online Daphne Loads, IAD
I wanted to talk about authentic teachers…. We have talked about authentic tasks, authentic assessment…not sure we’ve heard authentic teacher yet…

So let’s talk about what an authentic teacher may be, particularly online? Snuggest ions here include a catalyst for learning, someone who brings their own perspective and personality to learning, and someone who continues to learn.

So here’s my thinking… I’ve been a teacher for a long long time… Ie tried to be an authentic teacher for a long time, now trying to learn what it is to be an authentic teacher online… I’d like to take you through my thinking about being an authentic teacher…. About artefact… Some think a crown or jewels, something precious, something old, something that shouldn’t be there that is produced in the profile, hammers and spanners and tools.

Well often people talk about something made by a human hands, or art. (Seeing image of the tenth muse/Sappho, a relatively modern sculpture in Jupiter art land. Something invested with human meaning and that sets up human exchanges. And might be something historical, telling us something about the humans that created it.

Or it tends to be some sort of evidence of something… Something that reminds us of human error or weakness – for instance an X-ray with a. Shadow that is actually created by a braid of hair…

And the other response is just “a thing”!

So saying that an artefact is something made by human beings that tells us about being human, an object.

Teaching is an artefact. There was a time when saying teaching was almost a dirty word, needed to talk about learners and learning. So teaching is an artefact, something made by someone human, their creativity and perspective but also errors and issues from being created by a human.

I think as a teachers learning and teaching online I try to be genuine. But it comes and goes. Maybe I want to reach out to you. But sometimes I just want to run away home. But most of the time I am trying to be genuine, to share myself. Sometimes sharing my humanity brings something valuable to my teaching.

If you want an example see prof al Phil rice of Philadelphia university, his MOOC on modern American poetry. It was a series of interactive, engaged conversations with (graduate) students that gave me a real sense of what it was to engage with American poetry… Tackling difficult stuff, Gertrude stein, because of his authentic engagement. I got a bit of an unlikely crush on him because of his humanness… Carl Rogers said that one of the things that happens is that if we use our humanity we have an authentic feeling that we can use. And my being authentic tic can draw out the humanity and authenticity of students.

Parker Palmer talks about there being something important about being human and making a mistake… Look at what happens in the room… If students are ok calling it out, and the teacher acknowledges and discusses that then real learning is taking place, it’s subject centred learning…

Some yes sharing my humanity or our humanity can get in the way of teaching…

For instance I was learning to use illuminate live… The instructor I could see him, he couldn’t see me. I had to press happy or confused… I could press either… (Ace this wasn’t at this institution) and that was my choice as a learner…. There was no “shut up and let me do something button”. Or humanity is not always the nice bits, the warmth, the inclusion… Sometimes it’s the wanting to just tell you everything you know… He’d given me this unhelpful little piece of autonomy. Not good.

Another example, more complicated. A friend in another institution teaches on the sociology of pain… She was teaching online… Had built up good rapport with students… And she wanted to talk about her experience of childbirth… Something strange happened… Students stopped calling her “doctor” and started to call her “mrs”. A shift in the relationship… They had the ridiculous idea that someone who had had children could not be an academic expert. Now I think my friend should have come back at that with the sociology of power but I think she was too taken aback by the reponse…

And then we have the issue of not accommodating the context. When I first started teaching large groups it was after teaching small groups. I wanted to make eye contact… And I found myself running around the room like a demented chat show host! So I had to adapt… Getting them to talk to each other, to write notes to each other, things like that…

Recently I did a talk on collaborate and couldn’t see the students… And I found the silence really disconcerting… A colleague said “well that’s easy, get them to vote every two minutes”! Not being aware of the context, that got in the way of learning a little bit for me…

So I think authentic teachers online are not people who tell you absolutely everything, or disclosing everything…. But about making careful judgement about when that precious artefact of their humanity. So I think authentic teachers online… Are aware of their humanity and making careful judgements about how much of their humanity me to share. And those themes came out of others talks for me today!

Authentic Information – what can analytics tell us? Anne-Marie Scott, Information Services TELS
I wanted to share with you today some reflections looking at some of the analytics we can get from our technology enhanced learning contexts. I can’t thank daphne enough for setting me up well for this. My background, like Daphne’s, is in literature and Scottish literature… Thinking about moral fables like the cock and the jasp, a chicken who finds a jewel and throws it away… But it is symbolic of knowledge and and nature and of not valuing this precious thing… And I think that can be a lens for analytics…

I’ve been digging through some of our data… Kind of two facets… Learning analytics (personal use of data) and educational analytics (institutional data) and a lot of that work so far has been about figuring out what the heck this stuff is!

So an example from one of the medical VLEs (heat map of activity). This is based on some excellent work our team in MVM have been doing on analytics and student engagement… This is data on how and when students engage. One works mainly towards the end of the evening, the other works intensely early in the morning… But no value to bring to this without understanding the student, the stuff that is not in the analytics. This is a real fast and frugal sort of measure!

The next piece is data from inside our VLEs… Which tools two schools within the same college use… And here we have school A and school B… It looks really very different… Both schools about 70% of courses use learn, pretty high level. So how would this compare with obligatory online medical msc vle usage. Would logins be a good proxy for engagement Less pretty patterns here… Logins turn out not that useful…

So I did the same pattern checking for the school with all the use of social tools… Same patterns… But actually it turns out they don’t use these tools… Discussion boards are in the default template so every course has it, but they are not being used…. But it’s no proxy for understanding what is happening… There is useful stuff here but context and interpretation is everything… The machine can’t do this for us…

Great article – learning analytics: the new black (booth 2012) talks about learning analytics risking becoming reductionist approach for measuring “a bunch mod stuff that doesn’t matter”.

And I’m just beginning this work… We have some work this year and next year… So this year we are looking at what so of analytics might be useful in the VLE, gathering requirements… Maybe trialling MVM approach in moodle. Bet also how to find bette management information, quantifying the data available, seeking feedback. Sense making activity. Next year we have some funded resource available to make some of this happen… Developing tools to better expose data within the VLEs. Developing reporting on our use of central eLearning services. We want to particularly highlight to schools what is happening in their local context and to continue this work Ina. More engaging and inclusive way…

So, what can analytics tell us? Wella. Glib answer (a) quite a lot AND (b) absolutely nothing at all. Humans make these decisions, won’t be machines or business information systems that will make the difference here, context is everything.

Summary and close of formal presentations – Paul McLaughlin, eLearning@ed Forum Committee, School of Biological Sciences
I started this conference but it wasn’t my idea… We were trying to illustrate a concept to Ruby.. And came up with “authenticity”. But it was accidentally no great idea. Land I think it’s gone pretty well! I took note of some key themes I saw coming out today…

The idea of authenticity and messiness… Going against each other to an extent… Came up in Daphne and Ross and Lindys talk. And also we had the idea that our systems or structures can get in the way – in Inger and the business schools talk about cultures that exist… Also institutional failures – Bonnie highlighted failures to reward public scholarship and public engagement. And we haven’t come to terms with students making mistakes (as in robs talk) or teachers making mistakes in front of students (Daphne’s talk). Bonnie kind of talked about the dichotomy with real life. I think none of us will forget the highlight of the day, Lindy’s knitting stuff…. That physical learning matters. And then we saw David peer talking about authenticity and engagement in the portfolios. And I was amused by Micheal begs and the idea of prescribing under stress… The exam being stressful is probably authentic… But you could make it really authentic with switching between tasks – a plate spinning task! And I was impressed by the TESOL students form a moray house and their experiences of learning…

So we took an abstract idea and had everything from finger knitting to load balancing in the cloud! And we can reflect how lucky we are to have such diversity in the university!

Finally a note of thanks to all those who have organised today: Jessie Patterson, Jo Spillar, Marshall Dozier, Ruby Rennie, Sharon Boyd, and anyone else I may have missed!

And with that the talks are finished and we are off for refreshments and the poster sessions.

 April 10, 2014  Posted by at 9:55 am Events Attended, LiveBlogs Tagged with:  1 Response »
Apr 092014
 

After a day away at the Spatial Memories Design Workshop, I am back at Networked Learning 2014.

The spaces of networked learning
Sian Bayne is introducing the session by saying that spatial dimension have been raised at a number of sessions throughout the conference.

Richard Edwards

When we use the term “networked learning” are we being literal or metaphorically. What is the conceptual significance of the network in networked learning? For me network is inherently relational and therefore spatial. And that informs the we need to think about both networks and learning. Space is often treated like a black box, observed rather than analysed. The classroom, the blackboard… I think it is no coincidence that open learning is so central to networked learning but how do we frame this?

There is the concept of the framed economy of space, David Harvey et al. Focused on space divided by economic conditions, globalisation, urbanisation, inequalities. Then there’s Doreen massey’s feminist concept of space and of time, the gendered nature of such spaces and their orderings. There is the post structuralist idea of space, for instance Sian has used the open striated spaces. And then there is john murray and co’s socialist take on spaces. The power, the practice of associating and disassociating.

There are two particular things that interest me. One in relation to technology, the other our conception of learning. One is computer mediated. Codespace. The “hidden curriculum” of networked learning and digital education. When we think about spatially, we can decentralise the subject. That’s yet to really be teased out but very interesting.

Q&A

Q: what about pedagogues of networked learning?
A: one of things we. Have to think about in our research is to consider when we are using things as analytical concepts versus when we are using the. As pedagogues in practice. I’m interested in network spatial analytically. But one might want to think about them pedagogically, how we might do that in new and interested ways…

Q:
A: one of the things that’s such a Central concept is the nature of learning. Various ways to understand that. For me drawing on spatial theory, and indeed actor network theory, is thinking beyond ourselves and our cognitive engagement with learning, but of learning in relation to other people, other systems… But does that compromise the fundamental understanding of learning. I guess I’m trying to push at how far we can go with this. Assessment for instance… What does decentreing the subject mean for assessment when all of our existing practices are about centering the subject?

Q: if we decentre the subject aren’t we talking about resource management – assessing the environment and the subject…
A: yes in terms of ecology and sustainability in education
Q: so what competencies do we need to do that sort of thing?
A: competencies of relating to others and their environments.
Q: Why can’t an economist do this?
A: they can. But so can others. One of the issues of education is that we think too much about people detached from their context. Part of my work here is about understanding the nature of being human by understanding context, not to deny what it is to be human by denying that context.

MOOCs and spatial theory – jeremy Knox

I’m going to try and apply spatial theory to the MOOC. In particular talking about three different types of spaces: the global institution – a particular sort of space by coursera, Edex, udacity; the homely – the domestication of the global; the overwhelming – another way to think spatially about this problem, the global and the local coming together.

Sofi you look at how big organisations promote themselves, the image of the globe is really rather prominent. Edex shows the globe spinning and the nodes of a network forming in that. Significant because MOOC organisations appropriate the globe to symbolically talk about their reach, but also appropriating the idea of the network for what they want to do, to reach far corners of the world with education.

The other thing they do is have missions and visions. Udacity talk about a mission to bring accessible, adorable, engaging and highly effective higher education to the world. The key word is “bring”. And their publicity strongly suggests who they may be targeting. The MOOC companies both promote and explicitly are trying to reach developing world countries. At the coursera conference recently one of the dashboard metrics is percentage of student so rom the developing world.

I also wanted to talk about the colloquial sign up page features, so coursera talks about “Courserians”. So the way the organisations measure reach is sign ups to the platform, that’s how they measure how far they are getting across the globe. And coursera released a visualisation of how they think of themselves across the world. The image is of a globe and is generated by data. Data is something I haven’t heard much this week but it is significant and is something MOOCs are bringing into the education world. The shading of the coursera map effectively makes the world come into being by the uptake of the platform. And there is a significant difference between who delivers the education, and who it is delivered to.

So there is a dominant form of education here is of transmission. And it is transmission from the developed world – the US predominantly, Europe, Australia – and the developing world.

So those nodes delivering the courses… These are institutions with the buildings and architecture up front, used to legitimise the offering. Princeton, Harvard, etc. all present themselves in the MOOC via images of their physical campus, imposing buildings.

The form of transmission in MOOCs is largely video. Either head and shoulders videos, or to a room. I will come back to that…

So we have the idea of transmission in a very one way direction. But one interesting reaction and example against that concept is an institution considering the homely, and to domesticate what can be troubling about the global.

So this is al philright(?) who works at the Kelly writers house. This is where the MOOC is from. This is a significant presentation of the MOOc space. Al invites us in. He’s inviting us into the world of contemporary literary poetry world. It’s authentic. And you can go and visit. In that video you meet those working at that house. All of these people talk about the kitchen, the food, etc, and photographs of the community, this family who have visited the house are prominent. The videos are also notably al and some of his students discussing issues. He does most of the talking though.

And indeed some edinburgh students taking this MOOC tried to recreate the video in their own kitchen. It’s a domestic scene recreated in domestic ways. Students love the video tour, being able to see the place. And many ended their comments by saying “I wish one day I could make it”. Seems to be an acknowledgement that this won’t entirely work, the whole globe can’t visit.

And I wanted to move to the third kind of space. I hope I gotta cross transmission. And welcoming in. Simplistically these seem very separate, but how might we overcome this?

So this Part I’ve called the overwhelming… I am privileged to teach on an eLearning and digital cultures MOOC here. And we encouraged students to create digital artefacts. One of the prominent reactions to that was of overwhelming nene being overwhelmed, lots of somewhat negative comments there… But I wanted to flip that a bit. Some students responded visually to that chaos. We had a visual task in the MOOC, and we saw reimagined Tokyo tube maps, we see people falling through the rabbit hole. I think we saw are action to unfamiliar space – of neither dominant transmission not homeliness – in our MOOC.

But why is this interesting? Well seeing this student image of an overgrown house with the statement “yes, she’s home, she’s just a bit busy with edcmooc”. I like the idea of neither form dominant and structurally… The growth

Q&A

Q: on the buildings front… When you ask them institutions are looking for conversion rates… May want that presentation for that reason
A: certainly one reason for that presentation I think, others too… Not a significant driver for our MOOC, only a few sign ups from it.

Q: that first type of presentation is about imagining yourself there. That second type was about community… What emerged from your space?
A: I’m integer in that… Emerged from students, also from technologies… Interested in what came about spatially, rather than what we designed.
Comment from Christine sinclair: when we were approached to do the MOOC was “just think what you could do with 1000 students” but we had 42k. But it was exploration for us
A: what interested me most here was the focus on the individual and how things come together.

Terry Lynn Thomson – mobile work learning, spatial re orderings and digital fluencies

I am ingested in learning outside formal spaces. Savage , Rupert and law 2010 talk about the digital being bound up with reterritorialising space. Massey 2005 talk about multiplicity. I conducted research with 23 workers in Kenya, Rwanda and Canada and looked at their online learning practices – their engagement with others. Piecing together networks and materialities both with scale and focus, and having to live with fragmentary/also lashing things together… There were interactions that highlighted mobilities. But certain Immobilities were significant…

Many devices that are theoretically mobile do not, in fact, create or enable mobility. Ingold 2012 talks about the tensions of becoming here. Participants described complex cheorographies around his technologies, dictated by task, power supply access, etc. so material and relational practices really made these spaces. The making of mobile space was complex, the physical with the digital. Hemmet 2005 talks about the reassertion of the spatial in these types of ways.

Actually contrary to rhetoric, there are ,any frustrations and Immobilities. And for some of my participants the materials were struggling to become more mobile, in the same way as his gestures of mobility.

So four things here. The issue of digital fluency: flipping from one to one, to many go many; negotiations of openness, etc. so one way educators could support adult learners is to help them find ways through spatial re orderings and mobilities of work learning practices.

Q&A

Q: how do you educate around these mobilities
A: I don’t know, what do you think?
Q: seems tricky…but needed…
A: some school areas around the world attempt to do this, but in the context of a world where many adults do not have that understanding… Lots of questions here…

Q: I’ve been a learning technologist for a while, quite good at this… But still don’t understand everything… Don’t we need to educate ourselves before educating others,..
A: one of my participants Makori, an it consultant, took an online course, took the idea of one authoritative sources, and he had so many devices, wanted an iPad but

Q: I encouraged my students to share an app they find really useful. One way to address this is to be transparent and open, rather than apologetic for using devices
A: well one of the interesting things about entrepreneurs and the self employed is that this stuff is so time consuming, so difficult, can be so hidden

Q: how do you educate for digital fluency. Doesn’t use really do that?
A: technologies can educate you but if that’s how you want to learn… Well that’s about how you choose to be critical… I do think that all technologies try to lead you down a certain path… So I don’t know how you do that exactly.

Policy networks, database pedagogues and the new spaces of algorithmic governance in education – Ben williamson

I want to talk about new sorts of actors in education. Some on schools, some relating back to that. But I want to talk about several elements. The idea of policy mobilities – new structures and styles of decentralised, educational policy and network governance. To talk about mobile bodies and algorithmic traces. And finally the kind of mobile code spaces.

So I really want to ask questions about how new policy networks are seek b to reimagine education. Trying to map relationships between new kinds of actors and relationships between actors, and also the COde acts in Education project (link?).

Part of this idea is about government not being just my about central government, but also actors across public and private and third sectors. We see flows of policies from, say, private sector flowing into ouboic sector. Hultqvest 2001(?) talks about such flows.

In education we see educational governemce through mobile cross sector policy. Et works – govt departments and initiatives, commercial companies, NGOs, philanthropic organisations for instance. Steve ball (2012) has written about this wide range of actors in educations policy.

For me I am interested in the idea of their sectors, local controls think tank demos, nominet trust, the young foundation, nests, big society network, the innovation unit. All of these organisations seek to reimagine and reconfigure ouboic space through networks and digital platforms, they are also connected to a global picture of organisations nudge think tanks and policies are on the move, partly through these networks.

So we see these cross sector policy networks, labs, intermediaries, reimagining education. Moving from a sort of bureocratic place to a more individualised data driven concept.

So going to mobile bodies I want to talk about networked learning, database pedagogues, and the distribution of learners and learning.

So I wanted to start with the RSA, the Royal society of arts. They have a programme called Open Minds with particular curriculum eye, and they are also looking at using networks – see n squared report by Paul Or,arod. They blend the technical and the social networks at the heart of oublic sector learning.

The second area is The Innovation Unit’s report in learning futures – a report talking about institutions as “base camps”, in “learning commons” and “extended learning relationships”. And discussion of an innovation ecosystem. The same organisation has created reports like 10 ideas for 21st century education – calling for radical re imaginings of education, the disappearance of classrooms as spaces. Talking by out tailored and individualised learning. And particularly computer generated playlists of videos, seminars, small discussion groups, and 1:1 learning. Also a lot here about digital expertise, social networking for peer to peer research. And you can see here a collapse across institutions and relationships… By new kinds of actors.

And I wanted to also look at the education foundation – claims to be first independent think tank for education in the uk. One of their big reports has been a We book guide for educators, and taking that tool, mobilising it to the educational domain. That as an infrastructure it is transferable. Some complexities here, the horizontal nature of Facebook connections may not be what we want to reshape all educational relationships to be.

So we see a new expert knowledge and vocabularies of human behaviour and socialite based on theories from social media… Of digital commons, of smart mobs… Many of these are coming from places like Facebook. Facebook have an in house sociologist and data science team, those practices are coming through these kinds of actors into what we do… As if our networked social brains have evolved to demand networked social media. This may reflect Hacking 2007 ideas of addressing kinds of people, who others (Facebook say) imagine we are.

And that brings us to the issue of database pedagogues. Data is increasingly important to our understanding of pedagogy. Mackenzie (2012) talks about this centrality of a databases to the world.

Nests digital education programme very much focuses on learning analytics which gathers data about the learner in order to generate ideas or prescriptions for further learning. So we see things like Knewton. It gathers data on the learner, uses psychometric data etc. to “personalise” the data to the learner. So the learner is visualised and understood as data, metrics, numbers, and increasingly visualisations… The actor is transformed in order to be acted upon.

beluga markets itself as a smart service able to behave with intelligence, combines “intelligent data” collected by educational institutions with students own social media “off put data” to reshape the experience for the learner.

These code spaces are about automated and anticipatory governing and data doppelgänger a, production of objectified individuals whose data is used to classify and sort the individuals into types.

So we can see then that these kinds of activities, these kinds of bodies… Urey 2007 talks about us not as private corporeal bodies, but as bodies distibuted across systems through our trails etc.

So what I’ve begun to try to trace out is the idea of networked governance with wide range of players seeking to reimagine education through vocabularies, expert knowledges and techniques of dpnetworks and database providers. Governance through code spaces, understanding the learner through data traces. Activating learns through automatic pedagogues

Q&A

Q: I’m not convinced the network is determining the governance but reflecting those back for financial/political reason. And much of this data reflects growing commercialization of the student, data use mirrors that used in corporate contexts.

A: There is flow between these sectors and some of that is from government to these organisations.

Q) what methods are you using here?
A: that’s exactly the focus of methodological workshops we have been running. Partly with data scientists and programmers. Need hybrid methods, beyond more comfy social science. Ethics. I’m trying to follow the actors… Tracing the ideas and how they coalesce. I don’t tend to go and speak to them, although I have spokent to the director of research at demos on their new social media analysis tools

Q: all sorts of discourses about understanding the learner. We have the official discourse. Why is this policy move towards disruption, removing gates to traditional education. Where’s it going. What’s the benefits.
A: not sure I can answer that directly, I’m more interested in tracing the routes. Looking at a book edited by Martin law and ? Rice. They look back to the worlds fairs and this fascination with data goes a long way back.

Disrupting the illusion of sameness: the importance of making place visibke in online learning – Phil Sheil and Jen Ross

Jen isn’t here today as she is en route to Dallas. I want to use her journey to start thinking about maps… So her route from Heathrow to Dallas passes many cities… Comparing that airline map to this map of a journey by elephant – very detailed, very differnt.

So I will come back to maps… In education we see these ideas of learners quietly reading. Learners works together in ordered structured spaces. But I don’t think these imaginings particularly gel with the formal learning environments… Blackboard collaborate for instance. Some spaces reflect sameness.s they literally reflect our own faces back to us. Drawing on Sulla and djallalwer? Talk about education beyond the nation state, as a response to globalisation. But how may the realities be similar or differnt.

Some of my own colleagues work in Africa where only five hours of power were available. One student required a synchronous session be rescheduled because of restrictions on internet access in Egypt.

So Jen and I wanted to reflect on out own students. Students were able to indicate where they were located on the map, or favourite places… In some cases mini biographies emerge, often with temporal aspects “I work here, I live here, I went to a conference here.”. This allowed students to show spaces important to them…

Second example from the DiCE group, one task was creation of my digital postcard from participants on the MSc in a digital education: a photo, some text, some sound. For instance a computer with a cat and the noise of purring.

So Jen and I wanted to think about sameness and difference in digital education contexts.

Q&A

Q: could you talk more about distanced cosmopolitanism
A: particularly about dan Alger wrote an earlier paper on embodiment which is also interesting here.

Q: have you looked at how the student imagined the student
A: no but that would be interesting. But on that world map the staff took place too.
Q: I was talking to someone yesterday about imposter syndrome… Maybe that understanding of how the students imagine us might have a real impact for us as teachers.

Wider discussion

Q: Ben, that was an interesting overview. What is exciting, what is concerning… How does this data and doppelgänger a stuff relate to, say, open minded news
A – Ben: I’m not opposed to data analytics… A long as we area ware of how the data is generated… And how the system is interpreting the data and understanding the learner. Vying little critical work on that area. I am interested to better understand that. And what goes into those sorts of technical fix for defined problems
Q: what is the panel excited about. And what are you worried about.
A – Ben: I’m excited about the opportunity for research in layered code spaces here.. Software lays on top of but also underplays everything, managing how educational institutions function.
A – sian: exciting but real challenge is that the institution is so bound to campus, and to nation state. And difficult to think about how we challenge that
A – robin: I came out of foucoultian and Marxist analysis so I, concerned about those analytics… In terms of MOOCs and the monetisation of it… Obvious way to do this is to sell user data on as Facebook and Google do. But I remain optimistic about the use of technology for citizen science and citizen social science.

Comment: I’m fascinated by visualisations, by the manipulation a and map projections that distort reality. A lot of this is the methods questions… There area. Load of people structuring the language that shapes the world we engage in. Because they talk in code. Literally. I know that some digital humanities people coopt that, say “oh I, the social scientist”. How do we have a dialogue rather than handing social science over to those coding
A – jeremy: I think the key thing is to not draw the line. Seems to be driven by computer science, social network analysis… So we Need to engage. That code that underlies it…. We have this thing where we have to visualise to understand the code… And that could be more productive
Comment: numbers have been important for a long time. Combine that, with the visual, and the data doppelgänger who can be brought and sold… Is that a way to instantiate, to create a more sophisticated other self? In to way that may be easier to move, or commodify…
A-Ben: john hannay(?) talks about we make spaces manageable… Making the individual knowable, inspectable etc. we don’t often see the data, but the visualisation that is one step away…
A- sian: maybe that’s how we decentralise assessment – look at a students visuals at the end of the course!
Comment: isn’t half the problem that I don’t know my doppelgänger until I come up against some sort of problem – rejection for a mortgage or something?
A – Ben: and that data is constantly being recombined, reshaped, it’s an active practice or active accomplishment. Like rutherford’s work on a data analytics in children’s services
Comment: but I’m a really minor part of that dominant system, I’m a very small part of this…
A – Terry: I think when we talk about mobilities… Those maps, those spaces all disappear when points on a map… But doing that, reimagining the spaces, takes away from the specifity of space and the context…

Comment: I’ve been in conversations about big data and visualisations… Often presented with a visualisation as a done deal..
A-Ben: the concept of data not being theory free but not being an educational theory
Comment: or the issue of “we don’t need theory, we have data!”
A – robin: for us as educators the idea of education and learning sire ally hollowed out. There isn’t one shared understanding. We need to strongly re articulate what we mean by educational spaces or learning spaces. To some extend all spaces are learning spaces. Bernstein was against the idea of all spaces being pedagogised but, to some extent, that’s what we do.

Conference plenary

Christine Sinclair is introducing the discussion session. And we will be starting hearing impressions from different people on what has been taking place, what we have been doing. And we should have time for some discussions afterwards.

Jeffrey on the hot seat discussions
These take place ahead of the conferences. They are an opportunity for discussion, for reflection, often amongst those with more robust perspectives. The discussions took place on a site linked from the main site. You can find each of these online, including some who weren’t about to join us at the conference in person.

These discussions are about widening nerve extending the space… We look for input, to hear reflections on those discussions. So I want to recognise and thank those involved. These are exciting opportunities to have that sort of interactions. Eve heard really interesting papers here, and discussion, but there isn’t always opportunity to dig a little deeper. To engage in dialogue, to engage where this really happens. To understand what networked learning is really all about, and to put into practice what it is we talk about. And I hope we can get more direct input – if you took part, what was that experience like. Please do think about that, what could we do, and how can we look at that In the future.

Comment: as an experience of a newcomer to the conference for me it was great just to read everyone’s comments and get into it… Although tricky to get into the discussions sometimes. Display sometimes quite confusing, sometimes hard to follow… So layout could be easier to navigate.

Christine: it was in Ning this year wasn’t it?

Jeffrey: yes, we’ve used ning for the last two but other technologies before

Terry Lynn: I found it so useful taking ideas from those hit seats to feed into our paper, that was an incredible gift for us.

Martin: that was part of the idea of them… Not sure many have mentioned the hot seats during their presentations

Terry Lynn: we did that.

Christine: thanks to Jeffrey for the hot seats and the SCHED.

Jeffrey: yes, and feedback welcome on both of those.

Marshall dozier – doctoral symposium
Reporting on behalf of group of Tim fawns! Phil Sheil and jeremy Knox. Like all good students we made a bit of trouble. And like all good parties there were gate crackers and that made the sessions all the more fun. We questioned multimodality. We were introduced to the idea of “Skatology of digital sociology” from Tim o’Keefe. We also benefitted from mentors stirring up our discussions at the session. And we came to a really messy and interesting collective understanding of digital technologies.

Christine: the twitter feed really captures that event. And many of those who saw that wished they’d been three!
Marshall: like all good parties!

Reflections on the conference – steve fuller
Thank you for inviting me as an outsider. I’ve been trying to figure out just what this community is. The conference has run for 18 years when clearly networked learning meant something else entirely.

Are you a professional grouping? Neil Selwyn”s opening talk, which not all agreed with, called for that sense of criticality that might enable you to colonise an authoritative space in a productive way. Let me say where I would more strongly sit, which is this so of grouping in relation to the institution and the higher education organisation. How will this group be seen by HE administrators looking at the problems and concerns. You need a more defined idea of the university – it is not just a space. But this group is in a good space to understand this as a group looking at technologies mediating all aspects of the university today.

The other side is of this is research… My senses is of importing to this field, not exporting – although digital droppings may be a great contribution here – but where are the contributions, the concepts feeding out to other fields. Of you want to develop this field professionally you need to develop it in a more serious way…

One thing that frustrates me about this, and other fields with a support type function is that of observation, looking in. But it seems there is more of my responsibility not just to observe but to see problems at head, issues coming up and to offer suggestions and advice. And that means that I think you have a privileged and positive impact that you could make.

Comment: I think networked learning thinks of itself as being critical. You describe it as in support role. I understand that in a technical way. But there is increasingly a professional world of educational technology but networked learning sees itself as a serious and rigorous research event, at distance from a field that is professionally located in support role, where literature is often grey and funded by those with a particular agenda. But I took from your talk that we have to articulate that better so that others can see and understand it.

Comment: I really like that idea of colonising networked learning. I was thinking also after Neil’s keynote, talking about being critical and having the agenda of pessimism. But I’d like to see more positivism and action. Wondering how we could make something similar. I think at edinburgh you have maybe done that with the MOOC! But I’d like to think about how we. Can do that more.

Comment: Steve’s observations as. An outsider who has spent sometime with us are really worth us thinking about, particularly if we have set up artificial boundaries around ourselves, rather than crying out if we feel misunderstood – I say this as so done from a field that has complained about being misunderstood but it has been their/own fault.

Comments on the conference by former co-chair David
As a conference we really come from the perspective of pedagaoguey as a focus rather than technology. We came in around open pedagaoguey in 1998. We’ve moved away from that towards concepts like online trust, communities in online learning… And how that has been exhibited throughout the conferences. And more recently in the book series coming out of the conferences. How that has captured the feeling coming out of networked learning… Seeing the shifts and history of the space.

I talk about practice because even though we are researchers what we look at is so much about practice, about how we put things into practice. But we also see a move towards networks and how networks work in that sort of disembodied ways and what we y to achieve in networks… Away from pedagoguey in some ways… But reflecting our need to attract students. Add we are seeing technology perhaps trying to take over our role? Is it a meaningful thing? How does it compare to the thoughtful pedagogues within networked learning, and the issues of deep learning and pedagoguey? And our values here? I’m not sure about the answer but I am concerned about the possible consequences of disembodied learners understood through their data… So I am interested to look forwards to consider whether we focus on values, on pedagogies, away from MOOCs for attracting students…

Comments on the conference from former co-chair – Vivienne
I will take a different tack but that will surprise no one who knows both David and I. Networked learning has always been concerned with the role of the university and the insitution. We do aim to be mindful of the role of the individual in the networked world. We have always been thoughtful about who we ask, what our responsibilities are to ourselves and to others in this world. And with the huge range of papers this week we need to be critically thoughtful and aware about how technology is changing how we engage with, make sense of, and respond to the world.

Comment, Sian Bayne, university of edinburgh: what I’ve seen networked learning become is something we desperately needas a. Field. A space for critical space for discussion around what technology is doing to learning. We were so excited to host it here as it is the key thing in the year. So I hope we continue doing this and focusing on sometimes risky research in this space.

Vivienne: I was discussing with colleagues why this conference has become this space… I think it is because the networked learning raises those issues… So the networked learning becomes to disappear

Comment: I want to put in a plea for networked learning to not just be about technology, to not just be about HE. There are plenty of conferences for both of those. The interesting stuff here is about the impact of technology of learning, and of learning outside and beyond the university.

Martin (current cochair): the external factors have more pushed that. Maybe we need to think about articulating better the pedagogical theories we are developing, informing practices for learning. For me it’s profession learning, networked spaces… Not just HE for me.

Comment: I want to refer back to David’s comments… The axiomatic nature of networked learning makes it special, and different, and those underpinnings are crucial to differentiate us from other fields we touch shoulders with regularly.

Comment: my take on this is that in 1998 university websites were left to departments, development was very open… And we have moved away from that. We may not just be about technology but we have to engage, we have to udmerstand,otherwise we fail to understand the code. How else to avoid Facebook funding some policy somewhere that shapes our world. We have to engage with technologists, and to understand the code.

Steven: I agree with all of that.

Comment: we invited Judy Marshall to the last event, some scratched their heads as she applied sustainability to networked learning… But what other technology conference would do that! I also want to guard against seeing the social network as the meaning of networks… I would move against that. We need to think of networks in that broader sense…

Jeffrey: there are some nice definitions on the website there. Much of the discussion is about what networked learning means… It sounds like all of the possible understandings are valid… And useful when explaining to others what we are doing when we attend these events. And when we think about what was that, how does this apply to us… ?

Christine: I think that’s a very nice point to leave this discussion on! And with that over to our conference co-chairs martin and Donald?

Martin: I think that’s a nice point, reflection and bringing things together, for the next conference, our tenth anniversary. I was wondering how many review papers we actually have. Maybe for the next conference it might be a nice theme to think about… As well as looking back, to look forward.

Donald: I think that is a good idea… We know that the conference will be… Somewhere… In 2016. Probably also within Europe. Watch this space! I think it’s been a wonderful conference. And we have a huge list of people to thank.

So we would like to thank our keynotes steve fuller, and to neil Selwyn in absentia (steve receives a gift, and notes he has homework!), thank you to sian Bayne, Christine Sinclair and fiona Lennox, and Jen Ross and Hamish Macleod for all of their work organising this. Thank you to Marshall dozier, jeremy know, Phil Sheils, and Tim fawns for a wonderful wonderful doctoral symposium. Thank you also to Louise and the staff at the conference centre. And thanks to Jeffrey for the hot seats! And thank you to tom and to steve wright for their work on the hot seats. Thanks again for all the hot seat presenters. And there are two other people to thank. Alice jesmont and Chris jones, for those that know the, Alice continues, in many way, the spine of the conference providing administrative continuity. And finally we’d like to thank David and Vivienne for all of their work on the conference since 1998… And with that safe journeys home and we will see you in two years time!

.

 April 9, 2014  Posted by at 9:05 am Events Attended, LiveBlogs Tagged with:  1 Response »
Mar 252014
 

Today I am delighted to have been asked to liveblog another of the ARTIST ROOMS Research Partnership events, this time hosted in collaboration with Generation: 25 Years of Contemporary Art in Scotland, a forthcoming exhibition at the National Gallery of Scotland.  The seminar has been organised by ARTIST ROOMS and engage with funding from the National Lottery through Creative Scotland. Tweets can be followed on the wider #artistrooms hashtag.

The event, Gallery Education: Developing Digital Resources, is of particular interest to me as EDINA lead the development of a number of innovative digital resources, and I’m particularly interested to hear more about some of the challenges of digital resources around the arts because of our own work on the Jisc MediaHub service.

As usual this is a liveblog so I apologies in advance for any typos, omissions, etc. and welcome all comments and feedback on the post. And if you enjoy today’s post I would recommend looking back at the MOOCS in Cultural Heritage Education liveblog, which now has a number of additional resources and references added. 

Welcome – Sarah Yearsley, engage 

Sarah Yearsley, engage, the National Association for Gallery Education

Sarah Yearsley, engage, the National Association for Gallery Education.

Today is the second event that ARTIST ROOMS and engage have collaborated on looking at digital learning, and part of a series looking at best practice. We also are running the event in collaboration with GENERATION: 25 Years of Contemporary Art in Scotland, which is running events across the country. This is a busy year for ARTIST ROOMS. Engaging with young people is a common theme when we talk about engaging with young people in the context of both ARTIST ROOMS and GENERATION.

Welcome from Damien McGlynn, ARTIST ROOMS.

Damien McGlynn, ARTIST ROOMS.

Damien McGlynn, ARTIST ROOMS.

Damien is giving an outline of the day which will include two discussion groups and an opportunity to see the Louise Bourgeois exhibition, and to play with the Art Hunter app and my colleague Tessa, who produced that, is also here today.

We are running this event with several partner organisations: ARTIST ROOMS, GENERATION and the ARTIST ROOMS Research Partnership. Our colleague Professor Neil Cox from Edinburgh University is here today so do ask him any questions about the research partnership. Now over to Rosie who will chair the morning session.

Morning session: Mapping the terrain and producing content for your audiences.   Chair: Rosie Cardiff, Senior Producer, Tate Digital

Rosie Cardiff, Senior Producer, Tate Digital

Rosie is giving some background on her role, looking after much of the digital learning content on the Tate website. I also manage a small team that manage the Tate Kids and Tate Collective (resources for young people) part of our site, I just wanted to highlight these learning resources. One of these is Circuit, which is working with young people across the country, the website launched recently but will be showcasing digital content produced by young people over the next four years.

Another project which I thought might be of interest, done with Tate Collectives – a young peoples space around the galleries, where we did the 1840s GIF Party – GIFs based on the 1840s gallery. We provided training on how to make the animated GIFs. This was hugely popular. The girl in the grey dress has had over 77k reblogs, but we have also seen a huge spike in interest in the painting itself in the gallery as well.

So those were a couple of projects I wanted to highlight – but do talk to me during the day and ask me about the projects we have been up to recently at the Tate.

So now over to our first speaker, Jen Ross, who is director of the MSc in Digital Education, and also a tutor on the eLearning and Digital Cultures MOOC, which I have had the pleasure of doing and really enjoyed!

Content is just something to talk about: designing for active online learning’ – Jen Ross, Programme Director, MSc in Digital Education, University of Edinburgh

I’m not sure I need to do anything now that Rosie’s shown the work that Tate has done to engage people digitally in the collection… I will be talking about how we do that at the University of Edinburgh. Really the thing that I want to say, my contribution to this day, is “yes, content is amazing, but content is really just something to talk about” whether thats online or face to face in gallery or schools spaces. I will talk about what you can do in the online space, as sometimes its easy to think about what we might do in a gallery or a face to face space just because

“Conversation is king. Content is just something to talk about” – Cory Doctorrow

Jen Ross, Programme Director, MSc in Digital Education, University of Edinburgh

Active learning has been a huge movement in teaching and learning spaces. You see spaces where learning takes place around tables – like today – rather than lecture theatres. And digital spaces can be a way to encourage that active learning without needing to reconfigure the space. the barrier to creativity is lower when we talk about participating digitally. And you can really evidence that work you do with people – the Tumblr page that Rosie showed is an exceptional way to evidence the impact on young people they are trying to reach.

So as you think about these spaces today I want you to think about these spaces and how they can be connected, intimate, busy, creative, exploratory and inspiring. Its not just about putting content in the digital space.

So I wanted to show you some good examples of work that our students have done around digital creativity. We have a module “e-learning and Digital Cultures” which is part of the MSc in Digital Education. This was the starting point for the MOOC, it was also the first module we had run that was entirely openly shared – students had to be enrolled but what they produceed was all shared openly online. And we really asked students to make multimodal work, to express what they had engaged with in the course. So in 2013 we asked our small masters group to engage with the much huger MOOC course and how they had encountered that. So people made videos, they shared things on Pinterest, we had people creating visual people and making interpretations around the. And also using onlione resources that only exist online – for instance “ThingLink”. Some of our students are really digitally savvy, but even those who are not can use the big list of interesting resources to create engaging materials.

That process worked really well and so we set, as a final assignment on our 5 week free global #EDCMOOC course, we also asked for multimodal assignments. Not everyone commits to the course throughout but those who did were asked to create a digital artefact, which was peer assessed by others on the course. And we have gathered these publicly. So this padlet I’m showing represents only about 300 of those produced but it gives some sense of scale. But if you ask people to create things they really respond.

Image showing the padlet of #edcmooc artefacts

I also wanted to show you a trailer for a game which some of our students on our game based learning module did, “Tomorrow Calling Trailer”, this was so much more than was required for the credits for the course. If you give students the right content, and something to create with some relevance to them personally and/or professionally, than they really do respond.

This Open Badges and Open Accreditation open education resource is something that was produced for my module Digital Futures, and again this is so much more than you would expect in a postgraduate essay, and it’s lives on beyond the class.

Sometimes students go further still. And here we see a multimodal dissertation (multimodaldissertation.weebly.com). When you open these avenues up, then you really see unexpected things like this take place.

This is our Dissertation Festival, which takes place in Second Life. This is a space for collaborating and sharing experience with each other. These kinds of spaces and collaborations are another way to think about what you can do in an online space which are not about just creating new content or resources. A Digital resource can just be about making a space for interaction, a space for people to work together.

And this is a project that some students did, totally separate from their coursework, asking students learning online to reflect on the playlist that has inspired their work (www.elearnenmuzik.net). Again these are projects emerging from the context of the Digital Education programme, but come out of people engaging in digital spaces and being involved in things that they are interested in.

So what I want to leave you with… whatever you are thinking about or planning, do think about not only good content or resources but also how it can be a great active learning space for your learners, for your audiences.

‘Digital fear & beyond’ – Rohan Gunatillake, Co-producer, Sync

And now over to Rohan, co-producer at Sync a collective supporting technology and the arts, and he’s also been working in the Digital R&D fund for the arts:

What I’m going to do is talk about… well I don’t have a background in education but I have spent the last four or five years working with arts organisations and technology and digital. And talk about some of the issues either supporting or getting in the way of really interesting work in the arts. A lot of that for me is about recognising that working with digital technologies isn’t about technology, but about people.

Rohan Gunatillake, Co-producer, Sync

The story sort of begins… I moved to Scotland four years ago. Mainly for love, but also because I got a gig with the Edinburgh Festivals! I came to start a project with the Festivals called the Edinburgh Festivals Innovation Lab – Edinburgh Festivals is the group of both the very big festivals and the smaller and much more niche festivals. The question we have here was that, like you, these are great organisations and very busy and in the domain of innovation and digital practice they are doing what they think they should… but where are the other opportunities? What other possibilities are there? What have they missed? So I was looking across the 12 festivals to look at that. Some worked well, some worked less well but all were really useful for trying new ways of working.

One of the big core things we did with the festivals was that obviously the festivals are a rich resource… and Edinburgh University has generated a great technology and start up scene… but they didn’t talk to each other. So the thing Sync is best known for is for the Culture Hack Scotland 2011 which was this big event to bring these groups together. Scotland is not a big enough country for those sectors not to be talking to each other.

And then Creative Scotland liked that festivals work… and wanted us to do that across Scotland. So we had a two year project called Sync and again this was about creative relationships, not just transactional relationships. And we carried on running the Culture Hacks – these are 48 hour opportunities for technologists, producers, artists, arts organisations, all getting locked into a room to create stuff. Amazingly each year people come and about 30 projects get made. And we have supported that with the Geeks in Residence programme where we’ve taken developers to arts organisations from the Royal Opera through to arts organisations in Eigg. We wanted organisations to see what it would be like to have a technologist in your organisation, in your building coming up with ideas and projects specifically for you. And that’s been really interesting and challenging. We’ve also created this magazine, Sync Tank, highlighting this type of practice across the UK and across the world.

I have also been working on the Digital R&D Fund for the Arts – this runs in England and Wales, and on a smaller scale in Scotland – which funds experimental projects (about 60 each year) around galleries, often around education as well. I am a “learning partner” – I listen across all the projects and pick up the themes and the big stories, and tell the story of that to the wider sector. We do that through the website and also through a print magazine which will be printed in the summer.

Even summarising the insights from Sync takes a long time so I just wanted to highlight three things coming out of these three strands of work. And these are in the context of what makes the most

How we are arts organisatins can move past the commissioning mindset. The pastiche of how the arts work with technologists or digital agencies is that, I give you £5k, technologist goes back to studio in Leith. And when they come back everyone is unhappy. That’s been how we have commissioned in the past, often about websites. Often these projects are approached like physical builds – big plans, fundraising, and unexpected ongoing costs. That’s the pastiche. What Sync – and others – have tried to do is to break that. Sometimes that is the right approach – if you know what you want to achieve and have requirements to deliver again. But when we see the kind of experimental work Jen was explaining, the Tate Digital world… where do those ideas come from? How do you assure that those are strong enough ideas? Sometimes your team can do that, and that’s great, but often the best ideas come from conversations with others coming from another perspective, an outside point of view. We can easily think our audience want the same from our work as us. So arts organisations can feel that metadata is really interesting – really useful for them but is it what people want to do? Maybe a niche! So you need to try to establish what users really want. We’ve seen that where that really works, two things happen. Firstly the organisation lets the digital talent to bring their intelligence into the room in a really open way, not just give them things to do. So in our Geeks In Residence programmes I went and interviewed directors of the organisations wanting a Geek, they talked about collaboration, and when the Geek showed up they closed into a commissioning mindset. We used the mindset that if you have a photographer in residence you wouldn’t tell them what to take, what lens to use, what shutter speed… as they began to understand that metaphor, that you would never do that, then something more constructive could take place. So as long as you think about what you want to achieve, but not be too perscriptive, that goes well. And the other thing that works really well is co-creating with audiences, involving them in the design process. And we’ve seen Unlimited Theatre doing fantastic work here. That’s one big message.

The two cultures thing?. We like to tell the story of technology and the arts as being two cultrues… but what I notice in practice is that the clash of cultures is actually the “deliver the project” methodology that people are using. I said people approach digital like capital builds. there is a gantt chart or basecamp… that’s how they see digital project delivery. But there is another part to the process in start up and web culture around prototyping, iterating, testing in public, taking feedback to improve. That iterative model is very different approach. It’s like Prince2 versus Agile. What we have found is that some arts organisations really understand that… they are used to the culture of the rehearsal room, to creating that way. Others are cautious, if not terrified, of showing something half made in public. Because of how they normally present work. Agile and Lean are thrown about but if we want to successfully do that stuff, it can require a different mindset.

Digital Fear. Part of what Sync does is take people out to drinks. Once every three months we invite four or five people running arts organisations out for a drink, with an invite along the lines of “when it comes to this digital stuff, the common complaint is I don’t have time or don’t have money. Tell us what you really feel”. That invitation tries to create as a conversation… the Festivals work was around a big data project, technical challenges that were solvable, there were business issues which we solved, and then we still had the “I’m just not quite sure about it” factor – the emotional part. And thats the most significant and most under talked about part of digital innovation in the arts. We call it Digital Fear. What comes up in these conversations re things like “I’ve been an expert in my field for 20 years, now I’m not an expert”, “I speak to the web developer and I feel like I’m talking to  my children”. That’s real stuff. That’s messy emotional gut stuff that is much harder to solve than the business or technical challenges.

As a coda to my kind of “drunk uncle” speech/provocation here… we are a relatively small player, we’ve been invited by Creative Scotland to bid for another two years, and we are placing three things at the heart of our proposal is:

  • Practice – if whatever digital work you are doing isn’t about your core practice then you will never get senior buy-in. If it’s not about what’s on stage for you, then that’s not what you should be doing. How are digital tools changing your core practice?
  • People – it’s the people who actually make the work. You hear about amazing work but behind all great digital projects is a very tired and very brave digital person!
  • Process – we often talk about projects… but the risk is that if you just fund projects we just have lots of nice projects but no organisational or sector embedded learning. So how do we embed innovative learning and processes into our organisations. This is a much more sustainable way to build this stuff – the teach a man to fish idea.

So that’s Sync, and that’s us… thank you!

Discussion group one: Audiences and digital content

We are now moving to discussions in our groups (I’m on the Purple table) so notes here will be sparse as we get chatting but I’ll be capturing the reporting back to the room shortly.

Rosie is introducing our discussions here drawing the sheet on each asks us to consider:

  • Who is your target audience?
  • How will you measure the success of your digital learning project?
  • How will you produce content for your audience? Can you repurpose existing content?
  • How will you engage and interact with your audience? Where will that be?
  • What will the ongoing legacy of the project be once it is produced? How will it be maintained and sustained – not just technically but to keep things fresh.

So, we will be thinking about and discussion these for the next 50 minutes. And I’m sure anyone reading the blog today would be encouraged to do the same and to tweet any comments to the #artistrooms hashtag.

Discussion group sesstion

Some thoughts from our group:

We have been talking about our own contexts and backgrounds, and the kinds of projects we are working on. There is a fairly common focus on young people so we are just unpacking that a bit: thinking about how to make young people feel welcome in the space, using the right language for young people, the use of the right spaces (such as Tumblr, Twitter, Instragram, and custom websites) and a focus for young people as co-producers in these space – posting to the accounts for the night etc, and to some extent training young people in the skills and confidence to use these technologies, and the meaning of doing so on behalf of an organisation (professional skills). Ownership and active contribution are being flagged as the most effective way to create better digital projects, and to build ambassadors in those groups.

Some discussion of practical issues and kit: phones, ipads, that can be used, laptops within the office – but the logging on/take over of accounts takes place when they are in our spaces – we log them in. Discussion of a real sense of caution about how acceptable that stuff is, how much control the organisation can and should have, and what challenges some submissions can raise – do you show critical work? what is the impact of that? Can raise really thorny issues, so you need processes in place to deal with that. The more you involve your audiences, the more those issues are raised. When content is out of copyright, this stuff can be easy, but often you are much more restricted than that… that area of IPR is tricky. You cant let people use the artwork, maybe they have to be inspired by it instead… you have to think laterally. And then when you do ask for contributions you have to have clear guidance, clear terms, ways to ensure that any clear problem can be dealt with but there are lots of grey areas.

We are now discussing the types of projects we might have in mind… one of our participants talks about schools groups coming to a museum in the same building, but not tending to come to the gallery. And real challenges around creating materials for teachers during the time ever changing exhibitions are running. But a recommendation: Group for Education in Museums, part of the Scottish Museums group, which are the generic resources/activities called “Hands On” – a downloadable PDF. And Glow also offers potential – you can bring an archive to life, getting artists to talk about their work – almost creating little programmes, setting a series of challenges. Glow Meet works really well as it’s live and interactive and at the end there are resources to explore. An online platform to use… but it hooks in pupils but also teachers and parents now primed for involvement. But schools work is obviously working with the teachers, not necessarily the young people. But there are also new youth arts club ideas that take it out of formal spaces… that’s happening this year. There will be 12 hubs around Scotland, a different way to connect with young people. Also Code Club (for 9-11 year olds) learning online coding, also apps. We have kids using Scratch, and now HTML, and working with an online gaming company whose staff volunteer – as part of staff development. That company are now looking at accreditation. But code clubs are free to do, need to be volunteer run, there are free resources to use. And it seems effective and really creative – lots of ideas and collaboration taking place, a whole group to continue working with… skills based and bipassing and teachers’ own Digital Fear. And it gets interesting as the youngsters start encountering code used in industry.

And we are moving onto the idea of measuring success and how one might do that… depends on the aims, and how clear those are. And about what the audience thinks is working for them, what they would change, how they feel about it. Changing attitudes in an organisation can also be part of what you are measuring. Also discussion of Retweeting young people’s comments, using texts to reach people. Discussion of what counts for an organisation – just about the physical space or can organisations appreciate online engagement? How does that take place? How is that measured? Are likes etc. useful? How do you reflect richer interactions and what ways can you find to encourage that. Discussion of how to stay cool – Tumblr is engaging but niche, Instagram is big, Google+ Hangouts offer great opportunities for live Q&As.

Questions and Feedback

Red Table: we talked a lot about young people as our audience. We talked about finding platforms to use, in consultation with groups. We also talked about working with groups for longer periods of time, and leaving space for platforms to be changed or developed over time. Copyright came up in relation to the challenges of engaging with modern contemporary art.

Blue Table: we talked some of the projects taking place, particularly the GENERATION projects. The audiences often quite different as within gallery sector and artist educators. Talked about th eneed to provide something to different, not to replicate what you do, not to replicate what others are doing – e.g. why would we replicate MOMAs online courses. We talked about successes and moving away from the idea of numbers, and followers, and hits, but instead the quality of engagement. Hard to do, people don’t always comment or respond. It can look like you are failing – people can have a good experience without feeding directly back to you, so we talked about how you can understand that people enjoy what you do. And we also talked about organisational change and cultures. Also talked about the time and cost and challenges of suiting multiple platforms. And also social platforms – how we interact rather than push things out, and how that builds your organisations reputation. Also briefly talked about legacy – short term digital and social contractors result in expertise just leaving with the person.

Yellow Table: talked about sometimes the need to use digital is communicated, rather than the use of technology coming from the audience, and being about their experience. About focusing more on content than on technology delivering it. Some digital fear. Real feeling that many of the stats funders require are not actually that useful, that qualitative feedback is key and that there are better ways to capture that stuff. Then there was also the feeling that once digital content is out there you can find unexpected audiences…. seen to be a good thing but then do we respond to that… do we have more opportunity to learn from that inside of a  project – not just use in the next one. Lots of interest in Geek in Residence approach, the scariness of iteration too though and the organisational change required for that to work.

Green Table: We had a fairly wide ranging discussion. One thing that hasn’t come up yet… a kind of internal issue that we can be quite object focused, that the engagement with a particular object and the mediation of that encounter with the object. Lots of work already there but more to be done there. Talked about some local authority challenges – how things like ARTIST ROOMS or GENERATION can create opportunities for trying things out, to feed into other projects. Words like “risk”, “trust” and a fear about sharing a not fully formed project or website, that can be quite scary particularly in this economic climate. And we did mention the word copyright.

Purple Table: We spoke quite a bit about audiences. The majority of us were working with young people, schools or teachers. And the challenges of that. And talking about what success looks like – is it museum or gallery footfall? Or can you change organisational culture to count those engaging online, what does success look like there. And we also talked about some of the challenges of asking young people in to participate – and what happens if contributions are critical of your organisation or of your funders and how you deal with that. Also some concerns around very quick turnaround exhibitions,lots per year, and how one can generate content or resources to meet that schedule – and the possibilities of generic resources to help in those scenarios. Also talked about platforms… maybe drift from Facebook, where are young audiences going… and the potential of Google Hangouts – guiding people around a space at a distance. We had on our table some people who don’t have collections to work with, some fully formed apps, through to young peoples groups. Real diversity there so real opportunity for more sharing around these models.

And now more questions and discussion:

Q1) I wanted to ask about “bring your own devices” – I’ve heard schools talking about not investing in hardware but encouraging their students to bring their own devices…

A1) We allow students to bring their own devices, but have a device pool for accessing our app on site as well. It’s just launched so will be tested this summer.

Comment) Nick would just like to mention the Warhol MOOC with Glyn Davis, from the school of design. For those that want to try that technology.

And now we will move into the lunch part of the day…. so the blog will be quiet again for a wee bit!

Lunch – An ARTIST ROOMS exhibition Louise Bourgeois, A Woman Without Secrets will be on display in Modern One (Scottish National Gallery of Modern Art).  In-gallery ArtHunter app demonstration will also be available.

Image of Damien demonstrating how the app works on an iPad outside the New Acquisitions exhibition.

Damien demonstrates how the app works outside the New Acquisitions exhibition.

Image of a group Testing out Art Hunter at the Scottish National Gallery of Modern Art

Testing out Art Hunter at the Scottish National Gallery of Modern Art

Welcome back, Christopher Ganley, ARTIST ROOMS

So I just want to welcome everyone back with some thank yous. Thank you to Nicola for blogging today – we will circulate the link after the event. Thank you to Rosie, Rohan and Jen for this morning’s session. And above all a huge thank you to our funders for making this event possible.

Afternoon session: Marketing and evaluating digital resources – Chair: Tessa Quinn, Head of Digital, National Galleries of Scotland

I wanted to start off by talking a bit about what we do. We are quite lucky in having a digital team. We’ve been doing things including the Art Hunter app, and  the Titian and Diana iOS7 App for iPad, that latter was something our funders were really keen for. We are also creating a mobile version of the website, and that has included some changes and decision making around the website. I have also been developing a Digital Engagement Strategy and there are four key areas of that: Working collaboratively within and outwith the organisation – digital is no longer just for the geeks but about part of the mainstream; To Open up our collection; To grow our audiences; And to increase income.

Our strategy says we want to grow our audience by learning more about them. We did talk a bit about analytics, about what is or can be useful. And we want to know them to design for them, not for what we think they want. And with all this learning, we need to take some action. So you need room to look at your learning, look at your analytics, and look for possible change and improvement.

Tessa Quinn, Head of Digital, National Galleries of Scotland

Even though we have a digital team doesn’t mean that we don’t learn or make mistakes…. what we found useful was WeAreCulture24 Action research – they brought organisations together to talk about sharing analytics across 22 organisations. It allows you to see the differences and similarities across the sector, across the organisations. I highly recommend the report “Lets Get Real 2″ – and we really started to learn how to learn. When we look at redesigning the homepage we learn from where people click. Every department wants a page there but that’s not how visitors actually use it… and that’s about coming up with a question, then seeing if the numbers will give us an answer.

One of the things that we are trying to learn is that for ten years we have been trying to build digital content, some great stuff… but we are terrible at telling people about these products, making sure they find them… its something we could be much better at. And with that in mind I want to introduce you to David Craik Director of engagement consultants Bright Signals, he’s also formerly head of marketing for S1.

‘Agile Marketing’ – David Craik, Director, Bright Signals

I’m going to talk today about Agile Marketing – a good marketing buzzword (because it is). I will tell you a wee bit more about what we do. We set up Bright Signals about four years ago. We really create content. We do digital marketing for Tennants lager, working for Channel 4 for the Commonwealth Games, working with Ambition Scotland, also the National Piping Centre.

Hopefully what I’ll get across today is that marketing is really changing. The days of pushing and cajoling are over. Marketing is about giving people what they want, content they want and enjoy. Either we all work in marketing, or – maybe a better way to think of it – none of us do!

So we are about lots of creativity, but there is also increasingly an expectation that we can measure everything. There are many more channels than we ever had in the past. The pace of change is very fast! So one approach here is Agile Marketing. So to explain what this is… a history lesson… way back before 2001 most IT projects were based around bamboozling Gantt chart. And as digital began to explode it became clear that this “waterfall” approach didn’t work for digital – they were delivered late, over budget, and worst of all they didn’t deliver what users wanted. So a bunch of techies in California came up with the Manifesto for Agile Software Development. This set out principles that highlight flexibility. Away from that idea of project management, structures, etc. towards this flexible stuff. Did it catch on? Yes, it really did. Google for instance is all about changing based on what the audience wants. Mark Zuckerberg talks about “moving fast and breaking things”.

So what does this have to do with marketing?

David Craik, Director, Bright Signals with one of his slides

David Craik, Director, Bright Signals with one of his slides.

Well traditionally marketing has been about “big bang” campaigns, TV stuff, etc. With marketing… John Wanamaker said “I know half my advertising is wasted, I just don’t know which half”. It was about knowing it wasn’t working, and not knowing how to measure that. And then we have the HIPPO – the most HIghly Paid PersOn’s decision. Often who drives decisions, but rarely represents users need.

So Agile marketing principles?

  • Less big band campaigns, more small scale experiments
  • Less subjective opinions, more evidence
  • Less talking, more doing

We work every week on content, we develop a “back-log” of relatively small scale targeted marketing activities – or break up the bigger stuff – each with defined performance measure. And we deliver in short fixed cycles of creative and content development, then we ship, prototype etc. And we re-prioritise all the time in response to those measures.

Does anyone use Google Ads? Well if we look at search results to hotels in Glasgow… you would bid on a click on that advert (e.g. £1 per click). Advertising is interesting here because of the side of the audience. There are 694,000 searches each second. And that audience is pretty close to the point of conversion, the point of buying. On facebook the ads might be distracting or annoying. If I’m searching Google I probably want to buy or book things. And all that makes ads a great way of getting insights. Not necessarily sales but insights into what audience wants.

So if I’m setting up a wine shop I can set up an A/B testing ads… I’m trying to work out “wine order” or “ordering wine”, and “deliver wine” or “wine home delivery”. I place two ads, I use the two different words. I find out which one people click on. I don’t even have to set up delivery to test what will work. So that’s an example of how this can work….

Thinking about that pipeline… we use a tool called Trello, like a virtual stickies board. So we have ideas in the right hand column. Loose notional thoughts. Then it moves into a development phase…. might not go anywhere, might go into development and cost checking. And if it goes into production we then usually commission others – techies etc… and then it goes to live. There are two live columns that we use – Live – proven, Live – measuring. It only moved to Proven when run several times and engagement checked – e.g. on facebook it would be likes, comments and sharing. Only when it hits a level, e.g. 1% it moves to proven. If it doesn’t work it shifts to ditched….

So, for example… Brooklyn Museum had an exhibition of Indian Paintings. They set up a thing online where you saw a painting for a few seconds each and asked users to rate their favourites.

Looking at stuff we do every day… we work with Hornsby’s Cider – asking poeple to identify a building; project for Channel 4 called 9point88 about the Oscars; and for Tenants Lager about fixtures etc. None of these are directly anything to do with the product…. it’s tangental… that Brooklen Museum is example is just about the product. You in galleries and museums are so lucky in that you have great products, people care about that. For many brands there isn’t always that sort of devotion to the content. People are passionate about your product…

David Craik, Director, Bright Signals showing examples of new, innovative museum experiences.

David Craik, Director, Bright Signals showing examples of new, innovative museum experiences.

So how do you bring content into the museum experience. I was lucky that I was able to attend South by South West last year – and learned a lot. I made a point of attending a talk by Leslie Walk on the future of digital content for museums and galleries. She said “museum attendance are in decline. and hat;s partly because there’s something missing from most visit experiences… Play”. Now we can argue that perhaps but the examples she went on to give, about play in a good way, was the Cleveland museum of art and something called the “Gallery One” project. Now this had a £20M benefactor. But one of these ideas was just asking participants to take a picture of themselves replicating a sculpture – using an Xbox basically so quite cheap, similarly making a face like a sculpture, and then a bit interactive wall to engage with the content in a different way. Now this latter screen was so costly and perhaps more obvious. But the cheaper fun stuff – those statue and sculpture exhibits – were so much more engaging because they are playful. And that’s the tone of the brilliant Art Hunter app we’ve been trying out as well…

We are seeing Google putting the gallery exhibition tour online through google maps. We have Google Glass on it’s way and, by all accounts, wearables will be a big sector. The hardware – the hard to do stuff – has been done. The opportunity potential is content relevant to the location of the person wearing them. A relatively easy way to add content to the user experience.

And there’s a project called Google Tango, which uses 3D sensors to measure the space around you… making it much easier to put a layer of interactive content around it. Would cost a fortune to develop but this will be shared by Google very soon…

I wanted to talk about conversion iteration – facebook does this all the time because tiny tweaks make such a difference. SkyScanner, for my money the most successful digital company to come out of Scotland, they also iterate. They also do a form of A/B testing. They have a button that gets you to book, and that generates income for them. They started with “book”, they decided to test out “continue”, “select”, “go”. Which do you think led to the biggest improvement. Most of us think the latter…. The actual results…. for “continue” they had +0.95% clicks; “select” had + 6.41% clicks; and “go” saw -1.80% clicks. Now any increase is massive for their income. But I would have guessed, like most of you did, that “go” would be huge. Now you can test this easily… and in some markets they saw a 30% increase for “select”. So testing is so crucial here. Now Skyscanner has a really clear focus on a clear metric. They develop hypotheses from task-based one-to-one user testing. They keep it simple – isolate on A/B at a time. Senior Management respect the data. And they test this stuff with Google Analytics – which is free. And this process makes a huge difference but is ongoing, it needs iterating over time…

So, to summarise… we have issues like HiPPos, we need to be flexible, AdWords are low risk and low cost, interactive experiences can bring digital into a physical space…. and to remember Skyscanner’s approach to A/B testing…

Q1) I suppose you had Google Glasses on there… there is a company bringing out better virtual reality hardware – the Occulus Rift – will that be better?

A1) Really it’s all about content, what’s useful to the user…. don’t overthink it about what’s new and spectacular or cutting edge. You have content. That’s what people are interested in. It’s about how you connect up that digital thing to that real thing. The planning for that Cleveland gallery and that huge interactive wall… they wanted to create interactive experiences in the gallery… but they pulled back to focus on enhancing the real pieces of art. It’s about what layer of content can you apply to enhance the real thing…

So we now have table discussions…

Discussion group two: Evolving and marketing your digital resource

Tessa is outlining our key points for discussion:

  • How will your target audience find out about your digital learning project?
  • Is there anything else that can be done to help people find it?
  • What analytics will you collect? Is there A/B testing you could do?
  • Is there any other information you would like about how your audience responds to your learning project? How might you collect it?
  • How will you use this information? When will you use it? This is the key one!

Again I am on the Purple table…

We are discussing the silos between marketing and other teams…. and also how low conversions may be between social media activity and website information about exhibitions, and then to the exhibitions themselves… but the possibilities of competitions etc. Marketing can have a lot of control over channels which can complicate things… and how to challenge silos… and how to provide guidance to staff…  talk of QR codes and iBeacons (largely used in retail), the idea of something that detects your location that can be used inside… And of the potential unattractiveness of QR codes and potential for crafted objects… and of Augmented Reality (e.g. Layar).

Thinking about A/B testing, analytics… and how people find our stuff. We find that things on EventBrite helps… getting the band out there… interesting stuff like engaging with other cultural events, sitting near other events… if we want a younger audience being grouped by which bands etc. have gigs can be really useful. Affiliating yourself with things that your target audience are interested in. Noting that Tumblr gives a good visualisation of key influencers in material going viral… really useful to see that visually. Also talking about the importance of having something physical in the space that points to the digital… Physical spaces and physical/print materials… and the importance of memorability. But then working out what works…  There are also challenges. We really market stuff we know will sell well sometimes, for complex reasons. And there is audience awareness beyond your control…

Old stuff has value too… especially unique stuff. Make stuff findable for a much longer life… make sitmaps, tag stuff, add sharing buttons, Wikipedia is great for that… connections from there generate lots of referrals. Use unique materials or expertise in your digital resources… that makes a big difference. Exploit what you have.

How do you go further than the obvious local channels? Building your mailing list and audience helps… how do you push beyond that? The App Store is another channel to promote stuff… there is no arts area – there is leisure, travel and education. Can seek reviews etc. and have your team be ambassadors… A lot of this goes right to the top… your programme will attract different audiences in the same way… and appeal to different demographics, keep them aware of other shows coming up… Programming does make a difference, and won’t always align with your target audiences/overarching strategy. Discussions also of deep engagement with the work… that being the goal not necessarily numbers. And how one measures the quality of engagement. Visitor books can help in person…. but so many people don’t fill them in… A way of surveying people or recording that visit. Just writing down good comments etc.  Sometimes this stuff is ad hoc. But you could potentially do that digitally.. people like to see how others respond to the work. I was at the Oceans exhibition at Fruitmarket Gallery – collecting water from the seas, collecting stories alongside. That’s got such huge potential!

Feedback from Groups

Red Table: Had a meandering discussion about what digital learning is. About analytics and Google Analytics. We talked about a project on a closed forum – not as obviously relevant to analytics. We talked about interpretation, and relating everything back to the collection (where you have one). This came out of the Brooklyn Museum aspect,the ideas of the collection as asset… an archive of learning objects, or images, looking at content again.

Blue Table: We mostly ignored the prompts and had a sort of impromptu Google Analytics training… talking about what was possible, what could be done… what could be got in terms of demographics etc. And understanding who is using what and how useful that data might be. And we talked about better targetting online and offline… things like making sure you do promote digital learning adn resources on your website, linking to relevant works, etc. Doing what Google does in offering the right things for what people search for. And also about making sure that things are prominent in the physical space – like use of the Art Hunter logo at exhibition entrances and gallery entrances. Also about A/B testing and Google Analytics… try changing labels perhaps to see if clicks change.

Green Table: We had an interesting conversation. All day we’ve had problems thinking about our audience, and whether we know enough about them. Similarly around analytics – what we collect, how best to do that. Had some really interesting discussions about digital, about the possibilitis of the second skin, if you like, on top of what is already exhibited in collections. Also about if we are using social media and trying to market things… do people who are very busy fail to open emails? Twitter feeds are on our websites… are our websites as effective as they could be… or do we just change one thing on the site to make things better – like the button tweaks. So really how to make exhibitions more personal, more interactive, how do we get their responses? Do we still rely on paper? Is that wrong? Is a piece of paper in the post effective? Should we be doing that if advertising digital learning or resources. David’s point about not making things too difficult… and really think about what your key message is, and your key measures… people said they had used measures but probably not regularly enough.

Purple Table: We had quite a wide ranging discussion as well… augmented reality… tracking locations in other ways too… also how you are set up organisationally… that marketing targets might be different to what you want to achieve, and how you can get around that… we also talked about the importance of the physical space and the linkages around physical and digital resources. For some audiences, particularly older audiences, print can still be important. Talked about digital resources potentially having a longer lifespan – through Google, through Wikipedia, etc. And really thinking about what the unique selling points of your organisation might be, if you have iconic items or key unique expertise, then you have something unique that no-one else has, you can really help get other things out there. And talking about digital stuff – reviews being important for instance. Asking your audience to share and support it and be ambassadors for you, influencers that reach more people… etc.

Yellow Table: Also a wide ranging discussion. Talked about how flexibility to respond sounds good but some concern about timescales. Discussion of social media… could we do it better? Could we do it for each other and help each other? And talking about quite physical digital resources – creating material that can be brought out into the space, not just online. The HiPPo thing struck a chord. Thought analytics and data could be really useful from that regard. Got a bit confessional about making assumptions – and the wrong assumptions – about what people want. Research being important here. Also we may not know what we will change… but we may go back and ask different questions. To find quick wins and small changes.

Panel discussion with speakers (David, Rosie, Tessa)

Image of the Panel Discussion, showing David Craik, Tessa Quinn, Rosie Cardiff

Q1) How have you monitored Art Hunter in the last year? Also how could it be used in events or one off things – not just artefacts

A1 – Tessa) There is an app evaluation package, called Flurry, which allows us to see downloads, usage, how many items they are collecting etc. tines of day being used. But so much data… we needed to find the questions we wanted to ask. We saw a spike when we launched it. We currently have around 100 downloads per month of each version (iOS and Android). And because we are about to redevelop it for GENERATION we have been able to use that data to help us do that. We found, for instance, that 60% of people use it outwith gallery hours… we don’t know why but we can now ask those questions… and we want to also see what we can do that takes it beyond the gallery space to see how to make it more useful. And we have also been doing some research on gallery visitors about whether they have used the app, and how it has impacted their experience.

We have tried to keep the app as open as possible with unlockable content. We have the button called “Extra” – could be any number of things which could include events. We did try using it for two Friends events. But for partner offers our partners wanted to track a lot of what was going on which was harder to do… but for GENERATION we’ll think about that again.

A1 – David) In terms of analytics…. Google Analytics has several thousand measures. We talk about “the critical q” – for any organisation there are key questions – probably three of them – that really matter, and those need to be questions you can actually address. So conversion rate (e.g. for Skyscanner) might be on of those measures.

Q2) Was wondering about that Lets Get Real report – and the key findings there…

A2 – Tessa) that one question “why did you visit the website today?” was just part of the website. Of those only 30% wanted to visit the gallery that day, but our website was so focused on that. There was so much more being looked for… and a need for consumable, browsable data…

A2 – Rosie) We found that we had about 40% international visitors… so they may not step foot in the gallery. Particularly in terms of learning resources they have an interest far beyond those who visit… but is that what your organisation is about? If you are all about attendance and ticket sales that might not be useful to focus upon. To have those priorities…. I think each person in Tate would have their own response. It would be hard to get a consistent organisation-wide view of that. We all want to give a great experience… but we have very different ways to do that.

A2 – David) With Critical Q it doesn’t really matter if they differ… having your three for you to focus on is useful.

A2 – Tessa) One way to do that is to have different dashboards for different parts of the organisation…. to help deal with that.

A2 – Rosie) We are trained in how to use the analytics but… you can do brilliant against your metric but noone else might care!

Q3 – Rosie) One thing about the Agile Marketing… how do you measure the successful things….? We have 1 million Twitter followers…. we aim at a number of retweets… but what are you measuring as success.

A3 – David) Measurement for small scale actions like a post is about engagement…. we use various tools to grab that. Likes, comments or shares. for Tweets it’s replies, retweets or favourites. Poeple can get hung up on it. Reach can be useful… even if people don’t “like” it. Engagement helps us judge things… each sticky in Twillo for us is a theme, we have maybe 10 posts around the themes…. we divide clicks or comments or like by the number of page likes. So you can see the engagement proportionate size (e.g. 0.1% engagement).

Comment) our group talked about success being about getting funding for your next project!

Q4 – Christopher) This morning we talked about copyright and contributions etc. Are there examples of people working around copyright issues…

A4 – Rosie) It can be a real challenge. That can be very tricky to manage. And worse somehow as a Google Image search might well surface materials that an artist’s estate will not permit you to use or share online. At Tate we have to be really really careful about it. We are protecting artists as well as creating a great user experience. It would be good to look at themes that the artist addresses, that their work addresses, and activities for interaction broadly around that…. With Tate we did a kids activity around Lichtenstein but not branded as such, about artwork from dots. But there are things you can do… and thinking around what the artist address. That 1840s room was brilliant that we could just put the artwork out there… if only we could do that for everything… but we can’t! Getting other artists involved – a contemporary artist interested in that work – can be another way…

A4 – Tessa) I think every year the artists are becoming more flexible. When I was working 10 years ago just getting images on the website was difficult, now most artists are keen. Certain artists are particularly complex. There were moves to trademark the name “Picasso” for instance. So you have to think about this stuff first. And then you need to be creative about how address the issue. And if you are shooting video for instance then just including images you are fine to include.

A4 – David) This stuff is a creative challenge. What’s the human angle? Is there a twist? So for the Commonwealth Games we are looking at making a water cooler device with a secret beer dispenser for locals…. so there’s a human interest factor… the human interest is that there is a huge audience coming to Glasgow, will drink lots of beer, and the twist: what if we don’t want them drinking our beer…. and so voice recognition (not too complex at all – a phone, a computer, a human listening in!) will mean that those with a Scottish accent get beer, those without get water…. there’s a human angle and then using the public to give it a twist.

Q5) Thinking about sponsorship, or partnership… any examples of more creative ways to work with sponsors through the digital platforms that you have?

A5 – Rosie) Not particularly about working with sponsors. I’m not sure how interested they are about being collaborative with us…. with technology companies and developers though – as Rohan said this morning – is about a more collaborative aproach. A lot of the funding out there – the NESTA/AHRC fund for digital arts – are looking for more collaborative work here.

A5 – Tessa) We still find it hard to find sponsors for digital projects… maybe we don’t pitch it right, maybe our regular sponsors aren’t interested in digital. When we have had funding often part of the bigger funding process. I think the NESTA thing taught us the benefit of a real relationship with your digital provider. But that’s trickier with procurement processes – already a procurement type relationship. But maybe there is a need to restructure tendering processes. But the NESTA fund means you partner with the provider in the bid. But sometimes the technology provider may also own the idea… can be collaborative though. You don’t put digital to one side once done… you have to iterate.

A5 – David) Speaking from the point of view of brands. Savvy brands know that ads aren’t interesting any more, content is. Most stuff we do has cultural aspect. Markleting spend moving from ads towards content, to jointly created content. Questions here about “selling your soul to the devil” but otherwise huge potential for making content that’s genuinely engaging to audience, to customers, etc. And reach new audiences. But it’s about picking the right partners.

A5 – Rosie) we have challenges with sponsors. BP sponsor a lot of stuff but we have campaigners who oppose that. They look to us to boost their profile, but that also impacts on our too.

A5 – Tessa) And the approval too… everything needs approving for some sponsors. Really challenging for the practical arrangements.

Round up of the day

Christopher is starting to round up the day by saying that this is the sixth event in 13 months with engage. We will send round a link to the video

Sarah Yearsley, engage 

Keywords I found interesting today:

  • iteration – and the process of developing content online
  • play – really important
  • content
  • experimentation – liked hearing this. A lot of what loearning people do is about experimentation
  • failing small – making small changes and building upon them, to make bigger changes perhaps
  • be clear, think carefully – think about the benefits of what you do
  • unexpected outcomes
  • sharing models – engage, ARTIST ROOMS, and GENERATION really help here
  • digital space to meet – not just about content and structures, but spaces for people to meet
  • hidden content – loads hiddden on our website… great learning resources but can be hard to find. maybe where A/B testing comes here
  • digital fear – maybe still some of that but if we start small we can then think bigger as useful…

Sarah Yearsley, engage

Comment from David: the digital fear thing… the biggest challenge is the creativity, the ideas for content…. ironically people in this room are great for the hard bit, the content, the creativity… the technology is easier by comparison!

Back to Sarah:  And I will say what I say at the end of every event: There are some evaluation forms on this table. Please fill them in and let us know what you think. Either on paper or via SurveyMonkey.

Christopher: this event was a wee bit different to the london event because of feedback of wanting more discussion.

Sarah: the next engage event is at the Scottish National Portrait Gallery. Where we will hear about how we engage with audiences away from the gallery space – in the community, through games, etc. Also a free event. Please do book and come along – 23rd April.

Finally thank you all for coming, thank you to the galleries for hosting, thanks to Nicola for blogging, and finally thanks again to all of our speakers!

And with that we are done….

Find Out More

Related Resources

The delegate pack included a wide range of resources which will be of interest both to those who attended the event, and those following the blog (only). “These online resources were suggested by contributors as background reading and reference for the seminar. The selection aims to offer some starting points; from projects that contributors have found useful and inspiring, to details of many of the projects referenced by contributors in their presentations.

Mar 102014
 
Jisc Digital Festival - watch live (inspired by flickr.com/photos/jdhancock). ©Jisc and Matt Lincoln (www.mattlincolnphoto.co.uk)

A brief post to let you know that on Tuesday 11th and Wednesday 12th March myself and various EDINA colleagues will be taking part in the Jisc Digital Festival 2014.

I will be livetweeting throughout the event – you can view all the tweets on #digifest14 and you can also view a stream from the event via the Jisc website. There will also be materials shared on that site following the event – including my own (see also below).

I will also be running a social media surgery on Wednesday 12th March (9.30am in the Chill Out Lounge) – if you have questions you’d like answered then do come along or tweet them to me. Even if you are not along in person, I’ll do my best to tweet back an answer ASAP!

The full programme of EDINA participation in the event is:

 Tuesday 11th March 2014
11:30-12:15 Increasing the offer to FE Surgery (Chill Out Lounge) Speakers include: Anne Robertson and Conor G. Smyth, EDINA
All Afternoon Going beyond Google (1): content-rich mapping for the classroom and the field Tech demo (Hall 3 Gallery, Demo Pod 3) Addy Pope, EDINA
All Afternoon Going beyond Google (2): using the right media Tech demo (Hall 3 Gallery, Demo Pod 3) Andrew Bevan, EDINA
14:30 – 15:15 Location aware apps: design patterns and solutions surgery Surgery (Executive room 2) Ben Butchart, EDINA
Wednesday 12th March 2014
09:30-10:15 Increasing the offer to FE Surgery (Executive room 2) Speakers include: Anne Robertson and Conor G. Smyth, EDINA
09:30-10:15 Social media best practice surgery Surgery (Chill Out Lounge) Nicola Osborne, EDINA
9.30am and 10.30am Fill your repository from around the world: Repository Junction Broker (RJB) and its potential to increase open access content in your institutional repository Tech demo (Demo Pod 2) Muriel Mewissen, EDINA
9.30am and 10.30am Going beyond Google (1): content-rich mapping for the classroom and the field Tech demo (Demo Pod 3) Addy Pope, EDINA
10am and 11am Going beyond Google (2): using the right media Tech demo (Demo Pod 3) Andrew Bevan, EDINA
11:00 – 11:30 The strategic developer: a new role for HE? Expert speakers (Hall 10a) Paul Walk, EDINA
14:45-15:30 Stronger together: community initiatives in e-journal management Panels Speakers include: Peter Burnhill, and Adam Rusbridge, EDINA

 

Materials from the Social Media Best Practice Surgery

My session was a surgery so I based the format on an open discussion and question and answer session. There was no central presentation as such, but I did create a brief prezi as a jumping off place for discussing topics in more depth. The prezi links to other presentations and materials and can be found here:

http://prezi.com/o2wchskexxdm/jisc-digital-festival-2014-social-media-surgery/

I also produced a resource lists which you can download as either a PDF or a .doc. I am happy for anyone who wishes to edit/update and reuse at their own institution to do so if they would like.

 March 10, 2014  Posted by at 11:23 pm Events Attended, Social Media at EDINA, Week In the Life Tagged with: ,  No Responses »
Feb 242014
 

This afternoon I will be liveblogging the MOOCs in Cultural Heritage Education event, being held at the Scottish National Gallery of Modern Art in Edinburgh.

As this is a liveblog please excuse any typos and do let me know if you spot any errors or if there are links or additional information that should be included. 

Our programme for today is:

Welcome and Intro – Christopher Ganley (ARTIST ROOMS, National Galleries of Scotland and Tate)

Image of Christopher Ganley (National Galleries of Scotland) Christopher is the learning and digital manager for the National Galleries of Scotland and Tate. In case people here don’t know about Artist Rooms, this is a collection that came to Tate and NGS in 2008. Around 1100 items of art from Anthony d’Offay with the National Heritage Memorial Fund, the Art Fund, the British and Scottish Governments. The remit was to be shared across the UK to engage new audiences, particularly young people. The collection has grown to around 1500 items now – Louise Bourgeois is one of the latest additions. The Artist Rooms Research Partnership is a collaboration between the universities of Glasgow, Edinburgh and Newcastle with Tate and NGS led by the University of Edinburgh. And today’s event is funded by the Royal Society of Edinburgh and has been arranged by the University of Edinburgh School of Education as part of the outreach strand of their research.

Year of the MOOC?: what do Massive Open Online Courses have to offer the cultural heritage sector? – Sian Bayne, Jen Ross (University of Edinburgh)

Sian is beginning. Jen and I are going to situate the programme today. Jen and I are part of the School of Education working in Digital Education, and we are ourselves MOOC survivors!

Image of Sian Bayne (University of Edinburgh)We are going to talk about MOOCs in a higher education context, and our research there, and then talk about what that might mean for museums and the cultural heritage context. Jen will talk about the eLearning and Digital Culture MOOC and expand that out into discussing cultural heritage context.

So, what do we know about MOOCs? It’s a bit of a primer here:

  • Massive: numbers. Largest we ran at Edinburgh had 100k students enrolled
  • Open: no “entrance” requirements.
  • Online: completely.
  • Course: structured, cohort-based. And we don’t talk about that so much but they have a pedagogy, they have a structure, and that distinguishes them from other open education tools.

In terms of where MOOCs are run we have EdX – they have no cultural heritage partners yet. We have Coursera and they do have cultural heritage partners including MOMA. And FutureLearn who have cultural heritage partners yet (but not who are running courses yet).

The upsides of MOOCs is that they have massive reach, a really open field, high profile, massive energy, new partnerships. But on the downsides there are high risks, there are unproven teaching methods – and the pedagogy is still developing for this 1 teacher, 20k students kind of model, and there is a bit of  a MOOC “backlash” as the offer begins to settle into mainstream after a lot of hype.

In terms of cultural heritage there isn;t a lot out there, and only on Coursera. American Museum of Natural History, MOMA, California Institute of the Arts and the new Artist Rooms MOOCs are there. Some interesting courses but it’s still early days, not many cultural heritage MOOCs out there.

So in terms of the UK Jen and I have just completed some research for the HEA on MOOC adoption. One aspect was which disciplines are represented in UK MOOCs. We are seeing a number of humanities and education MOOCs. FutureLearn have the most of these, then Coursera and then there are cMOOCs in various locations. In terms of the University of Edinburgh we launched our first MOOCs – 6 of them across 3 colleges – last January and were the first UK university to do so. This year we have 7 more in development, we have 600k enrollments across all of our MOOCs and sign ups for the Warhol MOOC is well past 10k already.

So why did we get involved? Well we have a strong and growing culture of digital education,. It was an obvious for us to take that step. There was a good strategic fit for our university and we felt it was something we should be doing, engaging in this exciting new pedagogical space. Certainly money wasn’t the motivator here.

MOOCs have been around for a while, and there is still some things to learn in terms of who takes them, who finishes them etc. And we’ve done some research on our courses. Here the Philosophy MOOC saw over 98k students but even our smallest MOOC – equine nutrition- saw a comparable number of registrations to our total on campus student body (of approx 30k). Of the 309k who enrolled about 29% of initially active learners “completed” with a range of 7 – 59% across the six courses. We think that’s pretty good considering that only about a third of those who signed up actually accessed the course – of course it’s easy to sign up for these and hard to find time to do them so we aren’t worried about that. The range of completion is interesting though. We had 200 countries represented in the MOOC sign ups. And age wise the demographic was dominated by 25-39 year olds. And we found most people who took the MOOCs, at least in the first round, mostly had a postgraduate degree already. They were the people interested in taking the MOOCs. And now over to Jen…

Image of Jen Ross (University of Edinburgh)Jen. I want to tell you about the experience that lecturers and tutors had on the eLearning and Digital Cultures MOOC that took place last January. Firstly I wanted to talk about the xMOOC and the cMOOC. the xMOOC is the highly structured, quite linear, institutional MOOCs – the Coursera or FutureLearn model. Some peer interaction, but as a side benefit of the content as the main thing. Teacher presence in these sorts of MOOCs tends to be very high profile – the rock star tutor concept. You won’t meet them but you’ll see them on video. A lot. The other sort is the cMOOC, the connected MOOC. these were thought of by Canadians in 2012/13 before MOOCs became built. Around the theory of connected environments, participants create the course together, very loosely structured, very collaborative, very focused on participant contributions. Not about the rock star professors. This difference has been quite a big press thing, xMOOCs have had a bashing, people suggesting they are “elearning from 1998 minus the login button”. But actually what Sian and I have been finding is that in ANY MOOC we see much more than these two different forms. Our own MOOC is really neither an xMOOC or a cMOOC but had a lot of other content.

So our MOOC, #EDCMOOC, was based upon a module of the MSc in Digital Education module that generally has about 12-16 participants, and instead trying these ideas about the self in online environment in a MOOC format, at huge scale. So we decided rather than doing week by week lecture heavy format, we would do something different. Instead we did a “film festival” – clips for participants to watch and talk about. Then some readings on theory of digital education. And questions to discuss. We asked students to create public facing blogs which we linked to, we also used the built in discussion spaces. And instead of weekly tests etc. we had a single peer assessed “digital artefact” final assignment.

We gathered all blogs, which they had registered with us, in one place – so you could see any post tagged with #EDCMOOC. And we had a live hangout (via Google+ / YouTube) at the end of every few weeks – and we would pick up on discussions, questions that were coming up in those discussions and coming in live. The students themselves (42k of them) created a Facebook Group, a G+ group, used the hashtag but also these additional groups meant there was so much material being produced, so much discussion and activity beyond a scale anyone could keep up with. A hugely hectic space for five weeks, with everyone trying as best they could to keep an eye on their corner of the web.

Bonnie Stewart described our MOOC as “subverting it’s own conditions of existence”. And it was a chance to rethink that xMOOC/cMOOC divide. But also what the teacher is in a MOOC. What it means pedagogically to be in a MOOC. There are interesting generative questions that have come out of this experience.

So, I want to show you some examples of materials participants made on the MOOC. Students shared these on Padlet walls. We also had an image competition halfway through the MOOC. e.g. “All Lines are Open” by Mullu Lumbreras – the Tokyo underground map re-imagined with many “You are here” markers – emphasizing the noisiness of the MOOC! There were many reflective and reflexive posts about students trying to get to grips with the MOOC itself, as well as the content. There was such a variety of artefacts submitted here! There were images, videos, all sorts of assignments including super critical artefacts, such as Chris Jobling’s “In a MOOC no-one hears you leave” – although interestingly we did. There was also a chatbot assignment – allowing you to talk to “an EDCMOOC participant” and used comments from chats and from the course to give back comments, really interesting comment on the nature of the MOOC and the online environment. We also had a science fiction story all created in Second Life. This must have taken such a lot of time. We have found this on the MSc in Digital Education as well that when you give people the opportunity to create non textual assignments and contributions they give such creative and take such a lot of time over their multimodal work.

We also had  – a nod for Artist Rooms colleagues – a Ruschagram tool as an assignment. And indeed people used their own experience or expertise to bring their own take to the MOOC. Artists created art, scientists drew on their own background. Amy Burbel – an artist who does lots of these online videos but this one was all about the EDCMOOC.

Image of Jen Ross and Sian BayneSo I’d like to finish with some ideas and questions here for discussion… Elizabeth Merritt from the Centre for the Future of Museums asks about MOOCs in terms of impact. Rolin Moe talks about MOOCs as public engagement on a different scale. Erin Branham asks about reach – why wouldn’t you run a MOOC even if only 20k people finish. We have comments on that actually… David Greenfield emphasises the innovation aspect, they are still new, we are still learning and there is no one single way that MOOCs are being used. There is still a lot of space for innovation and new ideas.

Q&A

Q1) I work at the Tate in visual arts, the idea of assessment by multiple choice is very appealing so I wanted to ask about peer assessment. How did that work? Did there need to be moderation?
A1 – Jen) It is quite controversial, that’s partly as the MOOC platform don’t handle peer assessment too well. We didn’t get asked too much to remark assignments. Peer assessment can work extremely well if the group know each other or share a common understanding.

A1 – Sian) It was strange how assessment focused many people were for a non credit bearing course though, they wanted to know how to pass the MOOC.

Q2) I wanted to ask about the drop out which looked absolutely huge…

A2 – Sian) You mean people who didn’t begin to engage with the MOOC? It is problematic… there has been a lot of criticism around drop outs. But we have been looking at them from a traditional education point of view. MOOCs are free, they come in, they sample, they leave. It’s about shifting our understanding of what MOOCs are for.

Q2) What did you learn from that…?

A2) I think it would be too hasty to make too many conclusions about that drop off because of what it means to be in a MOOC

A2 – Jen) there is some interesting research on intentions at sign up. Around 60% of people signing up do not intend to complete the MOOC. I don’t think we will ever get 90% retention like we do on our online MSc. But Sian’s point here holds. Different demographics are interested for different reasons. Retention on the smaller equine science MOOC was much more about the participant interest rather than the content or pedagogy etc. The 7% retention rate was the more innovative assessment project.

Q3) We would love to have that data on drop outs. We aren’t allowed to fail at that rate in public. I work in the National Library of Scotland and we know that there is “library anxiety”.  I would hate to think this is a group with inflated library anxiety!

A3) Absolutely and I know there will be more on this later on. But its about expectation setting within the organisation.

Q3) Just getting that data though – especially the research on those who don’t want to complete – would be so valuable for managing and understanding that completion in open contexts.

Q4) Perhaps the count should be from the first session, not from those who sign up. It’s not the original email we are concerned with but the regular drop out which would be more concerning. We get people doing this with on site free experiences. This is more about engaging with the higher up decision makers and marketing about how we could use MOOCs in cultural heritage.

A4 – Sian) It was unfortunate that many of the MOOCs really marketed sign up rates, and inflated expectations from that, as a way to promote the MOOCs early on. Very unhelpful to have messages like “we want this one to hit a million sign ups!”

Q5) These aren’t credit bearing but are there MOOCs which are, how do they work?

A5 – Jen) Quite new territory. Some allow you to have some sort of credit at the end of the MOOC on payment of a fee. And some – including University of Central Lancashire – are trialling MOOC credit counting for something. Work at European level there too. But no one has cracked the magic bullet.

A5 – Sian) Two offering credit so far – one at Oxford Brookes, one at Edge Hill.

Q5) Maybe credit will appeal to those currently absent from the demographic profile – moving to those with few or no higher level qualifications

A5 – Sian) we did ask people about why they did the MOOC, many for fun, some for professional reasons. none for credit.

Q6) what are the indirect benefits of the programme?

A6 – Sian) We have had five or six people enrolling on the MSc as a direct result of the MOOC. We also got great publicity for being at the forefront of digital education which is great for the University. That indirect benefit won’t last of course as MOOCs get more mainstream but

A7 – Sian) 40 days academic staff time to develop, 40 days to deliver it. And that doesn’t include the Information Services staff time to set up the technology, In terms of participants I’m not sure we have that data

A7 – Jen) We kind of have it but it’s taking a long time to analyze it. You get a lot of data from the MOOCs. There is a whole field of learning analytics. We have the data from both runs of the MOOC but it’s hard to find the best way to do that.

Q7) Interesting, for people reflecting on their own time investment

A7) We gave guide time of 5-6 hours per week for the basic involvement but actually many people spent a lot of time on it. And there was a lot of content so it took that long to read and engage with it for many participants.

Q8) How do you assess 40k people?

A8 – Sian) Well that’s why we spent a lot of time trying to make the assessment criteria clear for people marking each other.

Q9) Can you say a bit more about xMOOCs and cMOOCs. A lot seem to be xMOOCs?

A9) There is a lot of discussion around how to go beyond the bounds of the xMOOC.

A9 – Sian) Our MOOC was seen as quite innovative as we were a bit of a hybrid, but a lot of that was about participants using social media and just having a hashtag made a difference.

Q9) So are there people trying to move out of the platform…

A9 – Jen) for the credit and microcredit courses you try to bring students into the MOOC platform as that is easier to measure. And that’s an area that is really becoming more prominent…

A9 – Sian) Would be sad is the move towards learning analytics took away the social media interactions in MOOCs.

A9 – Jen) We do see AI MOOCs where there is some opportunity to tailor content which is interesting…

Comment) Can see these working well for CPD.

:: Update: Jen and Sian’s Prezi can be viewed online here ::

The changing landscape of teaching online: a MoMA perspective – Deborah Howes (Museum of Modern Art)

It is a pleasure for me to tell you just a little bit about what has been going on at MOMA, especially having to spoken to just a few of you – I realise you are very savvy digital education, cultural education audience.

I like to start with this slide when I talk about online learning at MOMA – of MoMA education broadcasts in the 1950s. We have always been interested in technology. It is part of our mission statement to educate (the world) about the art of our time. This image is from the 1950s when MoMA had an advanced idea of how to teach art and creativity – and they invited TV crews in from Rockafeller Centre to record some of what was going on in terms of that education.

So online learning for MoMA can be as something as simple as an Online Google Hang Out working with seniors who go on a field trip once a month without them having to leave their apartment – they have a museum visit and discussing the art. Some have mobility issues, some have learning disabilities. But they have these amazing opportunities to visit and engage all the time for free. We use Google Hangouts a lot and this is an example that really hits home.

Image of Deb Howes (MoMA)

This example, like much of what I’ll talk about today, isn’t strictly a MOOC but it’s from that same open online concept and the MOOC is changing. However we have, at MoMA been running online courses since 2010. These are NOT MOOCs as we charge for them. You can take them in two ways. You can be self led and there is no teacher responding to you and there are no students but you go at your pace whenever you want. Or you can do the teacher led version with a teacher, with fellow students, with responses to your comments. We started the concept of starting these courses. We did this with Faith Harris, who now works at Khan Academy, and she was teaching online in the New York Museum of Fashion. She had a clear idea of what the format was – a structured course led by an educator. We did a studio course – how to paint – to see if that would work. That seemed such an usual idea at the time but they are really popular, especially as an instructor led experience. They like to see and share progression and to get feedback on that. Just like a real studio experience. So the “how to” videos, one of the things we tried to replicate online was the feel of exclusivity you have in an on-site course. If you enrol in person you get to paint in our studio then you get access to the galleries when no-one else is around. So here we have Corey Dogstein and he’s also an artist, the students love him, but you can see this video of how to paint like Jackson Pollock and really get into that free form, jazz playing vibe.

My previous role I came from a gallery where I had no idea who was doing my tour, or what they were getting from it, then I was in an academic place where I knew who everyone was, how they were progressing, assessing them etc. So in this role the online teaching experience has been really interesting. In particular taking out the temporarility and those barriers to speak up, you open up the accessibility to a much much wider audience. The range of learning difficulties that students come in with and feel able to participate online, that wouldn’t feel able to participate as fully in person is striking.

We use a course management system called Haiku. No matter what you do it looks like a bad high school newspaper. It organises content top to bottom, welcome messages, etc. 60% of our students to the MoMA online course have never taken an online course before. They tell us they’d rather try it with us! We have a lot of first timers so we have to provide a lot of help and support. We try to make them engaging and lively. The upside of the highly controlled space is that the teachers themselves are making these courses, it’s easy for them to change things, that’s the upside.

We try to think thematically about content, rather than thinking academically along a timeline say. So colour as a way to explore modern art came to mind, and also broadens the base beyond painting and sculpture – design and architecture for instance. So this way we can interview the curator of design, Paula Antonelli, on colour in design. [we are watching a clip of this]. Talk about exclusivity! Even on my 11 o’clock tour I couldn’t get you time with Paula. The students really respond to this. And we also created videos of the preservation techniques around colour.

This course: “Catalysts: Artists creating with sound, video and time” brings all those ideas together, and is a hybrid xMOOC and cMOOC although I only just realised this! We got the author Randall Packer to put this history together using artefacts and resources from MOOCs. It’s so hard to do this history – why read a book on the history of video artworks?! As an educator how many museums have the space to show a whole range of video art? Even at the new Tate underground you have a rotating collection. Rare to have an ongoing historical way to explore these. One of the reasons MoMA was able to jump into online courses feet first, is that Volkswagen are a corporate sponsor of the galleries and were keenly supportive. And as part of teaching the Catalyst course Randall, who is also a practicing artist, thought it would be great if we could get students to make and share work, wouldn’t it be great to make a WordPress blog they could use to share these and comment on each other. And my colleague Jonathan Epstein suggested digital badges – they get a MoMA badge on their blog and badges for LinkedIn profiles etc.

So, over three and half years we’ve registed about 2500 students. Small versus MOOCs but huge for us. Around 30% of enrolees are not from the US and that 30% represents over 60 countries. For us it was about engaging people in a sustained way with people who couldn’t come to MoMA or couldn’t come often to MoMA, and we really think we’ve proved these. This is one of those pause moments for us… so, any questions…

Q&A

Q1) That quote on your slide “the combination of compelling lectures with the online galery tours and the interaction with the other students from around the world was really enlightening and provocative” – what do you learn from these participants?

A1) We do find students who set up ongoing Facebook groups for instance, and they are really active for a long time, they will go on a trip and write to their peers about what they’ve seen. We learn whilst they take the course, but also over time. What is so hard for museums to learn is what the long term impact of a museum visit… there is no way to know what happens months or years later, or when they are at another gallery… But you get a sense of that on the Facebook groups.

Image of Deb Howes (MoMA)

Q2) At the moment it’s $25 to come into MoMA. How much are the courses?

A2) It is. But it’s a sliding scale of prices. For self-led courses… 5 weeks is $99 if you are a member. or $150 for a non member (of the museum) 10 week course. For instructor led it’s $150 to $350 per course depending on time etc. They may fluctuate, probably go down. I like the idea of a cost recovery model. Free is hard for me as instructor. But there is a lot of free stuff, and especially in the MOOC world, they are comparing what’s available, what the brand is worth, which is worth doing.

Q3) Member?

A3) Of the museum. Typically at the museum you get lots of discounts, free entry etc. as part of that. I think it’s about $75 for an individual membership right now and that’s part of a wider financial ecosystem I don’t get into too much.

So… we have all these courses… We got contacted by Coursera who said “oh sorry we can’t take your courses as you don’t award degrees” but here is a sandbox for K-12 for you. In fact MoMA does a huge amount for teachers. We had just done a huge new site called MoMA Learning with resources for all sorts of classes. So we thought, well this will be our textbook essentially. If we leave it there we don’t need to renogiate all the content again. So we decided to do a four week “art and inquiry” MOOC. There is a huge focus in the core curriculum on discussions around primary source materials, we do a lot of training of teachers but we can’t fit enough of them in our building. We have taught a class for teachers around the country, perhaps beyond, who come for a week in the summer and talk about inquiry based learning. It just so happened when this came together that we were the first MOOC in the primary and secondary education sandbox – I think that has everything to do with why we had 17k ish participants. We had a “huge” engagement ratio according to Coursera, they told us we were off the charts – people are watching the videos “all the way to the end!”. Huge validation for us, but if you think carefully about all the ways people are learning that satisfy them, people look for something to engage with – and museum educators are great at this, great at finding different ways to explain the same thing.

At the end of the course we had a survey. 60% were teachers. The rest were taking the course for different reasons – doctors wanting to talk about x-ray results better with patients. 90% of all those who answered the survey had not been to MoMA or had an online MoMA experience but they did visit the website or site afterwards. We had more friends, we had people following and engaging with our social media. It was a wonderful way to have people access and engage with MoMA who might now have thought to before.

So I have a diagram of MOOC students. It is kind of Ying-Yang. The paid for courses tend to be my age or older, highly educated, have been to many international galleries. Coursera they are 20-30 year olds, it’s about their career, they take lots of Coursera courses. And what struck us was that putting our content beyond the virtual museum walls, people really want to engage with it. In the museum we want people coming to us, to speak to us, but here they don’t visit us at all but they still want to engage.

We had 1500 students get a certificate of completion. In MoMA we have 3 million admissions per year. I have no idea how many take that information with them. For me as a museum professional 17k people made an effort to learn something about MoMA, word is out, and I taught 1500 teachers in the way I would like to in an academic way, and I taught more than I could teach over three years, but in one single summer. And the success of that means we have followed up with another MOOC – Art and Activity: Interactive Strategies for Engaging with Art. The first one runs again soon, this new course runs from July.

There are a few other things we do online… MoMA Teens Online Course Pilot. This was a free 5 week course in art appreciation at MoMA. These were teens that had taken probably all our teen courses as part of after school programmes. They brought back to us this Real World MoMA episode. [very very funny and well full of art in-jokes].

You get the idea right? I should just let the teens do all the videos! We have a new group of teens coming in doing a completely different thing. This is their medium, they understand. They combine the popular with the collection in an unforgettable way, the kids will never forget these five artists they focused on.

I just want to go through some pedagogical background here. There is a huge body of really interesting reseach on how the brain works, what makes memories… One of the things I always try to think about is what makes your brain remember, and why a museum is such a great way to learn. So one thing that is that you learn when something new comes in – a new sight, a new sound, a new smell… Museums are like that. They are new experiences. For children they may never have been to a museum or even to the city before. I try to make the online courses take that into consideration. How can we do that, and make the brain hold on to what it being learnt?

I don’t know if Howard Gardner is familiar to you? His ideas that different brains work differently, and that we need to present material in different ways for different people. We have hands on aspects. We have scientist experts, we have critics… we try to present a range of ways into the material.

So here also is some student feedback – the idea that there is more in the course than can be absorbed but that that is a good thing. We also try to ensure there are peer to peer aspects – to enable sharing and discussion. So here we have the learning communities from that studio course – where participants share their art… increadible learning experiences and incredible learning communities can exist beyond the museum and beyond the university but it is great to be there to support those communities – to answer questions, share a link etc.

I wrote a post you might like: moma.org/blog search for “how to make online courses for museums”

Moving forward we have a couple of hundred videos on YouTube but we were asked if we would put these into Khan Academy. We filtered the best down, gave them embed codes, and they have created a structure around that. As a museum you don’t have to do everything here, but reusing is powerful.

And moving forward we are doing some collaborations with the University of Melbourne.

And my forcast for Museum-University Partnerships forecase? Sunny with a chance of rain! There are real challenges around contracts, ownership etc. but we can get to a place of all sunny all the time.

Q1) We would be developing online learning as a new thing. When you decided to go down the online route did you stop anything else? Did you restructure time? How does that fit with curator duties?

A1) We didn’t drop anything. The Volkswagen sponsorship allowed us to build the team from myself and an intern to include another individual. But it’s a huge time commitment. Curators don’t have the time to teach but they are happy to talk to camera and are generally very good at it. I was at John Hopkins, and previously to that at the Metropolitan Museum… I was used to having media equipment to hand. There wasn’t that at MoMA but we created a small studio which makes it easy for curators to pop in and contribute.

Q2) Could you say a bit about the difference of practical versus appreciation type class?
A2) for practical classes the key is *really* good videos. Being able to replay those videos, if shot well, is really helpful and clears up questions. It lets them feel comfortable without asking the teacher over and over again. If you’ve ever been in a group critique that can be really intimidating… turns out that the level of distance of photographing your work, post online, and discuss online… students feel much better about that. There is distance they can take. They can throw things at the wall at home as they get critiqued! It is popular and now online you find a lot of low price and free how to courses. But our students who return it’s about the visits to the gallery, the history of the gallery, connecting the thinking and the artwork to the technique

Q2) So unspoken assumptions of supplies available?

A2) No, we give them a supply list. We tell them how to set up a studio in their own bedroom etc. We don’t make assumptions there.

Beyond the Object: MOOCs and Art History – Glyn Davis (University of Edinburgh)

Our final speaker is one of the “rock star lecturers” Jen mentioned!

So, in comparison to the other speakers here the course I have been preparing has not yet run. We have just under 12000 signed up so far, we anticipate around 20k mark. I am an academic and I teach film studies, particularly experimental cinema. A lot of the films I talk about it can be hugely hard for people to get hold of. That presents massive difficulties for me as a researcher, as a writer, but also for these sorts of learning experiences.

Where I want to start is to talk about Andy Warhol. A book, Warhol in Ten Takes, edited by myself and Gary Needham at Nottingham Trent University. We start with an introduction about seeing a piece called “does Warhol make you cry?” at MoMA – and he was at the time. So many rights to negotiate. That book is solely about Andy Warhol’s cinematic work, focusing on 10 films in detail. Those that are newly available from the archive, those where there was something new to be said. He only made films for five years – making 650 movies in that time. A lot even in comparison to Roger Corman (5 a year or so). Some are a few minutes long, some many hours. The enormous challenge was that in 1972 Warhol took all of his films out of circulation – he wanted to focus on painting, he was getting sued a lot by collaborators who wanted money from them. And they remained that way. Just before his death he said “my movies are more interesting to talk about than they are to watch”. He may have been joking but that sense has hung around studies of his work. Take a film like “Empire” (1964) it’s a conceptual piece – 8 hours and, in terms of content, time passes and it gets dark – has been little shown. Very few of his films are in circulation. MoMA has around 40 circulation copies available but that’s a rare place you can see them, you can see screenings at the Celeste Bartos screening rooms. The only other place to see them is at the Warhol museum in Pittsburgh on VHS. If not that its 16mm. You can’t pause or rewatch. It’s cold. It’s really hard to do Warhol research… so many pirate copies also out there…

12751417225_a7c167590f_z

So are his films worth seeing or are they just conceptual pieces? Since the films have started to come out of the archives films like Empire have been shown in their entirity… people then discuss the experience of sitting through all of them. Indeed in his PhD thesis (Motion(less) Pictures: The Cinema of Stasis), Justin Remeselnik suggests they are “furniture films” – you can admire and engage with them but not to be paid attention to for an increadibly long time… and yet in Pamela Lee’s book Chronophobia talks about seeing Empire the whole way through, as a phenomenological record of pain it’s fairly incredible. She’s not alone here… another writer, Mark Leach, asked an audience to provide live tweeting during a screening of Empire, and then compiled these into the book #Empirefilm.

This is a long diversion but… Gary Needham and I tried to think hard about the experience of the Factory and the working environment there, what was it like to see Warhol’s films in the context of other experimental filmmakers in the 1960s. In trying to put together a MOOC these ideas sat with me, as the rights negotiations for the book took place over 18 months. We had 30 new images created – we had to apply for grants to get these made, rather than reproduced – by the Warhol museum. We had materials from BFI. We were able to use publicity materials as well. And we had to get agreements from so many people. The Whitney Museum has a Warhol Film Project and acted as our fact checker. It’s a 500k word book so that took some time. One of Warhol’s assistants, Gerard Malanga, allowed us to use his diary entries in the book. I came to Warhol knowing the rights access issues. And I came to the MOOC knowing those issues, knowing the possible time lag…

Chris provided a great introduction to Artist Rooms earlier. I head up the Art and it’s Histories strand. Sian and Jen head up the education strand but I work with artist historians and theorists doing research projects around the materials. So making a MOOC was an idea we thought about as a way to bring out Warhol to a wider audience, and to highlight the Artist Rooms content. I had a lot of questions though and I knew we could not use moving images at all. Could we talk about Warhol’s work without images or clips? What does that mean? Can we assume that people taking the course might source or be able to watch those things. I’ve been teaching Warhol for 15-20 years. I can show all manner of images and clips to students for teaching which are fine to use in that context but which would be impossible to use online for copyright and provenance reasons.

So, there are roughly 250 Warhol pieces in the Artist Rooms collections. There are particular strengths there. There are a great number of posters, as Anthony d’Offay said to me, these give a great overview of events during his lifestyle. There are also stitched photographs – another strength – and these are from the end of Warhol’s career. There are not many so to have a number to compare to each other is great. There are also early illustrations and commercial works. And there are self portraits from the early to mid 80’s. So for me how do I put together a course on Andy Warhol based on this collection? His most famous work is all from about 1962 to 1966. These pieces are silk screens of Monroe, Electric chairs, guns, Campbells soup cans. They are hugely expensive and not in the collection. But are these so familiar that I can assume those taking the course will know them. But the other partners in Artist Rooms – from the National Galleries of Scotland and the Tate – that did cover some of this famous 1960s material, to sex up the course a bit!

So this let us take shape. This will be a five week course. Each week will be a video lecture from me (sex, death, celebrity, money, time) and then a video interview who have worked with Warhol’s work in one way or another – curators, academics, conservators etc. Who could give a fresh perspective on Warhol and what he means to them. I’ll come back to them shortly.

I’ve talked about Warhol’s ubiquity and that’s been an issue as we finalised materials, looked at editing videos. Warhol is one of the most well known artists in the world. His images circulate so widely on such a range of objects (maybe only exceeded by the Mona Lisa) that familiarity with them is high. You can buy just about everything – from mugs to skateboards… the Warhol story is extraordinary. What’s really interesting for anyone teaching art history or theory is that he provides a really interesting test case with regards to reproduction and distribution.

For instance the Marilyn Diptych ( Andy Warhol, 1962). This was based on a publicity still for the 1953 film Niagara which he cropped to his liking. He started to make works just after her suicide in 1962. They have been described as work in mourning. And they are important examples of pop art, collapsing the worlds of art and pop culture. But also commenting on the mass media reproduction of imagery. The uneven application across this piece suggest the blurring of images in newspapers, and the important difference between similar reproductions. Thomas Crow (in his essay for Art in America (May 1987), “Saturday Disasters: Trace and Reference in Early Warhol”) writes that Marilyn disappears quickly when you look at this work, what becomes clearer is the blurrings, the paint level variations. But I have been using this image to teach with Walter Benjamin’s essay on mass production in relation to art work. His essential argument is that endless reproduction, owning of facsimiles etc. changes our relation to the original. It could seem less valuable… or more valuable… as we have seen with Warhol’s work. And Warhol’s own work is a reproduction itself of course. And his painting is the valuable thing… not the press still…

Being able to talk about this work and reproduction through the MOOC and the digital format adds another layer. MOOCs raise the question of what the use of gallery visits may be. What’s the difference of talking about a work and engaging with the original piece. The process of art or art history has always involved travel to galleries, biennials, festivals. Writing about it means seeing the work, there are financial angles there, there are green angles there. For example I am going to Newcastle for three days to see “Crude Oil” (Wang Bang, 2008). It is a 14 hour movie, you can only see it in installation. I intend to move in… my husband thinks I’m mad!

And what about the experience of engaging with the stuff here. I spent three days at the Warhol Museum in Pittsburgh preparing for the MOOC watching to VHS, speaking to staff, and also looking at Warhol’s “time capsules” – receipts, ephemera, e.g. a box from 1978 is just “Concorde stuff”. I was accompanied by a curator, they opened boxes for me… some smelled bad due to moldy stuff, exploded soup cans, a still-inflated silly birthday cake which was a present from Yoko Ono. They are treated as art works. They are still cataloguing these things. So I spoke to the curators about how they are making the time capsules educationally engaging. They have video of celebrities going through them, for instance John Waters gives a great critique of one of the time capsules. They did a live opening, streamed to the ICA, of one of the time capsules. I mention these because these were really interesting examples of opening this type of content and artist up to others.

Let me just say a bit about how we have made the videos for the MOOC. My colleague Lucy Kendra who had filmed other MOOC content saw this filming experience as unusually immediate and intimate in form. We spoke to curators and conservators at the galleries, Gary at Nottingham, and Anthony d’Offay himself. We were also given access behind the scenes at the Tate Store – they took out 10 pieces as a backdrop which was so valuable. We had interviews of an hour, an hour and a half. We have so much materials. For the Warhol class there will be a required 10 minute version of the video, but we will then give a longer, possible unexpurgated, videos for those that want to see them the whole way through. These are fantastic and extraordinary videos. I think they are fantastic representations of these institutions but I think it may open the doors to careers in some of these roles. We hope they may open doors in ways other art education courses may not do.

These interviews I could not have forseen, but they have become the bedrock of the course, the USP, the main draw, and these first time perspectives on the artist and his career. Why Warhol is still of interest and the personal interests of the interviewees themselves. We started by thinking the issue would be about content and rights but the interviews have gone beyond the object there.

Image of Glyn Davis (University of Edinburgh)Q&A

Q1) Will there be assessment at the end? Will they be assessed by peers.

A1) Yes, I think there has to be for Coursera. I have PhD student Teaching Assistants. I have left some of those decisions to them. They have suggested allowing practical responses to the materials – to get a sense of materials and present day materials, contemporary approach. Or a short written text, a 2-300 word response to a work of their choosing – perhaps from Artist Rooms or perhaps another. These are great TAs though with ideas like building a map of the nearest Andy Warhol to the participant, opening up possible discussion of access. Peers will assess the work and this is where drawing on the expertise of colleagues who have run MOOCs before is so valuable.

Q2) When we did our MOOC we had an easier rights time but we really wanted to use films that it was hard to find legal clips to… we avoided anything we knew was of dubious origins. But we found students sharing those clips and images anyway! What do you plan to do with that?

A2) As far as I know the Warhol Museum in Pittsburgh are well aware that material leaks out… if our participants link to those things we can’t help that. We just create that distance and leave that in the students hands.

Comment – Debs) I feel your pain entirely! In addition to the academic excellence issue, at MoMA part of our job is about preserving the identity of the work, of the artists in our collections. We can’t distribute unofficial copies of works by artists in our collection, it wouldn’t look good. And yet… we were one of the first museums to go to Electronic Arts Intermix about using video online. They’d never really been approached to digitalise their works in that sort of context. The first person I spoke to was extremely pessimistic about these once-cutting edge technology using artists works being able to share these works online. We were able to say that in the environment of this course – a limited course, not a MOOC, we have a lot of details on them – it is very comparible to the classroom. We stream it and although you probably could capture the content but most won’t. They were OK with this. We got Bill Viola, Yoko Ono, etc. allowing us to stream the content. It was costly… but I hope as we push these boundaries more the artists and rights holders will go with that. Otherwise we will have a loss to art history and accessing this hard to reach art. That arguement of the most famous work being the most visible already is one I’ve used before, I hope that rings true.

Q3) Do you have specific goals – educational or a specific combination of enrolees – for this MOOC?

A3) There are two or three key goals. Part was a partnership between the university, the Tate and National Galleries. And part of that was about trying a MOOC as a way to do that. It might be that the Tate or National Galleries want to use one of those interviews somewhere else too. For me it is also about trying a new tool, and what is possible with that. I am interested in testing the boundaries of what Coursera will do.

Q4) With the MOOCs which you have completed… with hindsight now is there a lot that you would do differently?

A4 – Deb) Not a lot but… with the videos I wish we had done differently. I wish we had done them straight without “last week you did X”, or interviews with curators etc. I wish I had had the insight to bring in the right people or to make it more long term useful.

A4 – Sian) for our second run we did make changes. We refused to make videos the first time, we were being hard line. But the dominent comment online were “where are the professors” and “where are the videos” so we made introductory videos for each week. That was the most significant change.

And with that a really interesting afternoon is complete with thanks to organiser Claire Wright, and to the Royal Society of Edinburgh for providing funding for the event.

Find out more

Dec 042013
 
Screen Capture of the Digital Participation Inquiry Website

Today sees the publication and launch of the Interim Report from the Royal Society of Edinburgh Spreading the Benefits of Digital Participation Inquiry.

I have been delighted to be a member of this Inquiry Committee as we have spent the last year or so investigating existing research and speaking to people across Scotland about their own experiences, concerns and ideas. And I wanted to make sure the report was shared here as I hope you will help us get word out about it.

We are really keen to ensure that the Interim Report is read and responded to by many new voices, particularly those who we have yet to engage with. We are keen to hear your honest and informed feedback, comments, and suggestions as we reflect upon the Interim Report and make changes and improvements before a final report is launched in Spring 2014.

The best way to get in touch with your feedback is to email the Royal Society of Edinburgh (digiscot@royalsoced.org.uk) but I will also be happy to pass on any comments left on this post or sent directly to me.

Find out more: