Data analysis: trickier than you'd expect

Well, at least it's trickier than you'd expect if you expected it to be easier than it ought to be.

Now that I'm gathering data, I'm starting to look at ways to analyze it. My first attempt has been to take some off-the-shelf tools to try to see how students are moving through the website. I've got a nice pile of data to look at (535 megabytes of BZip2 compressed log data), so I decided to analyze a day's work of logs using Statviz. I've come to a couple of conclusions:

1. I need to put a lot more work into data cleaning and massaging before I'll get useful results out of this data. There is just too much noise in these logs to give useful output
2. I need a faster machine! I set Statviz running on my PhD machine, and gave up three days later when it just kept processing. I started up a run on another, much faster machine in the house, and took over 24 hours, but it eventually came up with a result.

This was just one day's results - in other words, the website is generating log files faster than I waill be able to process them. On the bright side, the entirety of the log data isn't my primary concern - it's the usage patterns of the social tools that concern me, which will involve a smaller subset of the log files, and contents of databases.

As an example, here's one of the graphs generated from the log files:

It's substantially shrunk down from the original, which is 49680 by 6460 pixels in size.

First analysis

I'm still not sitting on a vast pile of data, but I've done my first bit on analysis on my log file data, and it mostly showed me what I was expecting. I had put the link to upload resources on a page that I thought was rarely visited. My analysis shows that this page is in fact visited 3% as often as the other page I could have put the upload link on.

Compass, the Curriculum Management System/Virtual Learning Environment that I'm adding these social tools into, is built around two main objects: the Teaching Activity and the Learning Objective. It is a searchable database of those two things, and the linkages between them - around 2,000 of each are in the database. It also holds learning resources, provided by the teachers, which are linked to the Teaching Activities. Part of the idea behind my project was to add prominence to the Learning Objectives, by linking the student resources to them.

The problem was that I did the logical thing, by adding the "upload resource" button onto the Learning Objective pages. The students currently have no need to visit these pages, since the key parts of the Learning Objectives are displayed on the page of the Teaching Activities they are linked to. The predictable result was that almost no students discovered the Add Resource button. In fact, it was predictable enough that I pointed it out as the second requirement for a successful social network for learning in my first post on this blog. Oops.

I've now finished most of the coding to add the tools onto a range of more high-profile web pages within Compass. The coming weeks will show whether that makes any difference. In the longer term, I'll be adding even higher profile links, such as an Activity Stream of the latest uploaded resources on the portal page for the web site. I'm expecting each new tool will add to the amount of utilization my network receives.

Data!

Things are starting to get a little bit interesting - I've finally got some data! Not a lot - in fact very little - but it's better than where I was a few weeks back, when I launched my software to deafening silence.

My social network now contains three posts, and two students have sent through feedback. Still disappointing, but to be expected. I'm now working towards the release of phase two of the software, which I plan to roll out this coming Friday. It contains a few bugfixes and cleanups, but mainly it's making the social tools visible in more places across the site, to encourage participation. I would have liked to have put more changes into this upcoming release, or to have released it sooner, but I've had about six weeks of illness - depression, stomach bugs, and now a cold - which have deep sixed by productivity. But overall, things are looking up.

Productivity

I've been working on productivity a lot this year - step by step, incremental improvements to the way I run my life, with the aim of becoming more productive. I've been actually considering setting up a separate blog about it - it's been a big thing for me this year. Over the last few years - since starting my PhD, and even before that, I've been trying to work out ways around my laziness and torpor. I'm not one of those people who feel a natural urge to be doing productive things all the time - I'm happiest in front of a TV and computer, playing games, reading, and/or watching comedies and shonky horror films. Planning has never been my forte, nor has systematically working towards a goal.

This all needs to change. After various experiments over the years, I've found some things that work:

1. Remember The Milk. It's just a to-do list, really, but it's got a number of features that make it more effective than any other to-do list I've used. It was good web and mobile clients, so it's everywhere I am. It lets me set dates and categorize things. So, everything in my to-do list is set to happen at a certain time, so each day, I know I can't really rest until I've done everything I need to. I'm using it for work, study, and in my personal life, and in all three, I feel my productivity has significantly increased.
2. Weekly PhD planning - each week, I schedule what work I need to do each day - whether it's software development, reading, data analysis, etc. I set a minimum standard for what comprises a successfully completed task. This means I no longer have the luxury of thinking "hmmm, PhD night or TV night?" when I've put the kids to bed and dealt with everything - if there's something on my list, it needs to be done. Remember the Milk is where this to-do list lives.
3. "Leads to follow up" - a PhD involves gathering vast amounts of information about a subject area, and trying to ingest it. You're learning a huge pile of new skills and concepts. Every day, you hear about new things that you ought to look into to see if they're relevant. Early on, I would try to remember them, and hope they didn't slip away. I didn't remember, and they did slip away. My solution is Evernote - a folder called "Leads to follow up" contains anything that might be important. Also, a repeating task in RTM to follow up a lead every few days. Currently there are 70 items sitting in my leads to follow up notebook, and 41 in my "Leads - followed" list. It's really handy, and also helps prevent distraction - if I find something useful thread of info while working on something else, I can just pop it into the Leads notebook, and get back to what I'm doing.
4. Evernote in general - I set up my Evernote account a couple of years ago, but I'm finding it more and more useful as I go along. Recently I started using the "todo" system in there - If I make a note that requires a followup action, I put that little checkbox in there. I can then run a search, for "todo:false", and find all the things in evernote that need doing, and follow up on them. Fantastic for ensuring I do things I've promised in meetings.
5. Triaging. I schedule in several sessions each week, to make sure things are in the right lists. The "leads to follow up" notebook is one example, but there are more - I schedule in a weekly session of searching through evernote for todos, and put them into RTM, so they get done. I also go through my email, to move things into RTM and other places.

That's where I'm at now, but after years of failed efforts to improve productivity,  I now have some velocity, and want to keep improving. The aim is to get to a point where

• The things I say I'll do, I actually get done
• I don't have a backlog of things that are overdue, but instead am tackling things in advance
• When I'm not working on things on my list, I don't feel guilty about that nebulous pile of stuff to do

That last one was a doozy in my undergrad degree - during those years, I always felt bad when I wasn't studying. It wasn't until three or four years after I graduated that I read a fiction book, so ingrained was that feeling that I ought to be working on something serious. I wasn't productive at all; I just spent all my time feeling bad about it, and not properly enjoying my leisure time. This time around, I'm changing things.

There are a bunch of next steps in this. Firstly, some longer term planning, especially around the PhD. Currently I'm only planning out each week. I've got a few tasks set to recur after four weeks or six weeks, but the bulk of my planning is short term. I'll be putting together a multi-month plan, probably through to the end of the year, so I can see where I'm going, and give some larger structure to my work. I want to play around with the pomodoro technique - especially once I start writing. I've heard good reports from many smart, productive people. I need to find a way to get a big picture of my literature review - mind mapping, maybe? Mind maps irritate the hell out of me, but I need to find some way of getting that high-level view and drilling down. There's a lot more room for improvement, but the great thing is that I'm making progress.

Types of social network

One thing I'm thinking about at the moment is how I'll structure the first version of the social network I'm building. Social networks come in a lot of flavours, from the messy, real-world face to face social networks that we all live in, to online discussion forums, where people meet to chat about things, to the more formally structured Social Networking Sites, where people make explicit links between each other. I'm most interested, from a research perspective, in the explicit social networking site in the context of a virtual learning environment; but even then, there are a lot of choices in how to develop the site:

1. Is it a "pure" social network, where the interaction and linking between people is the primary purpose - Facebook feels a bit this way (though it has a lot of other features, mostly people just post statuses and photos, and their friends like and comment on them), and this seemed to be what Orkut was mainly about, at least in the early days (I'm not Brazilian, so I haven't been to Orkut in years). Or is it some other site with a social network built around it - like Flickr, or Tumblr, which are mostly about photo and video sharing, but have the ability to connect to other users in order to keep up with what they are posting?
   My initial plan is that I'll be developing a resource-centric social network. Probably the major reason for this is that I want my participants to have a reason to use the system. They are students at a major University, all in the same course, and so they already have a strong social network among themselves; giving them a pure social network just replicates what they already have in Facebook. The other main reason is that I'm aiming for this to be a social network for learning - making it resource-centric means that they will be sharing learning resources that they have created or discovered.
2. What is the structure of the network? Here I'm talking graph theory - not this kind of graph:
   
   but this kind:
   
   Imagine each of those dots is a person, and the lines are connections between them. Does a link need to be agreed upon by both parties (as in Facebook), or can it be initiated by only one party (as on Twitter) - ie. will the graph be directed or undirected? Do the links have additional properties (as in Google+'s Circle feature; or Facebook's ability to make certain links as being family or spouse)? How many links do we expect a typical user to have, and is there a limit (Facebook allows a maximum of 5000 friends; Twitter and Google+ are unlimited) - ie. what are the typical and maximum outdegrees of the network?
3. What does a connection mean? Does making a connection to another person in the network imply they are a "friend" or a "study partner", or is it more an "I'm interested in what you're posting"? How I name and manage the linking function will suggest to users what is meant by it, and hence affect the way the network is used - people will react differently if they're "following" someone's posts, as opposed to if they're declaring them a "bff", even if the functionality exposed it identical.
4. What kinds things can be posted into the network? Is it plain text, (à la Twitter), or is it a multimedia extravaganza (à la Tumblr)? Does each post lead to a discussion (Facebook), or can it only be liked or re-shared (Twitter, Tumblr) and replies need to be posted in the replier's own posting stream?
5. How are posts rated by other members - are only positives allowed (Facebook "like"s, Twitter "favourites"), or are negative ratings also allowed (Youtube's like/dislike, Slashdot's moderation tools)? What do the ratings mean? Do they have additional properties (Slashdot's "Funny", "Informative", etc. ratings)?
6. What are the posts attached to - are they attached to a person (Facebook's statuses and walls), or are they attached to some other things (eg. Facebook's groups, Flickr's Sets, Tumblr's blogs)?

There are many other more subtle choices that affect how the system is used. So, where to from here?

My objective with the system I'm building and the research I'm doing is to build a social network that students willingly use for learning. I'm lucky in that I have a large cohort of very keen students who, if presented with a sufficiently useful tool, will use it. I'm building this into an existing course in an existing LMS, so it needs to follow the conventions and requirements of those. At this stage, my thoughts are that the most useful thing for students is a resource sharing network - they're all out there finding useful web sites, creating summaries of learning material, and so on, and being able to share these could be very useful. The most obvious way to handle the connections is to allow students to follow other students, so that they can build up a feed of interesting new resources. This means building a directed graph, and that the meaning of a link is of the "interested in your output" kind. It means allowing student to post anything into the network (Tumblr style), since it's impossible to predict what the most useful kinds of resource will be. It means building a rating system to allow constructive criticism and collaborative moderation. And it means using the curriculum as the crux of interaction, in the form of Learning Objectives.

Of course, all of this could change, depending on how the students take to it, and what their actual needs turn out to be. It's very unlikely I'll stumble across the Right Solution first time around - it's much more likely that I'll have to try a number of things, until I find something that the students take to. This kind of research is very context sensitive, and (as shown above) there is a large space of possible social networks, some of which would work very well in other contexts, and a few of which will work in my context; so I'll need to wander around, looking for that local maximum of affordance and functionality.

ICLS 2012

The week before last I went to my first ever international conference: the International Conference of the Learning Sciences.

It was a little less international than could have been wished - it was right here in Sydney, which meant I didn't get to travel anywhere exciting. But it was international in the sense that most people attending were from overseas.

I found it really worthwhile - the theme for the conference was "The Future of Learning", and it really did give an exciting picture of things to come. I've seen demos in the past of the "interactive classroom", but this was the first time I've ever seen things that actually looked like they could be  used in a real classroom - a symposium ("Interactive Surfaces and Spaces: A Learning Sciences Agenda" by Michael A. Evans, Jochen Rick, Michael Horn, Chia Shen, Emma Mercier, James McNaughton, Steve Higgins, Mike Tissenbaum, Michelle Lui, James D. Slotta), showing projects with big touchscreens, projectors, kinect sensing, and others based on iPads, was an amazing glimpse into the near future of learning. Matthew Berland was also a very impressive presenter, and I really enjoyed his talk about using strategic board games to support computational thinking (and was surprised how many people in the room were familiar with Pandemic, the game he was using).

A quiet start

So, I launched my software, Friday a week ago. It hasn't been met with the fanfare I expected, to say the least. I have a grand total of zero posts into the system so far. And 57% of students have consented to be a part of the study (ie. 43% declined to participate). I was disappointed at first, until I thought about why it might be the case.

In my last post I talked about what would make the network a success, and one of the criteria I mentioned was that it needed to be embedded throughout the system. When I look at what I've rolled out, the interfaces I've created are quite visible. But they're on pages that are rarely visited. In fact, the add button, to share a resource into the network, is on one of the least visited pages on the system. Other pages will display the resources once they're there, but the use of the system isn't getting started, because the students haven't found that crucial "add" button. On the bright side, if I do add the buttons in more places, and students actually start using them, then I have some solid evidence to back up my claim that embedding is crucial.

Also, I haven't done a proper job of explaining the new features. I'll put this one down to a rather unpleasant bout of depression and stress I've been in over the last month (work-related, not PhD related), which has meant that I haven't had the emotional energy to communicate. In a lot of ways, talking to the participants (even via a bulletin system) is a lot more emotionally draining than any of the other tasks. Coding can be done in most moods - productivity can be affected, but the quality of the work ends up not too variable. Quality of communications, however, is greatly affected by mood. I assume this is the case for most people, except maybe professional comms people.

So, I guess the plan over the next few weeks is to remedy these teething problems. Luckily, I've got a couple of years for data gathering, so there isn't a huge rush, and a couple of weeks isn't a major setback.

Also, I went to a conference. It was great. I'll write about it another time.