Cleaning the CAM database

I'm now pulling nice big piles of data into my CAM database, but I'm finding it rather messy. I'm parsing log files, and there are over 100 distinct URL patterns I need to handle. This obviously brings in a lot of places I can make mistakes - mis-parsing URLs, overly broad matching of a URL pattern, missing variants in a particular URL pattern. So I'm now working on cleaning that all up. The main items I'm looking at are called PBLs and TAs, and they have clearly defined patterns regarding what they are.

It's slow going - I'm gradually fixing up and improving those 100 patterns, and finding all sorts of ways they can be wrong. I'm also finding "errors" in the data - which appear to be incorrect URLs formed by users playing with things in the address bar of their browser, or occasionally from javascript incorrectly accessing unusual URLs. I'm not sure what to do with these events; it's probably the case that I should just delete those URLs from my analysis database, as they will just make analysis harder, with no real benefit.

I'm interacting directly with the database here, using the SQL console. It's the quickest and most trustworthy way to get at the data; but I'm starting to think having some other visualization tools might come in handy at this point to confirm that I'm getting the patterns I expect.
Here's some of what I've done:

Phase 1: add a "pattern" tag to events, to allow mapping of parsing patterns to items in the database. This will let me check that each pattern is catching the right items, and that the code isn't catching and misinterpreting any false positive matches

Phase 2: Look through every single pattern and ensure that it catches URLs that will be parsed correctly by its parsing rule. Use commands like:

select distinct followedlink from event where tags='pattern60';

This will take a long time to do.

Phase 3: Look at items, ensure that they are being mapped correctly. Initially, I was getting PBLs with values like "4635" and TAs with values like "4.04" - I had mis-mapped items. Use commands like:

select  item.title,item.itemid,count(*) from item,eventitem where type='TA' and item.itemid=eventitem.itemid and not item.title ~ '^[0-9]+$' group by item.title,item.itemid;

To find non-correct TAs, and

select  item.title,item.itemid,count(*) from item,eventitem where type='pbl' and item.itemid=eventitem.itemid  group by item.title,item.itemid order by count;

to find all the PBLs and work out which ones are OK.

Some results!

OK, admittedly, they're not very exciting results, and they don't tell me a whole lot that I didn't already know, but I've finally got my data into a kind of shape that allows me to analyze it.

My CAM structured database now has about 12 million Event records in it (12,383,032, to be precise, drawn from 18,764,699 lines of log files). From that, I have created a graph that show how long each resource is used (specifically, the number of downloads each day after it is downloaded).

Average downloads per item by day after posting. Click to embiggen.

What it's showing us that the vast bulk of the downloads of each resource happen in the first week after it's posted, (see that red spike at the start there - that's an average of 50 downloads for each resource in the day it's uploaded). In the second week there's an average of one download per day, and after three weeks the downloads pretty much drop to zero.

This is to be expected on a course built around a weekly timetable like this. Once the week is over, students no longer (a) need that resource any more, and (b) won't see it through the interface any more, since the links to the resource are in the previous week's section of the site, which they aren't visiting any more.

There is more detail in the following graph, which shows the download stats for all the student resources (each as a separate line with randomly generated colours):

Downloads per item by day after posting. Click to embiggen.

It's a lot messier and harder to read, but it's interesting in that there are two specific resources that stand out - one (represented by a red line) became popular at about 125 days after it was uploaded, and remained popular for about six weeks; and one (represented by a aqua-ish* line) became popular about 300 days after it was uploaded, and remained populat for about three weeks. I'm not sure what was special about these - whether the became relevant to the curriculum for some reason, or whether it was a UI change that I made that threw them into the limelight, prompting students to download them. But I think they warrant further investigation.


* The problem with randomly generated colours is that they don't lend themselves to easy naming. XKCD proved it.

The lure of shiny toys

As mentioned previously, I have a shiny new machine to do my PhD work. It's delightful - the latest 27" iMac, with i7 processors, 16 gigs of RAM, and a fusion hard drive. It's delightfully fast.

It's especially nice for analysis work. Prior to the upgrade I had to run my analysis on a separate machine, wile I used my study machine. It was slow and painful - analysis runs took hours (close to a day in some cases). They now take minutes - I can import log data into my database at the rate of 1500 records per second.

The main advantage, though, is the pleasure I get from sitting down in front of it. With the previous machine, the lag that would happen when I wanted to do anything slowed me down, and caused me to avoid doing too much - sometimes the 20 seconds it would take to log in and get my application running meant that I would avoid doing it at all. Now, everything happens straight away.

It's another lesson - it's worth removing those little barriers to productivity. In my case, working full time means I can afford to throw money at the problem - making my working environment nicer by buying a new machine means the hours I can afford to spare for my PhD are that little bit more productive and pleasant.

How MOOCs will take over the world

Everyone in the e-learning world is talking about MOOCs right now. In a lot of ways they aren't anything new; just the stuff we've been doing for a decade or two, scaled up rather dramatically. They are succeeding and failing in interesting ways - they are getting very large numbers of enrolments, but they have enormous drop-out rates. There are all sorts of questions and assertions floating around, ranging from "are they a fad, with no quality learning emerging from them" to "they are going to destroy all Universities within five years".

In reality, they are just a segment of the existing spectrum of learning experiences. In fact it's more than a spectrum - there are multiple dimensions in play. Perhaps something like this would start to give a picture of where they sit:

The claims that MOOCs will take over the world are driven largely by economics, ignoring the various types of learning experience that exist, and the value each of them delivers. The argument is basically that if Stanford can deliver the best quality lectures on a topic, and can give them out for free, then other Universities won't be able to compete, and people will stop coming to Unis.

There are two reasons we enrol in a course. Firstly, to learn; and secondly, to be able to claim knowledge. The 10% or so of people currently finishing their MOOC courses are interested in the first part - the learning. Currently a certificate of completion from a MOOC doesn't hold a lot of value in the employment market, so the ability to claim knowledge from these courses is a bit tenuous. I believe that this is a very significant factor in the drop-out rates - it doesn't matter if you fail a MOOC, and further, it doesn't matter if you pass. So the second reason for enrolling, that extrinsic motivator which pushes us through the hard parts and the boring parts, is a lot weaker for MOOCs. We won't start to see more normal patterns of completion in MOOCs until this changes.

The thing that will change it is employer acceptance of MOOC courses as valid indicators of expertise in a topic. This will be a gradual process, as awareness of these courses becomes greater.

(Warning: here's where I start waffling about what would happen in an ideal world). What is really needed is some decent certification of courses - probably to be done by the industry bodies that already certify University degrees in many disciplines. To do it properly, you'd need a set of standard learning objectives in each discipline, and then each MOOC could get itself certified to a certain level against those learning objectives by these independent industry bodies. Only by doing this will learners be able to objectively demonstrate that a collection of courses they have done - from MOOCs, Universities and commercial training bodies - is equivalent to a University degree. What's more, it opens up the ability to demonstrate narrower or wider expertise, or cross-disciplinary expertise.

It's an interesting time - I'm looking forward to seeing what happens in this space. I'd hate to see Universities go, but I'd love to see degree structures shaken up and for people to have the ability to create their own degrees from a range of sources.

CAM: both easy and bewildering

As I'm sure I've mentioned before, I have several data sources to analyse for my PhD research - log file data, and database records of postings, comments, and ratings. In my reading of the literature I came across CAM - Contextualized Attention Metadata. It looks like exactly what I'm  looking for to do my analysis - a generalized schema for tracking information about learner's interactions with learning tools (and each other). It's a sensible schema, and there are a few pages about it on the net:

Fraunhofer Institute
CAM Schema (seems to be also done my the fine folks at Fraunhofer)

As well as a few academic papers that describe the schema, such as:

Muñoz-Merino, PJ, Pardo, A, Kloos, CD, Munoz-Organero, M, Wolpers, M, Niemann, K & Friedrich, M 2010, ‘CAM in the semantic web world’, ACM Press, New York, New York, USA, p. 1.

Scheffel, M, Niemann, K, Pardo, A, Leony, D, Friedrich, M, Schmidt, K, Wolpers, M & Kloos, CD 2011, ‘Usage Pattern Recognition in Student Activities’, Springer Berlin Heidelberg, pp. 341–355.

However, the information available online about it seems awfully sparse, compared to my experiences with open source software. There are a bunch of downloadable items, but no real "getting started with CAM" documents, not SQL create scripts, etc. The SQL binding is distributed as a PNG file or GraphML, not as SQL. This is not a major hurdle; just not what I'm used to from experiences as a software developer. I spent a couple of hours searching for some sample SQL create scripts; in the end I spent twenty minutes writing my own based on the available documentation.

The other oddity is the number of different versions of the schema, which seem to differ quite radically. I've settled on version 1.5, this one:

though I think I'll need to make some changes (the same Item is referenced by events in different Feeds; rather than keeping separate Items for the separate Feeds, I'd rather link the feed to the Events table).

It seems pretty good - there is space in the schema for most of the things I'd like to log. I've had a chat to Abelardo Pardo who, in addition to being very intelligent, is an author on both the papers mentioned above, and he seemed to agree that the schema isn't something set in stone; it's a flexible framework that allows you to use various components when you need them. The version he was using also had a User table, which will also be useful and which I'm considering adding to my existing schema.

So I have my data in there. I'm very pleased with my session splitting code (for detecting user sessions in the log data). My importer is loading over a thousand records per second* (which is important when each day's logs are five to thirty thousand lines - over half a million lines for January, which is a very light month in terms of site usage). Currently I'm recreating the database each time and loading all the data, but at some point I'll set it up to run with each day's logs as they are created, and just have a database sitting there with all my data, ready to go. Also on the list to investigate (at Abelardo's suggestion) is a NoSQL database - which apparently simplifies queries and improves performance.

* From the latest import: "636600 events inserted in 416 seconds - 1530 events per second". I am fond of my shiny new iMac.

Crisis averted

Over the last couple of months I've had a bit of a crisis looming. Some academics and students had been discussing setting up a Wiki (or something similar) for general sharing of resources between students. In one sense, it was great - it demonstrates a clear demand for these things among students and staff, and a willingness to put work in. On the other hand, despite knowing about my project, they were kind of shutting me out of the discussion. From my point of view, this stood a chance of undermining my PhD project completely - if students were presented with two resource sharing systems, one seeded with content and supported by academics, and another (my project), they would naturally choose the supported, seeded system. Leaving me with no data, and no project.

Fortunately, in the last few weeks the folks in that project have started talking to me - some of them didn't want to undermine my project, and understood the risks of having two competing projects in the same space. I think they also started to realize the amount of time, energy, and money they would need to put in to build and support the system they were talking about. Last week and this week I've had a few meetings with the team, and we've all agreed to work together - they'll proceed with their plan to seed some shared resources for students, and I'll make some modifications to my system to support what the students are wanting to do, and everyone wins :)   I'm very lucky to have supportive colleagues who are willing to work with me in this way, and I owe James a beer next time I'm at the pub with him.

This is nicely in line with my research plans anyway - I'm exploring ways to kick start and sustain a social learning network. If this seeding works, it will hopefully start a snowball effect, where students see the value of contributing, and start to use the network; and once that happens, I start to have enough data to work out how students are using the system, and how it can be improved.

On being a PhD student

Some days I am a PhD student, and some days I am a person who is enrolled in a PhD. They are two very different things, in my experience. Similarly, sometimes I am an IT manager, and at other times I happen to work at a university managing IT folks. Or a Parent, or a person with kids. I guess technically, I'm all of those things at once, but in reality, I move between those roles and identities. Of course, the raw external facts always apply - I'm enrolled in a PhD, I work at the Uni, I have kids, I have hobbies. But it's difficult to be a PhD student, a Manager, and a Parent all at the same time. Each of those people has different priorities, different needs, different ways of thinking about the world.

This has been on my mind lately a bit - towards the end of last year work got hectic and stressful, and left me with no energy for study. As time went on, that PhD student identity dissolved a bit, and the idea of spending my evenings reading, coding, thinking and writing seemed less important, and less like something I wanted to do. I would see other PhD students as "them", not as an "us". December was basically nonexistent, study-wise. January wasn't much better, but improved as the month progressed. A holiday last week seems to have restored me, and now I'm back to full strength (I hope).

It's all made me realize that doing a PhD involves being a PhD student. It's a whole identity - more than just an external fact about you're life, it has to become part of who you are. When that identity isn't in place, there's no stable source of motivation for spending all those hours tinkering away, to keep eating that elephant. Those other identities coexist, and of course they take precedence at times, but it's important to hold on to my student identity - letting go of that is the first step towards not completing this PhD.