Thoughts about my result

This is where statistics gets hazy for me. There are a lot of tests one can perform on a set of data, and the real skill is in knowing which one is the right test. I can comfortably say that the use of my rating tool is significantly different to the normal usage for the March reading (the Feb 2013 and Feb 2014 levels of usage are indistinguishable). This is both using Normal and Poisson distributions (and since these are hit counts, Poisson is the better distribution to use). However, when I try a Student's t-test comparing the January-July results for 2013 and 2014, the result isn't significant.

So basically, while my intervention cause a detectable spike in usage, overall it hasn't made any difference. I'll need to talk with my supervisor about how to look at the data to firm up what I can say about it. I think that also looking at which cohorts of students were involved might help. There may be a difference in who was involved in each spike - if the 2013 spike was four cohorts of students, and the 2014 spike was only the incoming first years, then the number of hits per student is dramatically improved. I need to get back into my data and do some more work.

Result!

Following on from my previous posts (Evidence and My first MOOC), I've put two and two together and used my new statistical knowledge to good use to work out if my rating change is actually significant. And it looks like it is.

I broke up the usage of the rating tool since its inception by month, and analyzed the results. Each year there is a spike in February, when students start (p < .001), but after I added the rating tool, the spike extended by a month (p<.05).

I'll need to confirm this, but it's very satisfying to have something.

My first MOOC experience

I was saddened to discover that the course on Quantitative Methods held by my Faculty is no longer run, since I have been starting to realise that I'm sorely in need of some statistics training. After a bit of looking around, I decided that a MOOC was the ideal answer - it kills two birds with one stone. It gives me an opportunity to learn stats, and also gives me first-hand experience in a MOOC, so I can actually claim to have a valid opinion on their usefulness.

I settled on the Johns Hopkins course on Statistical Inference (https://www.coursera.org/course/statinference). It was hard to figure out which course would be right, since I don't know enough about what I don't know to really make a good decision. The description for this course indicated that it covered the things that I needed to know.

My initial experience is that it all seems a bit simple and lightweight compared to previous things I've learned. There just isn't the "hang on, can you explain that bit again" that you have in a face to face course. But things are sinking in, and I'm getting a lot of lightbulb moments as things click and I understand things I've been reading in papers for the last few years. I'm certain I don't know as much as I'd like to from this material, and very uncertain about whether I'd be able to apply it correctly after I finish this course.

In all, I think the MOOC structure isn't a bad way to get some learning done. But there are a lot of gotchas in there. The learning experience isn't great, since if you fall behind on a part there doesn't seem to really be a clear alternative route to understanding (unlike, say, a tutorial where you can ask dumb questions until you figure it out). The teaching is very didactic - it's a series of lectures with a very small quiz and homework attached. Also, the quiz and homework are far to similar. I feel I'd be better served by a lot more example questions, to really stretch my understanding of the material. The week-by-week structure is helpful in driving me to complete the material; without this procrastination would be much easier, so it's good to have that structure. I can understand the huge dropout rate - unless you really need to learn something, it is extremely easy to just give up when it gets too hard.

Learning stats

I've read a lot about the edtech industry implications of MOOCs, and educators' pedagogical concerns regarding their implications, but it's only now (mid-2014!) that I've finally enrolled in one. I've come to the understanding that my exploration of my data in R is hampered not by my knowledge of the tool, but rather by my lack of solid statistical knowledge. It's time to fix that.

I was surprised at the number of stats MOOCs out there - multiple instances of even the same title from multiple universities and multiple MOOC providers. There's a great range of choice. And delightfully, most of them seem to use R as their tool of choice.

I'm going to jump mid-way into a series of courses from Johns Hopkins (https://www.coursera.org/specialization/jhudatascience/1/courses), hoping I'll be fine based on my mathematical background and having had a few months playing with R.

Scale

There has been a lot of interesting discussion online recently about Connectivism and learning theory lately - particularly on Jon Dron's blog and Stephen Downes' responses to them. I'm still very much in two minds about Connectivism as a theory. I'm not sure how it is different to Networked Learning (they seem to be the same practically, if not epistemically), and my inner positivist always screams that "knowledge" is something that resides inside people - that the stuff that resides in these networks is really something very different, even though it is a source for knowledge.

One sentence really stood out for me in Dron's post from last Friday ("What is Connectivism?"):

More is different in a networked system, resulting in a) large scale patterns and emergent behaviours not usually seen in smaller systems, and b) linear benefits of scale - a 1% contribution rate in a network of 10 people will be of a lot less value to all the people in that network than 1% contribution rate in a network of a million people.

It's not something that is a new concept to me - but it made me reflect on my project and how it performs, and has made me think differently other things I've read since then. That 1% contribution rate is not an unreasonable estimate of how likely people are to add ideas into a network if there are no extrinsic motivators; but then the network of ten people most likely has no contributors, and the network of a million has ten thousand. Ten thousand contributors can generate an awful lot of of content, and communities can be formed around that content. My own data bears these numbers out:

I have a group of over a thousand potential contributors - but only 37 (3.3%) have uploaded anything, and only 11 (1%) have contributed five or more resources. There is almost certainly not enough content and activity to generate a significant community around these resources.

I intend to keep improving resource tools in the hope of lowering the barriers to participation to the point where a community can self-start based on these numbers. But this has been a clarion call for me to start thinking about problems of network scale, and what they mean for my project.

Evidence

At last, I've got some evidence that a design change had an actual impact on the usage of my software! This is quite an exciting development. The proof*:












At the end of last year, I changed the way the rating system worked on the student shared resources - rather than a small "RATE" link, which opened up a rating tool, and wasn't very obvious or intuitive, I replaced it with nice big thumbs up/thumbs down rating tool - described here. It turns out, as shown in the graph above, that it has encouraged students to use it. In the 17 months that the first iteration was live, 53 ratings were added. In the last three months, there have been 52 ratings. Given the small numbers, the error bars are going to be large, but this has to be a statistically significant result.


* For probably barely statistically significant values of "proof"

R - loving it

I'm still impressed by R - I'm finding it delightfully powerful. Still trying to figure out what to do with my PhD data, but I'm finding it incredible useful looking at work data to uncover patterns. And the speed with which you can tease pattens out of data means that I can do things like sit down with staff and explore a data set to start to work out why a piece of software is playing up - or at least show patterns in the data which will lead good developers to understand how a piece of code is misbehaving. I'm now annoyed that I hadn't discovered this earlier :)