Ratings: trying to encourage student to express the usefulness of resources

One aspect of my system that hasn't taken off well has been the rating system - students are uploading things, but there are less ratings in the system than resources - which means there is an average of less than one rating per resource. From where I'm sitting, there are two possibilities for this - one is that the students aren't interested in rating resources, and the other is that the rating tool just isn't obvious, or is hard to use. The design of the tools is a big part of what my research is looking at, and affordance theory is the most powerful way of thinking about these issues. Sharon Oviatt explained (with reference to Gibson) that affordances "establish behavioural attunements that transparently but powerfully prime the likelihood of acting on objects in specific ways". Each tool needs to be designed in a way that pushes the user towards the desired behaviours. Designers can't control how people use the tools they create, but they can design in such a way that the desired behaviours are the ones that users are more likely to perform. In this case, I want students to be collaboratively discovering and sharing the best learning resources, so I need a rating tool that encourages constructive use.

I need to eliminate the second possibility before I can really start pondering the first, and gathering information from students about it in interviews. I have replaced the rating tool, to make it clearer how it works. The original looked like this - the rating tool was a simple "Rate" link at the bottom of the post:

 When you clicked on the "Rate" button, it popped up this:

And it would calculate your rating based on which adjectives you chose about the post. The idea was that it would encourage considered, focused feedback, as opposed to a generic "Like". I suspect the lack of usage was due to the obscurity of the "Rate" button/link rather than the tool being confusing. My stats showed that on a week where there were 2,000 downloads of the resources, there were only 2 ratings done on the resources.

So clearly, a redesign was in order. I did two things: increase the prominence of the rating tool, and simplified the process. The simplification involved separating out the scoring from the adjectives. Rather than just having a rate button, there is now a thumbs up/thumbs down directly on the page:

Hitting a thumbs up or thumbs down buttons opens up the rest of the rating tool, and adds colour to the thumb you hit (green for thumbs up, red for thumbs down):

So the new tool is:

• easier to see: a bigger button, much easier to see what it does
• immediate feedback on action - the colouring is hopefully satisfying to the users, and will encourage them to feed back on all items. Prior to giving feedback, the button looks a little empty
• Simper relationship between action and outcome - thumbs up gives a positive rating, thumbs down gives a negative rating.

The downsides are that the comment tool is now no longer available without giving the item a rating. Also, the user can't close the rating box once they rate an item - I probably need to add a close button on the rating panel.

It will be interesting to see how the students take to this change - whether an improved interface actually results in changed behaviour and increased use of the tool.


Oviatt, S. (2009). Designing Interfaces that Stimulate Ideational Super-fluency. New Knowledge Environments, 1(1).

Studying my work

My research project is in the same area as my day to day work. In the daytime I'm a software development manager, and one of the bigger applications we've developed is an eLearning system. By night I'm a PhD student who is studying eLearning (but rarely fights crime). My research topic involves adding functions to the big eLearning app I manage during the day. This is both good and bad, in a number of ways.

Convenience sampling

Not many PhD students have a group of 1100 students to test their theory on, but the eLearning app we've developed is used by that many students, which means that's my sample space. I've asked the students to opt in, so I actually end up with about 700 students in my study, but that's still a lot. My fellow PhD students have to do a lot of negotiation to get a class of 30 students to study. This is definitely on the good side for me.

Access to real source code

My position means I have access to, and permission to modify the application. I don't have to create a new application and set up an artificial situation in which students use the app - I have direct access into the app these students are using on a daily basis, and permission to deploy the changes I make into the live system. This means I'm really testing my theory in a real world setting, which will give me confidence in my findings.

Prevention of unexpected changes of direction

If I were developing in a system that was also being developed by other folks, I would need to worry about whether they might make radical changes to the system that break, or render irrelevant, my functionality. But since I'm involved in all the conversations around changes to the applications, I can ensure that I'm prepared for any changes, so I can adapt my code appropriately.

Keeping focus

Were my PhD in a completely unrelated field, I'd be thinking about entirely unrelated things by day and night. With my current situation, things that I do in my studies benefit my performance at work, and vice versa.

Overdosing

There's a lot of potential for just getting too much exposure to this system, and either getting bored, or being just too close to it to be able to step back and see the big picture. I don't think that's happened, but it's definitely something to be wary of.

Conflict of Interest

Then there is the issue that I am probably extra defensive about the system I'm building. If the University decided to shut down this application and replace it with a third party tool, my PhD would essentially be rendered irrelevant, and I'd have to start over again. So in discussion about the future of the system I am definitely biased, and probably can't make an unbiased decision about what is in the organization's best interests. But I was like that before my PhD anyway - the applications we build are like our babies, and we always want to protect them.

Can't quit my job

Leaving my current job would endanger my PhD work - it would be very hard to get the kind of access to the system that I have now, if I weren't working where I am. I have strongly considered quitting my job a few times over the last year, and one of the things that has prevented me was the difficulties this would pose for my study. Even if I didn't burn any bridges, it would be against normal policy for the university to grant me the kind of access I have now if I weren't a staff member. This causes quite a bit of personal tension.

Cleaning the CAM database

I'm now pulling nice big piles of data into my CAM database, but I'm finding it rather messy. I'm parsing log files, and there are over 100 distinct URL patterns I need to handle. This obviously brings in a lot of places I can make mistakes - mis-parsing URLs, overly broad matching of a URL pattern, missing variants in a particular URL pattern. So I'm now working on cleaning that all up. The main items I'm looking at are called PBLs and TAs, and they have clearly defined patterns regarding what they are.

It's slow going - I'm gradually fixing up and improving those 100 patterns, and finding all sorts of ways they can be wrong. I'm also finding "errors" in the data - which appear to be incorrect URLs formed by users playing with things in the address bar of their browser, or occasionally from javascript incorrectly accessing unusual URLs. I'm not sure what to do with these events; it's probably the case that I should just delete those URLs from my analysis database, as they will just make analysis harder, with no real benefit.

I'm interacting directly with the database here, using the SQL console. It's the quickest and most trustworthy way to get at the data; but I'm starting to think having some other visualization tools might come in handy at this point to confirm that I'm getting the patterns I expect.
Here's some of what I've done:

Phase 1: add a "pattern" tag to events, to allow mapping of parsing patterns to items in the database. This will let me check that each pattern is catching the right items, and that the code isn't catching and misinterpreting any false positive matches

Phase 2: Look through every single pattern and ensure that it catches URLs that will be parsed correctly by its parsing rule. Use commands like:

select distinct followedlink from event where tags='pattern60';

This will take a long time to do.

Phase 3: Look at items, ensure that they are being mapped correctly. Initially, I was getting PBLs with values like "4635" and TAs with values like "4.04" - I had mis-mapped items. Use commands like:

select  item.title,item.itemid,count(*) from item,eventitem where type='TA' and item.itemid=eventitem.itemid and not item.title ~ '^[0-9]+$' group by item.title,item.itemid;

To find non-correct TAs, and

select  item.title,item.itemid,count(*) from item,eventitem where type='pbl' and item.itemid=eventitem.itemid  group by item.title,item.itemid order by count;

to find all the PBLs and work out which ones are OK.