Archives for : design

PPIG Publication, November 2012

In press is a conference paper I’m about to present at the Psychology of Programming Interest Group (PPIG) workshop in London, 21st-23rd November 2012. As I noted on my HCI 2012 post, some of these have now been going on longer than the participants (this being PPIG no. 24). This is a full paper written with my PhD supervisor, Prof. Michael Kölling (well known in the field of computing education). The paper reports on a pilot study where we evaluated the effectiveness of new HCI usability heuristics (which we created), specific to the domain of learner programming systems (e.g. Greenfoot, Scratch, the Mindstorms kits, etc.). The abstract follows below.

EDIT: The paper is now published, and a PDF can be downloaded for free from the Kent Academic Repository, or directly from this link. The Q&A session after the talk gave an opportunity for some interesting discussion with the likes of Thomas Green, Alan Blackwell, and the other cognitive dimensions “grandees”!

Evaluation of Subject-Specific Heuristics for Initial Learning Environments: A Pilot Study

Fraser McKay & Michael Kölling

“Heuristic evaluation is a “discount” technique for finding usability problems in well-established domains. This paper presents thirteen suggested heuristics for initial learning environments (ILEs). To investigate the usefulness of these heuristics to other developers, we conducted a pilot study that compared two groups of evaluators: one using an older, generalised set of heuristics from the literature, and one using our domain-specific heuristics. In this study, we compare not just the number of problems found, but the way in which the problem reports were expressed. There was a significant difference in the length of written comments when problems were found (those from the new set being longer). New-set reviews touch on more themes – many make suggestions about what would improve the problem; many comments refer to a suggested cause-and-effect relationship. As designers, we find this detail helpful in understanding problems. Quantitative data from this study is not large enough to support any robust conclusions about the relative thoroughness of the heuristics at this time, but we plan to use lessons learned from this study in a larger version shortly.”