The last couple days have been about prototyping user interface and rendering techniques for improving the accessibility of old video games. I wanted to see if the basics—drawing things in different colors, tweaking contrast and saturation, labeling affordances, et cetera—were achievable in reasonable time, and indeed they are. I want to get some screenshots together for a demo post tomorrow, but for today my goal was to decide if it made sense to outline a research paper, and I’m pretty sure it does. The challenge for an FDG paper will be showing that it’s a significant enough improvement or new application over e.g. Batu’s work, and I think I can get there by showing generality and the fact that some labels (and potentially more in the future) are automatically generated.
To guide the paper, I want to showcase a problem and my contributions. The problem is that many videogames are inaccessible to people with vision impairments; while new games can be made to support larger groups of players, this depends on the cooperation of the game developer. For other games, we want to find ways to increase their accessibility after the fact. For the large class of Nintendo NES games, I have developed a technological approach and tool for efficiently labeling their graphics while playing with key affordances, which are then used to modulate the display to make it more accessible to players with different visual needs. Some of these tags are inferred automatically from play while others are explicitly given by the user. As a side effect, this also produces a rich dataset of game images tagged with fine-grained affordance data, which will be useful for bootstrapping this work to unseen games or onto new, non-NES platforms in a machine learning regime. I have a natural tendency to want to make this bigger—add in motor assists and point-to-move, add in visual cues for sound effects, learn all tags automatically—but I will try and keep it short this time for a change.
I’ll need to double check that recent FDG reviewers have accepted such “tools papers”—they did publish Eric Kaltman’s and my GISST work a few years back, so I think it’s plausible—or whether I need to put more effort into showing that it indeed improves accessibility, to create a set of accessible games, to generate affordance tags automatically, etc.