Xinyi Therese Xu ’24 and I were talking for a while about adding accessibility features to retro games, especially aids for players with visual and motor disabilities. This picks up a thread of research that I first saw carried by Batu Aytemiz et al. in the paper “Your Buddy, the Grandmaster: Repurposing the Game-Playing AI Surplus for Inclusivity”. I got pretty into this idea around the time I applied for CAREER in the summer of 2021, but 2021-2022 was a really intense academic year and I wasn’t able to explore it much further. While more and more games made today are implemented with accessibility features, so many games of major cultural import will never be patched to include such features. So, how can we hack existing games to add them?
There are a few areas here I want to investigate:
- Visual supports, probably with some game-specific information—as in modern PC games, can we highlight game entities with positive or negative valence, de-emphasize and reduce contrast of unimportant details, augment text displays with text-to-speech support, or crop and zoom the screen where that would be helpful?
- Motor supports, again with game-specific information about character control. Can we add click-to-move features to Super Mario Bros., use slow-motion or other time travel features to make games accessible to people with differing reaction times, and so on?
- Can the information needed by the supports above—the valence of game entities, the control scheme and character physics, et cetera—be defined in a way which is convenient for non-specialist players, potentially by the same people who want to play with these supports?
- Can the information needed by the supports above be learned automatically to support arbitrary new games?
Back in 2021, I added some Python scripting support to Mappy. It exposes the sprites and tiles on screen (in world coordinates) and gives write access to the screen pixels, so it’s a good starting point for adding visual supports. I think this is a project which is both interesting and useful, and it’s something that feels close to being publishable, so I’m hoping to pursue it over the next couple of months.
I started today by re-reading Batu’s paper, and picking up an email thread with a researcher at UC Irvine. I haven’t published in accessibility before, and I feel like my work is more likely to land in the general area of Batu’s—a technological intervention with some examples motivated by existing work, which could be evaluated with a real population in the future.
This is one of those projects where I feel pretty comfortable with the basic idea (change the visual appearance of things based on their semantics), and it doesn’t involve coming up to speed on machine learning techniques or other technologies. So I think my first goal will be to figure out, for Super Mario specifically, how I can identify background tiles, foreground terrain tiles, enemies, and powerups; then, how I can best communicate e.g. the state of Mario (big, small, jumping, etc) and the salient aspects of the game state. Since I’m OK doing this manually, I’ll add a way to click on a sprite on screen to assign it some valence or affordances (maybe calling back to my work with Gerard Bentley ’19) and collect game data that way. Then, I’ll probably add some convenience functions for saturating/desaturating/tweaking contrast/etc of rectangular regions of the screen (maybe later supporting an alpha mask to keep the shapes crisp) and map those onto object valences. For the paper, I think it’s enough to just do a few levels of Super Mario Bros., the first dungeon of The Legend of Zelda, etc, and keep track of how long the tagging task takes.
The paragraphs above are essentially an outline of a research paper already: name an area, scope it out, relate it to prior work, make an intervention, and show its utility. I watched Simon Peyton Jones’s “How to Write a Great Research Paper” recently—as eager as I am to start hacking, I think I might actually try to write the paper first this time around.