Cross-posted from fediverse user @peachy@goto.micromail.me
Is there such a thing as a configurable full-body input controller for computers? Is anyone working on that? I know there is work on controlling computers directly with the brain, which will be ace for people with full paralysis, but what I’m interested in is something that goes in the other direction - using more of the body. Think Tom Cruise’s interface in Minority Report but better. Sitting, or even standing, to work at a computer takes its toll on the body, especially the back. Our bodies didn’t evolve to be so static while we’re awake. Emerging from a flare-up of a slipped disc, it has got me thinking of better ways to interface with machines.
Imagine the following:
You come to see me in my studio to see how I and my colleagues do image editing and graphic design in GIMP 4.0. Some of us are stood in front of large displays but no one seems to be using a keyboard, mouse or graphics tablet. I appear to be doing a dance routine from a music video... As I bounce my knee up and across my body you see that the Move tool has been selected. As I raise my left fist above my head it is as though I am holding shift to toggle “Pick a layer or guide”. I draw my right hand across my body with my thumb and forefinger pinched and the selected layer moves with me. Finally, I quickly raise both hands, like I'm flipping over a table and my project is saved and closed. Now that I’ve stopped moving around so energetically you notice that my stylish and comfortable cotton loungewear and gloves have small sensors dotted around them. I explain that the position of these sensors relative to each other and to the space have been mapped to traditional keyboard and mouse inputs via my operating system.
Moving to the next workspace you see my colleague Babs. Her white hair pokes out above a VR headset and she has a number of small cameras tracking her movement to the soundtrack of Chinese classical music. She is an elder and a veteran and even contributed some of the code that makes this stuff work, back in the day. She says it was no big deal; she mostly just connected up different programs, some of which Hollywood has been using since the 1990s. Her movements are slow and smooth. It looks like she’s doing Qi Gong or Tai Chi or something. Raising a hand in front of her heart you see the Filters menu open and lowering it slowly the menu scrolls down to Enhance. Gracefully stepping sideways and lowering her hand further, Heal Selection is highlighted in the submenu. Turning her hand palm-up launches the plugin. She tells you that one of her first contributions to the interface was to make the body position tolerances configurable by the user in their desktop settings.
Lastly you watch my cousin Tommy at work. When we met I told you about how a head injury had left him partially paralysed and unable to speak. He too is using a VR headset, but instead of having cameras pointed at him he has a HD sonar array. His disability was caused by an error in the police’s facial-recognition software and understandably he’s had a thing about cameras ever since. The bad guy got away and he never caught the bus he was running to catch. Every couple of days he asks whether Nancy’s cameras are still disconnected from the network, which they always are. Tapping his ring-finger once on the armrest of his wheelchair selects the Text tool. Turning his head to the side, he purses his lips and sweeps his face back around to make his text box. You see his mouth moving but there is no sound. “Hi, nice to meet you” appears in his projects new text layer. “You too” you reply. Twitching his right shoulder you see his text layer is duplicated, blinking twice and nodding his head replaces the text with what you just said. He must have used speech-to-text to record your words to his desktop clipboard and then pasted them into the text field. Pressing his index finger against the arm rest and looking toward the ceiling brings the new text layer to the top of the stack. Running the same sequence of movements again, a third text layer becomes visible onscreen. “I’d never edited a picture in my life until I got into this tech as part of my physiotherapy treatment. My cousin ended up offering me this job and now I can work faster than anyone else here, especially Babs. I’m pretty sure she’s just here for fun but none of us mind.”
#tech #health #disability #GIMP #solarpunk