On a Future of Screen-less Computers

by Malte Skarupke

The current problem with computers was well articulated in the piece The Machine Stops by the late Oliver Sacks:

I cannot get used to seeing myriads of people in the street peering into little boxes or holding them in front of their faces, walking blithely in the path of moving traffic, totally out of touch with their surroundings. I am most alarmed by such distraction and inattention when I see young parents staring at their cell phones and ignoring their own babies as they walk or wheel them along. Such children, unable to attract their parents’ attention, must feel neglected, and they will surely show the effects of this in the years to come.

[…]

These gadgets […] have now immersed us in a virtual reality far denser, more absorbing, and even more dehumanizing. I am confronted every day with the complete disappearance of the old civilities. Social life, street life, and attention to people and things around one have largely disappeared, at least in big cities, where a majority of the population is now glued almost without pause to phones or other devices—jabbering, texting, playing games, turning more and more to virtual reality of every sort.

It reminded me of this quote by Wilson Miner:

The car shaped our environment in the 20th century in this huge, tectonic way. I don’t think it’s a stretch to say that the screen will be as important to shaping our environment in the 21st century.

I’m not sure if he meant this as a warning, but considering how little we like being in neighborhoods that are built more for cars than for pedestrians, I think it should be interpreted as one.

That last quote was from the 2011 talk When We Build, which is a great talk. One of the interesting observations in there is a minimal history of computers that goes roughly like this:

We started off with computers that were big, clunky and slow with punch cards and batch processing. Then we had computers with screens. They were a little smaller but more importantly the screen allowed interactivity, changing the kinds of things you could do with computers. Then we had computers with screens and networks, and the networks once again changed what we could do. After that we stopped adding things and instead took things away: The networks became wireless, invisible. We were back to computers with screens and the network was implied, always there and always online. But then the computers became smaller and soon disappeared, too, so that all we’re left with is screens. It’s just assumed that anything with a screen has a computer in it.

This is the starting point for his talk and he talks about the importance of screens and that is the context for the above quote. But as Oliver Sacks pointed out, there are problems with screens. And an interesting thing happened: Screens started disappearing, and now it seems like the future of computers might be screen-less, which could potentially save us from the isolating world of everyone staring at their screens.

An obvious example of this are smart speakers like Amazon Echo, which remind us of the sci-fi computers of Star Trek. I like the idea of a smart speaker, but I personally have little faith that they’re the future. At least not anytime soon. Technology will have to improve a lot before we can just talk with a computer and get the same quality of interaction that we get from a screen. You want the exactitude of tapping on a screen and knowing that your input connected, and knowing that no matter how fast you go the input can keep up. Also you need a screen to give visibility into what the computer is actually doing.

Another form of the screen disappearing is VR. Which sounds counter-intuitive, because you’re just putting the screen right up to your face. But it feels very different. In VR it feels like you’re in a separate world, not like you’re looking at a screen behind which there is a separate world. And most importantly you’re actually using your hands to grab things and move them around in 3D, rather than having to tap on a screen and having to use a 2D input abstraction. Of course VR is even more isolating than cell phones, so it hardly seems like a solution for the problems that I started this blog post with.

Maybe AR is a better approach. I have less experience with it, mainly because there isn’t a good cheap AR set out there yet.

But my favorite example of screen-less computers has got to be Dynamicland. Maybe because it has the characteristics that Italo Calvino talked about in his Six Memos for the Next Millenium. In Dynamicland the room is smart, and any piece of paper can be imbued with the smarts of a computer. And there’s nothing special about that paper. If you have a piece of paper that can act as a map, you can make a photocopy of that paper in a normal photocopier, and the copy will also act as a map. And suddenly you’re not looking at a screen any more. Reporting about Dynamicland is full of stories about how people interact much more naturally when they are interacting with real pieces of paper than with screens. Just take this brief quote from the beginning of the Geokit write-up:

[…] she began describing her work while we were gathered around the lunch table, which happened to have Geokit spread across it looking at Oakland. Without breaking the flow of conversation, I dealt the transit card to display the bus routes, and she grabbed the “zoom and pan dial” without any instruction to zoom in on a portion of West Oakland — noticing a huge hole in route coverage she’d never seen before where she knew there were tons of working families. We spent the next 15 minutes exploring as she taught me more about the city’s transit details than I ever knew I wanted to know.

[…]

In theory you could pull this data up on your personal iPad, praying there aren’t any embarrassing notifications and hopefully remembering what you were even going to look at by the time you’re past the lock screen looking at 24 rainbow-gradient icons and red bubbles, but I am personally not in the habit of pulling out my devices in the middle of parties

It’s much easier to talk to people about a shared map that’s spread out across a table than about a tiny map on somebody’s screen. Or to pass the controls around if everything is controlled using pieces of paper. And if you and your spouse were sitting in your living room, both doing something with pieces of paper, it’s much easier to join the other person than if they were looking at their own private screen.

If we think further into the future, stories from sci-fi come to mind where computers are just everywhere. People don’t even carry terminals around with them, because they can just talk to the air and a computer can tell them the thing they’re asking about, or project the information onto a nearby surface. But what I like about Dynamicland is that it requires little magic. It doesn’t require an omnipresent AI and complex software that runs somewhere in the cloud. The code is small and it’s right there and it works with current tech.

The missing piece is obviously that Dynamicland is not as portable as cell phones. So it probably won’t replace them. But maybe it’s enough that it replaces the screens in offices or at homes. It is much more fully featured than cell phones already in that regard: The map application that I linked to above was written on pieces of paper using only Dynamicland concepts. Which is very different from the map application on your phone, which was almost certainly not written on a phone.

And then maybe, if Dynamicland manages to remove screens from offices and homes, we can figure out how to remove screens from cell phones after that.

And then the evolution of computers => computers with screens => computers with screens and networks => computers with screens => screens will end with the screen disappearing, too. Computers will just be everywhere and normal household objects will be smart. (and not like a “smart fridge” which is a fridge with a screen added, but smart without a screen) Instead of having the isolation of people staring at their private screens, people will interact with shared objects that know how to talk to each other. It seems like the only logical next step.