The Big Mistakes of VR
The Lack of Quantitative Theory of the Human
I’m fairly bearish on VR, and even more so on the Metaverse.
With regards to VR though, it’s not necessarily that the tech is inherently bad, but rather that it is largely misunderstood and billions are being wasted on the wrong applications.
You would think software as a field would be very quantitative and rigorous. In some ways it is with its data and algorithms and such, but in many other ways it is notoriously dogmatic and superstitious. Religious wars over text editors and the like.
Programming languages for example contain a great deal of theory regarding parsing, type checking, analysis, code generation, and so on, though I would argue that some of these (parsing for example) are far more contrived than many would like to admit and others (code generation) fairly specific to arbitrary details of modern hardware.
On the other hand though, there’s a great deal of modern programming language design that lacks any proper theory at all. While parsing has some basis in math, the syntactic structures built using these parsing tools are more a product of fashion and the software industry’s collective unconscious.
Perhaps we can assume that the seemingly excessive number of programming languages has created a natural selection process that’s searched most of the meaningful niches, but humans don’t really think the same way evolution does. We’re far more mimetic, and are completely content to make a million programming languages with utterly meaningless syntax and type system tweaks, copying the same tropes and motifs that everyone else copies, rather than experiment with anything actually unique.
The real problem with programming languages here, one that serves a much more pervasive problem in computer science, and one that I see arising in this situation with VR, is that we have no real quantitative theory of how the human integrates with the tools we build.
One example of this that I find particularly interesting is code indentation. Early languages such as COBOL and FORTRAN often made little to no use of indentation, but by now it’s a practically universal feature. It’s also given relatively little attention theoretically.
Human vision works through saccades - through the eye scanning the small, high-resolution fovea across and jumping around an image rather than taking the whole thing in at once, and guiding the fovea’s movement with a low-resolution peripheral vision. This being the foundation of human vision has resulted in the visual arts being full of ideas about composition, leading lines, and similar tools for manipulating this process to convey meaning. Indentation is clearly a similar tool, but the theories from neuroscience and the visual arts seem to be largely unconsidered by computer science regardless.
A real theory here could probably provide us with a better idea of how to take this idea much further. If indentation is valuable enough to become a universal language feature despite so little theoretical thought addressing it, I expect it’s likely to be just the tip of the iceberg of what’s really possible once we get a better idea of what the possibility space really looks like.
While there have been a great number of attempts at graphical programming languages, they largely make the exact same couple of mistakes every time. No, colored blocks of code don’t really provide anything that syntax highlighting doesn’t already provide and are just a less efficient use of pixels. Drag-and-drop makes interaction slower. Drawing code as a network of nodes connected by lines and arrows has some potential merit to it, but scales incredibly poorly. Trying to make any nontrivial code look like anything but a giant incomprehensible hairball is an exercise in fighting impossibility theorems from Topological Graph Theory, a fight you’re mathematically guaranteed to lose. A hairball is a massive step backward if your goal is code that’s easier to comprehend.
There’s regardless a draw to graphical programming - there’s a reason why there’s a million failed projects, there’s a reason it’s a holy grail. Code indentation is just a mere hint at something deeper, richer, more powerful, and people can feel it! People can tell that there’s something valuable that can be done with conveying meaning and structure through the spatial arrangement of code, but actually achieving that happens to not be the low-hanging fruit everyone expects, and no genius designer has come along to point out the right path.
Design and engineering are orthogonal skills. It’s no surprise to me how heavily Christopher Alexander’s brilliant ideas behind pattern languages and theoretical design were butchered and completely misunderstood by the “OOP Design Patterns” crowd. Then again, Alexander’s work is not easy to understand in general.
Okay, I think we can talk about VR now.
Immersion is a Scam
Immersion is a scam. It’s the wrong metric to be optimizing for in VR. It’s hardly a metric at all, and is ill-defined to the point of being borderline meaningless.
A good book can be pretty immersive too, which is something I’ve found that VR fanatics don’t appreciate hearing.
If “immersion” is some kind of measure of being indistinguishability from reality as is so often implied, how “immersive” can VR really be when you have to constantly deal with the constraints and limitations of the technology? Is it “immersive” to be in VR when you’re just sitting down all day? Okay, maybe you get a headset that lets you move around and tracks your movement. Now you have to avoid moving into the tracking cameras’ blind spots, or walking into a wall. Anything you do in VR is either going to be tightly constrained, or is going to require you to constantly be thinking about the outside world. That sounds like the opposite of immersion to me.
There are parts of VR tech that I think people are kind of getting right, even in this faulty pursuit of immersion. My real critique is that “immersion” is merely a proxy for a combination of other, more objective factors, and that those factors would be a much better thing to focus on.
VR is not really an immersion technology, but rather one about increasing bandwidth, decreasing latency, and expanding the sensory domain of the human-computer interface.
A mouse or touchpad has 2 degrees of freedom. A keyboard increases that to some extent, adding a few extra bits per second of input bandwidth. A head-mounted display with motion sensors has 3 degress of freedom for translation and 3 for rotation, six total. Add in handheld controllers and you’re up to 18, not counting all the buttons and finger position sensors on the controllers.
The auditory cortex doesn’t just process the sounds we hear, but also can influence motor movements, particularly those involved in turning the head, often resulting in small, subtle movements in response to sounds. Movements that would be easy to detect with an HMD.
Multisensory computing expands the number of brain regions involved in our work, effectively increasing the brain power we allocate to our work. Sitting at your laptop or phone and tapping on the keyboard or screen uses only your visual cortex and to a much lesser extent your somatosensory and motor cortices.
Bringing in audio, motor movements beyond just your fingers and eye muscles, and adding some haptic feedback and head movements greatly expands the percentage of the brain actively involved in computer interfactions, likely expanding neural computing power available in such interactions as well.
The goal of some brain implant companies like Neuralink is to expand the bandwidth channel between humans and computers, but really we should be seeing VR in a similar way. VR is not about perfectly simulating and escaping into some virtual world; this is nothing more than a pipe dream.
Pulling larger numbers of bits per second out of the brain allows richer interactions with computers. Lower latency connections, measuring subtle and reflexive motions in response to stimuli increases this further and tightens the feedback loop between man and machine. Multisensory compute makes more efficient use of brainpower and leaves little of the brain unoccupied or uninvolved. Together, these can help us build much more powerful computational tools, tools that in many ways may more closely resemble the real-world tools humans have been using for millions of years.
VR is not about immersion. It’s a kind of non-invasive brain-computer interface, and we should think of it as such.
A Brief Aside on Brain Implants and VR
No, some kind of direct brain interface is unlikely to make any difference on VR immersion any time soon.
Sensory input doesn’t go straight into the cortex, but passes through a brain region called the thalamus first. The thalamus is deep at the center of the brain, far deeper than any brain-scanning technology can read from with any acceptable resolution. Accessing such deep brain regions is really only possible with invasive brain implants, and even then is a very risky and difficult procedure. This isn’t just any ordinary invasive brain implant we’d be talking about, but the most dangerous and difficult kind, and likely an exceptionally ambitious one at that if you want any considerable amount of bandwidth into the thalamus.
Maybe we could feed data directly into the cortex (which is much easier and safer to access), but we don’t know what the consequences of what the long-term bypassing of the thalamus does to how the brain processes information. Based on the fact that the thalamus is extremely closely connected to pretty much everything that the cortex does, and the fact it seems to play a major role in attention and consciousness, I don’t think it’s too unreasonable to suspect that bypassing it entirely would likely have a strong negative impact on “immersion”.
The Metaverse Dystopia
Art is memetic infrastructure. In much the same way that people rarely build their homes far from roads or towns, humans rarely explore ideas far beyond those they’re familiar with, and ideas are unlikely to spread without some amount of widespread familiarity.
A popular book, film, game, or other piece of media can help popularize an idea, planting the seeds into many people’s minds. A pervasive idea is easy for people to discuss with others, and easy to build upon. An obscure idea only understood by a small range of experts may struggle to get the attention it may deserve by simply being difficult to spread.
This goal of complete escapism in VR is a terrible idea to begin with. I know very few people who don’t see the “Metaverse” as a dystopian hellscape. Regardless, the idea persists and spreads.
The thing about dystopias is that they’re not just really terrible societies to live in, but they’re generally not very sustainable either. Authoritarianism for example usually just leads to situations such as a fallible leader shooting the messengers until she is unable to make accurate decisions anymore. The kind of diversity and dynamism authoritarian societies suppress is the exact thing they need most to remain robust in a world that no one can fully understand.
A world where everyone is plugged into the Metaverse is one where no one is doing real work. No one is farming, no one is running the chip fabs to make new headsets. The tech stack to maintain modern chip production is immense, and difficult to maintain. Getting good at VR will likely require even more sophistication. Sure, automation may replace some work, but 100% automation is extremely rare if it ever happens at all, and any society that can actually maintain a Metaverse will necessarily have a lot of work that needs to be done outside of it. Everyone escaping into the Metaverse would quickly lead to a collapse, and the Metaverse would likely be among the first things to break.
Automation and AI is a discussion I’d prefer to save for another time.
The Metaverse is a dystopia. It was always meant to be a cautionary tale, something to be avoided, but we were never given a compelling positive alternative. Unfortunately, imagining visions of the future and actually executing on them are orthogonal skills. Engineers will inevitably built a dystopia if no one describes an alternative to them.
A world where humans can be more closely integrated with computers doesn’t necessarily have to be a dystopia. It could be a vast improvement to society, if done right. I’d personally love computational tools closer to the kinds of tools that humans have been using for millions of years.
Luckily, dysfunctional and unstable societies are just that - unstable - and tend to trend toward more stable societies over time. Either that, or collapse. Of course, if no one wants to buy a VR headset on account of it being too blatantly terrible, we can probably bypass that kind of dystopia entirely.
I expect Augmented Reality will be much more valuable than Virtual Reality, as reality is what really matters. Augmenting our ability to process reality is what humans do best, whether that be though language, through writing, or through tools. Still being able to see the environment around you seems better than trying not to walk into a wall or trip over a cable when you’re moving around with a screen strapped to your face.
Evolution created brains so that animals could better navigate their environments. If evolution ever created animals that used their brains primarily to escape into a safer fantasy world, those animals have clearly gone extinct, replaced by animals with world-navigating brains. VR will be no different.
If you enjoyed this article, and find ideas like this interesting, consider sharing and subscribing to my newsletter. Most articles are free, though the occasional paid article slips in from time to time. Paid subscriptions are $7 per month or $75 per year.