Spatial computing and virtual environments like VR are more powerful mediums than screen-based mediums in that they leverage our bodily fluency with spatial and physical interactions. Further, as computers can simulate and depict arbitrary [e.g. any/unspecified parameter set] physical systems, the environments depicted can behave according to laws other than the laws of physics that material artifacts are constrained by.
By permitting such “exotic” systems, virtual environments widen the design space, allowing the development of more nuanced tools and representations.
Manipulability
The main way that humans interface with their environment is through their hands, protrusions capable of orienting and manipulating in three dimensions. Humans further gain an understanding of their environment through senses around their bodies and heavily localized in the head. Modern spatial computing systems track the position and orientation of the head and hands, rendering a scene from the viewpoint of the user’s eyes in perfect synchrony with their motion, giving the illusion of presence within a scene. Tracking the position and actions (like grabbing) of the hands allows the user to manipulate objects within the rendered scene.
As we develop from infancy we develop spatial intuitions and a fluent sense of body through continuous interaction with the material world. However these nuanced abilities have been underutilized by screen-based dynamic media, trapping interactions onto two-dimensional touchscreens or constrained, indirect interaction surfaces such as mice and keyboards.
Spatial computing combines the flexibility of dynamic depictions with interactions approaching the spatiality and manipulability of material environments and objects.
Manipulations are powerful not only because they transform the interacted entity and thus the perception of the system, but, critically, because they allow the body and mind to internalize the dynamics of the system interacted with (Hutchins, 1995, p140).
Internalization
The soroban is a prime example of manipulability’s importance. However, to assess it accurately, the traditional electronic calculator must be invoked.
When using an electronic calculator, the only actions the user participates in is the setup of the mathematical statement, inputting digits and algebraic operations. Once the equals key is pressed, all of the mathematical operations involved in solving the question occurs outside of the user’s perception, invisibly within the calculator. When the user receives the answer without themselves going through the steps, their perception of the mathematical relationships suffers, and their arithmetical abilities atrophy.
The soroban, on the other hand, involves intimate user manipulation to enact every step. It represents digits via the placement of pegs on decimal-place wires, and the user moves the pegs up and down in correspondence with the shifting placement of values during mathematical operations. Since the soroban requires explicit user manipulation to advance the mathematical operation, the user is a direct participant in every step. Such intimate involvement in the operations allows the body and mind to internalize the soroban’s structure. The user develops not only a muscle memory for the location and dynamics of the pegs, but over time builds an internalized, mental representation of the system (Hatano, 1988, p64). This is evidenced in relative novices but occurs to an even greater extent in seasoned users. With enough practice, soroban users do not even need a physical soroban around in order to perform calculations. They have internalized the soroban’s structure so completely that they can calculate massive problems using a purely imagined construct, perhaps rapidly waving their fingers in the air correspondent to the physical manipulations they have so fully internalized. “Sensorimotor operation on physical representation of abacus beads comes to be interiorized as mental operation on a mental representation of an abacus. By this, the speed of the operation is no more limited by the speed of muscle movement” (Hatano, 1988, p64).
Such an example demonstrates the power of bodily manipulation. Given enough time interacting with a system, the body and mind can internalize its dynamics and structure, building, at least in part, a robust mental representation. Since virtual environments supporting hand-presence empower users to manipulate their surroundings, users are that much more able to internalize the dynamics of the interacted systems, developing stronger mental models, growing more fluent at operating within the system, and, perhaps, developing mental representations usable outside of the virtual environment (Hutchins, 1995, p171).
Internalization need not be an exclusively intellectual phenomenon. Somatic internalization occurs when one develops the ability to balance a stick on a finger, as the body’s perception of force, pressure, and proprioception is correlated with visual feedback of stick angle (Heersmink, 2014, p58). The behavior of the overall system is initially alien but over time is explored and eventually becomes second-nature. Such is also the case for learning to drive a vehicle, painting, tying shoes, etc.. Any repeated collision with a manipulable system with bounded possibility-space [ ...as an unbounded space would produce infinite novelty and thus make long-term correlations difficult or impossible] will eventually produce some level of internalization.
Certain objects can be internalized in such a way that they come to be treated by the body as an extension of itself. The perceived locus of interface when using a pencil is at its tip and the paper surface, even though the body terminates at the end of its fingers and the edge of the pencil (Heersmink, 2014, p59). It is as if the pencil has been incorporated into the body schema of the user (Heersmink, 2014, p59). Similarly,
For the blind man, the cane is not an external object with which he interacts, but he interacts with the environment through the cane. The focus is on the cane-environment interface, rather than on the agent-cane interface. The cane is furthermore incorporated into his body schema and is experienced as a transparent extension of his motor system. (Heersmink, 2014, p59)
Otherwise dynamic media that in some way hinder manipulation consequently hinder their ability to be internalized by the body and mind. More restrictive control surfaces such as mice and keyboards constrain possible manipulations to a small subset of what the body is capable of, and purely visual feedback (on a screen, indirect and away from the control surface) limits the depth of internalization.
Depiction
The second critical feature of virtual, spatial systems is that they can depict objects, scenes, and transformations that are materially impossible (Biocca, 2001). While screens have classically been able to depict arbitrary visual arrangements including “impossible” or exotic arrangements, virtual environments offer the added benefit of robust spatial manipulability. Humans have traditionally been capable of only designing spatially-manipulable systems and tools within the constraints of material physics. With VR that veil has been lifted, opening the interaction-design space to novel tools and interactions previously impossible to not only manifest but possibly also conceive.
Leveraging our tendency to internalize systems we repeatedly interact with with the capacity to represent previously unrepresentable systems inaugurates a new relationship with theory. Previously, if we had developed a new theory or model of phenomena too large or small to be within the bounds of our physical interaction, we could only interface with abstracted versions of it, perhaps only through written or drawn notation. Now we have the capability of simulating such systems in ways that they are manipulable, allowing us to develop spatial intuitions from repeated interactions, possibly internalizing aspects that would have been otherwise invisible in less-realized or -manifested representations or notations.
Ryan Brucks’ parameter space value-finder is a powerful example of the sorts of systems that dynamic media can support (Brucks, 2017). Seeing it in motion communicates its dynamics better than a text description, so the link to the original Twitter post is included. Brucks arranged a two-dimensional grid of eyeballs freely rotatable in their spots, all attempting to aim at the location of his cursor. Critically, each eyeball has a different value of two parameters (speed of alignment to cursor and amount of spring dampening) set up as axes of the grid. As Brucks moves the cursor over the grid, each eye reacts slightly differently, its dynamics and behavior made visible and unique in comparison with its neighbors. Admittedly a surreal (and relatively simple) system, it serves to demonstrate what options exist for surveying parameter-space incorporating a combination of spatial manipulability and arbitrary physics. One imaginable application is using a similar setup to survey possible behaviors of a paintbrush/manipulator/tool in VR and directly plucking out the toolhead with the intended parameters as a sort of reactive, surveyable toolbar.
Representation
There are essentially infinite spatial arrangements of objects, only a subset of which are possible in physical space. This has classically severely constrained what types of tools could be created, designed, or even conceived of. Humanity needs powerful, manipulable representations and cognitive tools.
“The sciences frequently run up against the limitations of a way of representing aspects of the world” (Gooding, 2001, p131). Early algebra was stuck in long-form paragraphs, and because that format was so unmanipulable, algebra barely advanced. It wasn’t until Descartes developed modern algebraic notation that algebra’s representation allowed a modular manipulability, and mathematical advancement skyrocketed.
There are many complex systems that have previously been unrepresentable, or our confusion about them has stemmed from improperly constrained representations.
Conclusion
This new age of spatial arrangements allows for novel representations and explorable systems, giving us better understanding of the most important systems around us. Spatial computing and VR are still infant media, and such an unconstrained design space is daunting, but they are likely the best tool yet in our attempt to understand the universe.
References
Biocca, F. (2001). The space of cognitive technology: The design medium and cognitive properties of virtual space. In Cognitive Technology: Instruments of Mind (pp. 55-56). Springer, Berlin, Heidelberg
Brucks, R. [@ShaderBits] (2017). "Fun way to find ideal values in 2d parameter space. Damping decreases left to right, speed decreases front to back.” https://twitter.com/shaderbits/status/939302802098188292
Gooding, D. C. (2001). Experiment as an instrument of innovation: Experience and embodied thought. In Cognitive Technology: Instruments of Mind (pp. 130-140). Springer, Berlin, Heidelberg.
Hatano, G. (1988). Social and motivational bases for mathematical understanding. New directions for child and adolescent development, 1988(41), 55-70.
Heersmink, J. R. (2014). The varieties of situated cognitive systems: embodied agents, cognitive artifacts, and scientific practice.
Hutchins, E. (1995). Cognition in the Wild. MIT press.