Internalizing Simulated Systems: Manipulation within Virtual Environments

Spatial computing and virtual environments like VR are more powerful mediums than screen-based mediums in that they leverage our bodily fluency with spatial and physical interactions. Further, as computers can simulate and depict arbitrary [e.g. any/unspecified parameter set] physical systems, the environments depicted can behave according to laws other than the laws of physics that material artifacts are constrained by.

By permitting such “exotic” systems, virtual environments widen the design space, allowing the development of more nuanced tools and representations.

 

Manipulability

The main way that humans interface with their environment is through their hands, protrusions capable of orienting and manipulating in three dimensions. Humans further gain an understanding of their environment through senses around their bodies and heavily localized in the head. Modern spatial computing systems track the position and orientation of the head and hands, rendering a scene from the viewpoint of the user’s eyes in perfect synchrony with their motion, giving the illusion of presence within a scene. Tracking the position and actions (like grabbing) of the hands allows the user to manipulate objects within the rendered scene.

As we develop from infancy we develop spatial intuitions and a fluent sense of body through continuous interaction with the material world. However these nuanced abilities have been underutilized by screen-based dynamic media, trapping interactions onto two-dimensional touchscreens or constrained, indirect interaction surfaces such as mice and keyboards.

Spatial computing combines the flexibility of dynamic depictions with interactions approaching the spatiality and manipulability of material environments and objects.

Manipulations are powerful not only because they transform the interacted entity and thus the perception of the system, but, critically,  because they allow the body and mind to internalize the dynamics of the system interacted with (Hutchins, 1995, p140).

 

Internalization

The soroban is a prime example of manipulability’s importance. However, to assess it accurately, the traditional electronic calculator must be invoked.

When using an electronic calculator, the only actions the user participates in is the setup of the mathematical statement, inputting digits and algebraic operations. Once the equals key is pressed, all of the mathematical operations involved in solving the question occurs outside of the user’s perception, invisibly within the calculator. When the user receives the answer without themselves going through the steps, their perception of the mathematical relationships suffers, and their arithmetical abilities atrophy.

The soroban, on the other hand, involves intimate user manipulation to enact every step. It represents digits via the placement of pegs on decimal-place wires, and the user moves the pegs up and down in correspondence with the shifting placement of values during mathematical operations. Since the soroban requires explicit user manipulation to advance the mathematical operation, the user is a direct participant in every step. Such intimate involvement in the operations allows the body and mind to internalize the soroban’s structure. The user develops not only a muscle memory for the location and dynamics of the pegs, but over time builds an internalized, mental representation of the system (Hatano, 1988, p64). This is evidenced in relative novices but occurs to an even greater extent in seasoned users. With enough practice, soroban users do not even need a physical soroban around in order to perform calculations. They have internalized the soroban’s structure so completely that they can calculate massive problems using a purely imagined construct, perhaps rapidly waving their fingers in the air correspondent to the physical manipulations they have so fully internalized. “Sensorimotor operation on physical representation of abacus beads comes to be interiorized as mental operation on a mental representation of an abacus. By this, the speed of the operation is no more limited by the speed of muscle movement” (Hatano, 1988, p64).

Such an example demonstrates the power of bodily manipulation. Given enough time interacting with a system, the body and mind can internalize its dynamics and structure, building, at least in part, a robust mental representation. Since virtual environments supporting hand-presence empower users to manipulate their surroundings, users are that much more able to internalize the dynamics of the interacted systems, developing stronger mental models, growing more fluent at operating within the system, and, perhaps, developing mental representations usable outside of the virtual environment (Hutchins, 1995, p171).

Internalization need not be an exclusively intellectual phenomenon. Somatic internalization occurs when one develops the ability to balance a stick on a finger, as the body’s perception of force, pressure, and proprioception is correlated with visual feedback of stick angle (Heersmink, 2014, p58). The behavior of the overall system is initially alien but over time is explored and eventually becomes second-nature. Such is also the case for learning to drive a vehicle, painting, tying shoes, etc.. Any repeated collision with a manipulable system with bounded possibility-space [ ...as an unbounded space would produce infinite novelty and thus make long-term correlations difficult or impossible] will eventually produce some level of internalization.

Certain objects can be internalized in such a way that they come to be treated by the body as an extension of itself. The perceived locus of interface when using a pencil is at its tip and the paper surface, even though the body terminates at the end of its fingers and the edge of the pencil (Heersmink, 2014, p59). It is as if the pencil has been incorporated into the body schema of the user (Heersmink, 2014, p59). Similarly, 

For the blind man, the cane is not an external object with which he interacts, but he interacts with the environment through the cane. The focus is on the cane-environment interface, rather than on the agent-cane interface. The cane is furthermore incorporated into his body schema and is experienced as a transparent extension of his motor system. (Heersmink, 2014, p59)

Otherwise dynamic media that in some way hinder manipulation consequently hinder their ability to be internalized by the body and mind. More restrictive control surfaces such as mice and keyboards constrain possible manipulations to a small subset of what the body is capable of, and purely visual feedback (on a screen, indirect and away from the control surface) limits the depth of internalization. 

 

Depiction

The second critical feature of virtual, spatial systems is that they can depict objects, scenes, and transformations that are materially impossible (Biocca, 2001). While screens have classically been able to depict arbitrary visual arrangements including “impossible” or exotic arrangements, virtual environments offer the added benefit of robust spatial manipulability. Humans have traditionally been capable of only designing spatially-manipulable systems and tools within the constraints of material physics. With VR that veil has been lifted, opening the interaction-design space to novel tools and interactions previously impossible to not only manifest but possibly also conceive.

Leveraging our tendency to internalize systems we repeatedly interact with with the capacity to represent previously unrepresentable systems inaugurates a new relationship with theory. Previously, if we had developed a new theory or model of phenomena too large or small to be within the bounds of our physical interaction, we could only interface with abstracted versions of it, perhaps only through written or drawn notation. Now we have the capability of simulating such systems in ways that they are manipulable, allowing us to develop spatial intuitions from repeated interactions, possibly internalizing aspects that would have been otherwise invisible in less-realized or -manifested representations or notations.

Ryan Brucks’ parameter space value-finder is a powerful example of the sorts of systems that dynamic media can support (Brucks, 2017). Seeing it in motion communicates its dynamics better than a text description, so the link to the original Twitter post is included. Brucks arranged a two-dimensional grid of eyeballs freely rotatable in their spots, all attempting to aim at the location of his cursor. Critically, each eyeball has a different value of two parameters (speed of alignment to cursor and amount of spring dampening) set up as axes of the grid. As Brucks moves the cursor over the grid, each eye reacts slightly differently, its dynamics and behavior made visible and unique in comparison with its neighbors. Admittedly a surreal (and relatively simple) system, it serves to demonstrate what options exist for surveying parameter-space incorporating a combination of spatial manipulability and arbitrary physics. One imaginable application is using a similar setup to survey possible behaviors of a paintbrush/manipulator/tool in VR and directly plucking out the toolhead with the intended parameters as a sort of reactive, surveyable toolbar.

Representation

There are essentially infinite spatial arrangements of objects, only a subset of which are possible in physical space. This has classically severely constrained what types of tools could be created, designed, or even conceived of. Humanity needs powerful, manipulable representations and cognitive tools. 

“The sciences frequently run up against the limitations of a way of representing aspects of the world” (Gooding, 2001, p131). Early algebra was stuck in long-form paragraphs, and because that format was so unmanipulable, algebra barely advanced. It wasn’t until Descartes developed modern algebraic notation that algebra’s representation allowed a modular manipulability, and mathematical advancement skyrocketed.

There are many complex systems that have previously been unrepresentable, or our confusion about them has stemmed from improperly constrained representations.

Conclusion

This new age of spatial arrangements allows for novel representations and explorable systems, giving us better understanding of the most important systems around us. Spatial computing and VR are still infant media, and such an unconstrained design space is daunting, but they are likely the best tool yet in our attempt to understand the universe.

 

 

References

Biocca, F. (2001). The space of cognitive technology: The design medium and cognitive properties of virtual space. In Cognitive Technology: Instruments of Mind (pp. 55-56). Springer, Berlin, Heidelberg

Brucks, R. [@ShaderBits] (2017). "Fun way to find ideal values in 2d parameter space. Damping decreases left to right, speed decreases front to back.” https://twitter.com/shaderbits/status/939302802098188292

Gooding, D. C. (2001). Experiment as an instrument of innovation: Experience and embodied thought. In Cognitive Technology: Instruments of Mind (pp. 130-140). Springer, Berlin, Heidelberg.

Hatano, G. (1988). Social and motivational bases for mathematical understanding. New directions for child and adolescent development, 1988(41), 55-70.

Heersmink, J. R. (2014). The varieties of situated cognitive systems: embodied agents, cognitive artifacts, and scientific practice.

Hutchins, E. (1995). Cognition in the Wild. MIT press.

‘Life’ and Distinctions Among Matter

Schrödinger asks, in What is Life? The Physical Aspect of the Living Cell’s chapter Living Matter Evades the Decay to Equilibrium, “What is the characteristic feature of life? When is a piece of matter said to be alive?” Few would claim that a single atom is alive, or even a single molecule. It is necessary, then, when observing collections of molecules, to discern what structure and activity is present to warrant the claim of life.

The systems that we call alive are ordered in a way that is distinct from systems that we do not call alive. A periodic crystal is highly ordered, a sometimes perfect tessellation of its structural pattern, but it doesn’t act in a way that we would correspond with life. A rock does not act in such a fashion either. 

We look at lions and we see motion, we see intent to devour. They can be an active threat to us. We look at cows and come to understand that they eat and can be eaten, providing sustenance. Though their motion is achingly slow, we observe similar traits in plants—predation, the possibility of sustenance, etc. Even the ‘intent’ to grow towards the light. No such action do we observe in things we would not call alive, such as a rock.

It makes sense why this distinction (life versus non-life) would be useful to us. A rock on the ground need not pose a threat in the way a prowling lion certainly would, and it cannot provide sustenance in the way a juicy flank or orange can. We observe a whole class of entities that seem to share many aspects of action, and we begin to delineate them based on structure and operation, giving us categories of animal, plant, fungus, etc. As our awareness and knowledge of our environment expands, we uncover more entities, often with novel aspects, yet we still fit them within our categories with relative ease.

So it is that we come to a set of ‘requirements’ for what is life. Self-replication, growth, adaptation, etc. Schrödinger poses the question: “How does the living organism avoid decay? The obvious answer is: By eating, drinking, breathing and (in the case of plants) assimilating. The technical term is metabolism.” If we were to find an entity that possesses only some of these aspects, many people would claim that such an entity was “not alive” because it did not take part in every aspect of our definition. However, how is this intellectually honest? We observe many entities, and from the gamut of our experience, we create a delineation informed by the aspects that we observe in the entities. If we come to a new entity that does not share in all of those aspects, we say it is not alive. We anticipate nature when we create a definition and then apply it to nature, rather than letting nature inform the definition. In this example, the virus is an entity that does not take part in every aspect of our definition of life. It does not metabolize. Yet it reproduces and adapts. However, these aspects are not where we should look. To gain a sense for the relatedness of entities, emergent operation should not be the rubric. Instead the structure and how it produces the observed actions should be the rubric for relatedness.

We possess a general understanding of the presence of nucleic acids in ‘living’ entities and how their operation culminates in the larger action of the ‘living’ body. We say that many aspects of the entity result from the action of its nucleic acids, which in turn result from the structure of the nucleic acids themselves. 

We look at viruses and come to understand that their action is a result of the presence of nucleic acids with a genome that specifically determines their cycle of action. The whole species of nucleic acids, spanning plants, animals, microorganisms, and viruses, act similarly because they are of a like structure. We came up with a definition for life that was not informed by all available entities, and when we observe an entity that falls outside the definition, we discard its plausible inclusion into the hallowed ranks of life. This is not an honest way of defining. What even is defining, with regard to Nature? Are some delineations more valid than others? Surely this is so. We can observe the properties of a gram of mercury versus a gram of argon and be able to delineate them. The definition of life, on the other hand, is more suspect. For millennia, every time we found a new animal, it appeared and acted similarly enough to animals already held as alive to be itself considered alive. Such is the case with plants and fungi. These macroscopic entities shared a like nucleic structure, which in turn determined their like macroscopic structure and action, which was the basis of their being lumped into the same category of ‘living things’.

Once we became able to observe scales previously invisible to us, we found entities (microorganisms: bacteria, protists, etc.) that shared like action with macroscopic entities. This enabled their quick inclusion in to the ranks of living things. However, once we came to viruses, their unlike action (in some respects, namely their lack of growth and metabolism) had us claiming that they could not be alive because they did not share in all the characteristics seen elsewhere. Yet when we looked, we came to know that they possessed similar nucleic acids, and their nucleic operation was nigh identical to macroscopic entities. We came to understand that every entity we called alive shared the same foundation of operation, the nucleic acid. The fact that the operation of nucleic acids can produce an entity that does not need to metabolize is grounds to dissolve the definition for life.

What do we want a definition for, to begin with? We wish to delineate thing from thing, to find the basis of similarity and difference among what we observe. We can now look at all of ‘life’ around us as the product of the long-term evolution of nucleic acids. It’s astonishing how variable this species of molecule is. Its successful self-replication for billions of years has produced structures from viruses to aspens, multi-celled organisms from which a single cell can be separated and grown in isolation, massive colonies of insects that function cohesively, patterns of intelligence emerging from the summation of simple parts.

It’s possible that in the future we could find things that would externally appear “alive” as we recognize it today, whose structure does not utilize nucleic acids. However be the structure of their functioning, if it was locally anti-entropic and could self-replicate, we would still be able to distinguish them from our earthy ‘living things’ because their structure was distinguishable.

By relinquishing our definition for life that was created prior to our larger knowledge of the operation of ‘living’ entities, our understanding of the world around us can become fully a result of what we observe, rather than our applying ideas past their plausible relevance.

The distinction between rock and giraffe is a real one, and we now exist in a world where we are acutely aware of what makes them different. Gone are the days when the only observations we had were macroscopic. Understanding the physicochemical makeup of our objects of interest directly informs a more exhaustive understanding of their macroscopic action. The distinction lies in their makeup, not in their large-scale action. Convergent evolution produces similar large-scale actions, but the nucleus of similarity is derived from the hereditary chain, and the hereditary chain lies in the evolution of nucleic acids.