The depictive capabilities of spatial computing offer novel opportunities and challenges to the field of interaction design. I aim to articulate a framing of spatial computing that highlights unorthodox aspects of it, with the hope that such a framing might inform incoming designers, augmenting their preconceptions of how to approach this medium.
expectations from prior physics
The physics system that we find ourselves in at birth determines the nature of our bodies, the phenomena we encounter, and the affordances available in the environment. We become familiar with and develop intuitions about the ‘interactional grammars’ that we repeatedly come into contact with. Or, as Sutherland (1965) puts it, “We live in a physical world whose properties we have come to know well through long familiarity. We sense an involvement with this physical world which gives us the ability to predict its properties well.” This is the default state that designers have operated within since antiquity.
With the advent of computer-rendered dynamic media, phenomena could be represented that diverged from the phenomena driven by physical laws classically confining designed artifacts. This larger space of possible physical dynamics, of which the physics of our universe is but a subset, I refer to as hyperphysics. Since these phenomena are observed and interacted with by users who developed in ordinary physics, most users are presumably attuned to the nuances of phenomena (or do “not enter devoid of expectations that come from their previous experience” (Blom, K. J., 2007)) and may be immediately aware of the similarities, recognizing that “content that is familiar to the user from the real world will be initially and automatically considered the same as a real object” (Blom, K. J., 2010). Or, as Golonka & Wilson (2018) state: “When we encounter a novel object or event, it will likely project at least some familiar information variables (e.g., whether it is moveable, alive, etc), giving us a basis for functional action in a novel context”. The challenge is how to communicate “hyperphysical” affordances that do not have exact analogues in ordinary physics.
For example, many objects in rendered environments (such as depicted in “virtual reality” fields of view or superimposed on the outside world in “augmented reality” fields of view) are capable of being grasped and moved around, no matter their apparent mass, extent, smoothness, etc., even non-locally. Yet Gibson (1979)’s conception of what is graspable (granted, conceived prior to the availability of spatial computing) requires “an object [to] have opposite surfaces separated by a distance less than the span of the hand”. This requirement is now seen as being compartmentalized to only ordinary physics, but should designers of spatial user interfaces (SUIs) abandon it completely? Surely it’s useful to leverage the already-developed familiarity with ordinary physics’ interactional grammars, but at what expense? How tightly should SUIs be coupled to ordinary physics? What is conserved in intuitiveness is lost in the full exploration of the hyperphysics capable of being simulated, as “there is no reason why the objects displayed by a computer have to follow the ordinary rules of physical reality with which we are familiar”(Sutherland, 1965).
coherence and coordination of phenomena
Of course, the reason ordinary physics is intuitive is because we develop and spend our whole lives fully immersed in it. When the environment offers a consistent set of phenomena and consistent responses to input, the brain becomes accustomed to the perceived patterns and builds a set of intuitions about the phenomena. Piaget (1952) notices that “adaptation does not exist if the new reality has imposed motor or mental attitudes contrary to those which were adopted on contact with other earlier given data: adaptation only exists if there is coherence, hence assimilation.” This consistency comes from the fact that ordinary physics do not change over time or location, and the perception of the unity of events arises from multiple senses receiving coordinated impulses. In Gibsonian (1979) parlance, “when a number of stimuli are completely covariant, when they always go together, they constitute a single ‘stimulus’”. Piaget (1952), in noting that “the manual schemata only assimilate the visual realm to the extent that the hand conserves and reproduces what the eyes see of it”, communicates the unification of tactile and visual sensory input, that merely “the act of looking at the hand seems to augment the hand's activity or on the contrary to limit its displacements to the interior of the visual field.”
Usefully, since our bodies are themselves physical, we can directly impact the environment and observe the effects in realtime, becoming recursively engaged with the phenomena in question. Chemero (2009) describes this recursive engagement thusly:
Notice too that to perceive the book by dynamic touch, you have to heft it; that is, you have to intentionally move it around, actively exploring the way it exerts forces on the muscles of your hands, wrists, and arms. As you move the book, the forces it exerts on your body change, which changes the way you experience the book and the affordances for continued active exploration of the book.
This is assisted by the fact that our senses are not located exclusively in the head. “…in perception by dynamic touch, the information for perception is centered on the location of the action that is to be undertaken” (Chemero, 2009). Thus we can correlate the visual feedback of where, for example, the hand is amidst the environment, the proprioceptive feedback of the hand’s orientation relative to the body, and the tactile and inertial feedback provided by the environment upon the hand.
Being confined to the laws of ordinary physics, the parallel input sources agree, providing a consistent “image” of the environment. The fewer senses available, the less well-defined the final percept is, and partial disagreement between senses can override “anomalous” sense-inputs. This can lead to perceptual illusions like when, at a stoplight, a large bus in the adjacent lane begins to move forward, and provided it occupies an adequately large section of the visual field, the sensation of yourself moving backwards is induced, even if there is no vestibular agreement with the optical flow. Thus, to provide as rich and internally-coherent experience as possible, spatial computing systems need to provide many parallel sources of sensory input that agree, forming a unified sensory field. Sutherland (1965) agrees that “if the task of the display is to serve as a looking-glass into the mathematical wonderland constructed in computer memory, it should serve as many senses as possible.”
Two difficulties arise. The physical behavior of rendered environments depicted in spatial computing need not align with ordinary physics (the alignment in fact being a difficult if not impossible feat), and the rendered environments need not be internally consistent either (especially given that 1. simulated physics can change in realtime at the whim of the designer {something that ordinary physics is by definition incapable of} and 2. independent rendered-environment-designers can make available environments that have vastly different physics and thus different interactional “grammars”). Thus the lived experience of the user, navigating between ordinary physics and the variant and likely inconsistent physics of rendered environments, involves shifting between equally inconsistent interactional grammars. Will this have a negative affect on the brain? Will expertise with unorthodox physics developed in a simulated environment have a zero-sum relationship with the embedded expertise navigating ordinary physics? Is the brain plastic enough to contain and continue developing facility in an ever-increasing number of interactional grammars?
engagement with hyperphysics
The opportunities afforded by bodily engagement with hyperphysical simulated systems, however, are numerous. The usefulness of the environment is a function of its physical capacities, and thus the expanded set of hyperphysics within simulated systems supports, in principle, a proportionally-expanded usefulness:
Concepts which never before had any visual representation can be shown, for example the "constraints" in Sketchpad. By working with such displays of mathematical phenomena we can learn to know them as well as we know our own natural world. (Sutherland, 1965)
We lack corresponding familiarity with the forces on charged particles, forces in non-uniform fields, the effects of nonprojective geometric transformations, and high-inertia, low friction motion. A display connected to a digital computer gives us a chance to gain familiarity with concepts not realizable in the physical world. (Sutherland, 1965)
It is fundamentally an accident of birth to have been born into ordinary physics, but the mind is in principle capable of becoming fluent in many other physics:
Our perceptions are but what they are, amidst all those which could possibly be conceived. Euclidean space which is linked to our organs is only one of the kinds of space which are adapted to physical experience. In contrast, the deductive and organizing activity of the mind is unlimited and leads, in the realm of space, precisely to generalizations which surpass intuition. (Piaget, 1952)
A key constraint then becomes the ability of designers to envision novel physics to then manifest, as
computers are so versatile in crafting interactive environments that we are more limited by our theoretical notions of learning and our imaginations. We can go far beyond the constraints of conventional materials… (diSessa, 1988)
affordances
Hyperphysics supports novel behaviors that have no necessary analogue in ordinary physics. Thus the entire structural, visual, and dynamic “language” of ordinary affordances is inadequate to fully cover all possible transformations and behaviors that hyperphysics supports. Even fundamental material behaviors like collision are not in principle guaranteed. Dourish (2004) describes how collision can be an essential property for certain useful arrangements:
Tangible-computing designers have sought to create artifacts whose form leads users naturally to the functionality that they embody while steering them away from inconsistent uses by exploiting physical constraints. As a simple example, two objects cannot be in the same place at the same time, so a "mutual exclusion" constraint can be embodied directly in the mapping of data objects onto physical ones; or objects can be designed so that they fit together only in certain ways, making it impossible for users to connect them in ways that might make sense physically, but not computationally.
However, the greater space of possible physical behaviors offers opportunities to create new affordances with new interactional grammars that can take advantage of the specificity of computing power and the precise motion tracking of the body.
embodiment; homuncular flexibility
The body’s relationship to tools is often quite fluid, where prolonged use allows tools to be mentally fused with the body, and engagement with the world is perceived at the tool’s interface with the world rather than the body’s interface with the tool. Blind people can build a relationship with their cane such that “the cane is … incorporated into [their] body schema and is experienced as a transparent extension of [their] motor system” (Heersmink, 2014). The opportunities for spatial computing are even more potent here, where the medium’s capacities for tracking the body’s motion allows an even greater mapping between the rendered environment’s behavior and the user’s motion than ordinary dynamic media constrained to two-dimensional screens and rudimentary inputs.
The ability to depict the body in novel and hyperphysical ways, while still mapping the depicted body’s movement to the base movements of the user, enables startlingly compelling computer interfaces such as increasing the number of limbs,
Participants could hit more targets using an avatar with three upper limbs, which allowed greater reach with less physical movement. This was true even though motions mapped from the participants’ tracked movements were rendered in a different modality (rotation of the wrist moved the avatar’s third limb in arcs corresponding to pitch and yaw). Use of more intuitive mappings might enable even faster adaptation and greater success. (Won et al, 2015)
or changing the physical form of the hands to better interface with a task, as explored by Leithinger et al (2014): “…we can also morph into other tools that are optimal for the task, while controlled by the user. Examples include grippers, bowls, ramps, and claws — tools with specific properties that facilitate or constrain the interactions”. The question then becomes how many familiar aspects to include so as to conserve intuition, framed by Won et al (2015) as “…what affordances are required for people to use a novel body to effectively interact with the environment?”, especially when “such realism may reinforce the user’s desire to move as he or she would in the physical world.” Though, critically, the brain’s plasticity allows for novel environments to eventually become quite literally second-nature, as in the classic Heideggerian example of the hammer, articulated by Heersmink (2014): “When I first start using a hammer, my skills are underdeveloped and the hammer is not yet transparent. But gradually my hammer-using skills develop and the artifact becomes transparent which will then alter my stance towards the world.”
tools for thought
Ideally, the increased adoption and bodily engagement with hyperphysics will prove us with new tools to understand and represent not only the world around us at scales heretofore inaccessible (as Sutherland (1965) envisions about subatomic particles: “With such a display, a computer model of particles in an electric field could combine manual control of the position of a moving charge, replete with the sensation of forces on the charge, with visual presentation of the charge's position”, but also purer forms of knowledge such as mathematical relationships, and will lift our minds to new heights as previous notations for thought have already done. Gooding (2001) articulates it well:
Computer-based simulation methods may turn out to be a similar representational turning point for the sciences. An important point about these developments is that they are not merely ways of describing. Unlike sense-extending devices such as microscopes, telescopes or cosmic ray detectors, each enabled a new way of thinking about a particular domain.
The sciences frequently run up against the limitations of a way of representing aspects of the world — from material objects such as fundamental particles to abstract entities such as numbers or space and time. One of the most profound changes in our ability to describe aspects of experience has involved developing new conceptions of what it is possible to represent.
As the scale and complexity of problems experienced by humanity grows, it is critical to augment our problem-solving ability, a large part of which involves the creation of new forms of representation, ideally giving us a better grasp on the most fundamental questions. Gooding (2001), again, articulates it well:
But the environment is increasingly populated by artefacts which function as records and as guides for reasoning procedures that are too complex to conduct solely with internal or mental representations. In this way we are continually enhancing the capacity of our environment for creative thought, by adding new cognitive technologies.
These tools are still in their infancy, and only through an open exploration of the frontiers of their possibility-space will we find the most powerful means to augment our intellect.
references
Blom, K. J. (2007). On Affordances and Agency as Explanatory Factors of Presence. Extended Abstract Proceedings of the 2007 Peach Summer School. Peach.
Blom, K. J. (2010). Virtual Affordances: Pliable User expectations. PIVE 2010, 19.
Chemero, A. (2009). Radical Embodied Cognition.
Disessa, A. A. (1988). Knowledge in Pieces.
Dourish, P. (2004). Where the Action Is: The Foundations of Embodied Interaction. MIT press.
Gibson, J. J. (1979). The Ecological Approach to Visual Perception. Psychology Press.
Golonka, S., & Wilson, A. D. (2018). Ecological Representations. bioRxiv, 058925.
Gooding, D. C. (2001). Experiment as an Instrument of Innovation: Experience and Embodied Thought. In Cognitive Technology: Instruments of Mind (pp. 130-140). Springer, Berlin, Heidelberg.
Heersmink, J. R. (2014). The Varieties of Situated Cognitive Systems: Embodied Agents, Cognitive Artifacts, and Scientific Practice.
Leithinger, D., Follmer, S., Olwal, A., & Ishii, H. (2014, October). Physical Telepresence: Shape Capture and Display for Embodied, Computer-mediated Remote Collaboration. In Proceedings of the 27th Annual ACM Symposium on User interface Software and Technology (pp. 461-470). ACM.
Piaget, J., & Cook, M. (1952). The Origins of Intelligence in Children (Vol. 8, No. 5, p. 18). New York: International Universities Press.
Sutherland, I. E. (1965). The Ultimate Display. Multimedia: From Wagner to Virtual Reality, 506-508.
Won, A. S., Bailenson, J., Lee, J., & Lanier, J. (2015). Homuncular Flexibility in Virtual Reality. Journal of Computer-Mediated Communication, 20(3), 241-259.