# LLM Summary ## Chapter 1: A New Kind of Extension ### From Real to Virtually Real - [[Virtual reality]] (VR) is positioned on a continuum with physical reality, separated by mixed reality in the middle. It aims to induce perceptually convincing experiences in computer-generated environments. - Milgram et al. (1995) proposed a "reality-virtuality continuum" with the real environment at one end and virtual environment at the other, mediated by augmented reality and augmented virtuality. - VR environments are fully computer-generated and immersive, whereas mixed environments blend real and virtual elements. > "[[Virtual reality]] aims not at simulating reality but producing illusions" (Slater 2014, p. 2). - The term "[[virtual reality]]" has been used loosely, but key characteristics include multisensory stimulation, interactivity, and immersion. - Lanier popularized "[[virtual reality]]" in the 1980s to describe technology that could create interactive, immersive experiences. - Steuer (1992) identified vividness (sensory richness) and interactivity as the two main dimensions that determine telepresence. - Philosophical debates surrounding VR include: - Virtual fictionalism - virtual worlds are fictional, not truly real - Virtual realism - virtual objects and events have genuine existence on computers - Virtual worlds as "mental creations" vs "physical implementations" ### Extensions and Embodiment - Tools and technology, from language to VR, serve as extensions of human capabilities that have accelerated our evolution. This capacity for self-transformation through extensions makes us distinctively human. - Media theorist Marshall McLuhan recognized technology's potential to fundamentally change human nature as prosthetic extensions. - Anthropologist Edward T. Hall argued that humans and their extensions are interrelated into one system, mutually shaping each other. > "What makes us distinctively human is our capacity to continually restructure and rebuild our own mental circuitry, courtesy of an empowering web of culture, education, technology, and artifacts." (Clark 2003, p. 10) - The notion of the extended phenotype in biology (Dawkins 1982) suggests that an organism's genes can exert effects on the environment beyond the individual's body. - Tools and artifacts are likened to extended phenotypic effects - e.g. a beaver dam extending the beaver's "phenotype". - Culture can be seen as the extended phenotype of humans, continuing biological evolution by other means. - VR as an extension challenges our ordinary experience of unified body and self by allowing: - Presence - the feeling of being in the virtual environment instead of at one's physical location - Virtual embodiment - incorporating a virtual body (avatar) into one's body schema - Action at a distance - e.g. moving an avatar that's seen from a 3rd person perspective ### Presence and Breaks in Presence - Presence is a central concept in VR, defined as the subjective sense of being in a virtual environment. Alternative terms include telepresence, spatial presence, immersion. - It involves perceiving the virtual stimuli as if they were physically real, even while knowing at some level that [[the experience]] is mediated (Lombard and Ditton 1997). - Slater (2009) distinguishes "place illusion" (feeling present in the virtual space) and "plausibility illusion" (feeling that the virtual events are really occurring). > "When you are present your perceptual, vestibular, proprioceptive, and autonomic nervous systems are activated in a way similar to that of real life in similar situations. Even though you cognitively know that you are not in the real life situation, you will tend to behave as if you were, and have similar thoughts...." (Slater 2003, p. 2) - Breaks in presence (BIPs) occur when the user stops responding to virtual stimuli and instead attends to cues from the real environment. - Competing stimuli from the [[physical world]], if strong enough, can override virtual cues and "break" the illusion of being there. - Inconsistencies in the VR system itself (e.g. tracking errors, visual artifacts) can also disrupt presence. - Physiological monitoring of responses to BIPs is considered a more reliable measure of presence than post-hoc self-report. - Two main theories of how presence is formed and broken: - Presence as cognitive preoccupation - presence depends on focusing attention on virtual stimuli; BIPs result from shifting attention to real world cues (Wirth et al. 2007) - Presence as perceptual hypothesis - presence is [[the result]] of the perceptual system "accepting" the hypothesis that the virtual environment is the current space; BIPs occur when this hypothesis is rejected in favor of the real environment (Slater 2002) ### Cognitive Influences on Presence - Presence may be affected by top-down influences from higher cognition, in addition to bottom-up factors like sensory fidelity. This relates to the debate on cognitive penetrability of perception. - The traditional view holds that perception is modular and not influenced by knowledge, beliefs, goals (Pylyshyn 1999) - Alternative view argues that cognitive states can directly affect perceptual processing (Churchland 1988; Lupyan 2015) - In the context of VR, cognitive penetrability would mean that knowledge, expectations, and inference can alter low-level presence mechanisms. - Individual differences in cognitive processing may also modulate presence in VR: - Cognitive styles - e.g. visualizer vs verbalizer, field dependent vs independent (Witkin et al. 1977) - Personality traits - e.g. openness to experience, absorption, extraversion (Sas et al. 2004; Weibel et al. 2010) - Gaming experience and familiarity with VR technology (Gamito et al. 2010) > "Apparently, environments with impoverished spatial cues require a higher degree of participants' visuo-spatial imagery skills to construct stable spatial situational models." (p. 3) ### Physicality Illusions - Beyond just presence, VR can induce illusions of [[body ownership]], agency, and physicality more generally. These illusions blur the boundary between body and environment. - Rubber hand illusion - synchronized touching of real and fake hands causes misattribution of the fake hand to one's own body (Botvinick & Cohen 1998) - Enfacement illusion - seeing another face stroked in synchrony with one's own face leads to perceived facial similarity (Tsakiris 2008) - Full [[body ownership]] illusions - 1st person perspective of a virtual/fake body combined with visuotactile correlations (Petkova & Ehrsson 2008) > "However, the neural correlates of virtual tool use in humans are still not clear and it is also unclear whether similar multisensory mechanisms support the use of physical and virtual tools" (p. 39) - These illusions reveal the flexibility of body representations and [[the role]] of multisensory integration: - Body schema can incorporate external tools/objects through synchronous sensorimotor correlations - Ownership illusions do not require a realistic appearance, only 1st person perspective and intermodal matching - Phantom limb phenomena in amputees show how body image can persist without the physical limb ## Chapter 2: Self in [[Virtual Reality]] ### [[The Puzzle]] of Having a Self - [[The nature]] of the self has long been a puzzle, with some questioning whether a coherent, unified self exists at all. - Hume argued that there is no single, permanent self, only a bundle of sensations - Buddhist concept of anatta (no-self) denies the existence of an unchanging soul or essence - Modern philosophers like Metzinger (2004) claim the self is a complex illusion generated by the brain > "What we often, naively, call 'the self' in folk-psychological contexts is the phenomenal self, the content of self- consciousness, given in phenomenal experience" (Metzinger 2003, p. 303). - Common aspects of conscious [[self-experience]] include: - Bodily self-awareness (sense of ownership, location, agency over one's body) - Autobiographical memory and personal narrative (life story, personality, social roles) - Cognitive self-representation (self-concept, self-esteem, future plans) - The "minimal" or "core" self refers to the basic, immediate sense of being a distinct subject of experience, separate from the environment. It is pre-reflective and prior to language or conceptual thought. - Anchored in bodily sensations, interoception, proprioception, action monitoring - Develops early in infancy through interactions with caregivers and objects ### Disjoint Self Perspectives - There are many divergent philosophical and psychological theories of the self, each emphasizing different aspects or levels of self-awareness: - Physical, mental, and spiritual selves - Ecological, interpersonal, extended, private, conceptual selves (Neisser 1988) - Proto, core, and autobiographical selves (Damasio 1999) - Material, social, and spiritual selves (James 1890) - Cognitive neuroscience studies also point to anatomically and functionally distinct self-related processes in the brain: - Cortical midline structures (medial prefrontal, precuneus) - self-referential thought, autobiographical memory - Insula and anterior cingulate - interoceptive awareness, embodied sense of self - Temporoparietal junction - self-location, 1st person perspective, self-other distinction > "The self is a kind of emulation, constructed by the brain, for integrating and making sense of the inner world of the brain in [[its relation]] to the external world, including the other-person-world." (Churchland 2011, p. 48) - VR experiences can lead to a fragmented or "disjoint" self, challenging the assumption of a single unified self-perspective: - Seeing one's avatar from a 3rd person view - creates a "double" or alternate self - Acting in VR while one's physical body is stationary - dissociates sense of agency from body - Experiencing presence and immersion oscillating - shifting between the real and virtual "selves" ### Phenomenology of Disembodiment - Ordinary self-experience is characterized by an implicit sense of "ownership" over one's thoughts, feelings, and especially one's body. - This is supported by a pre-reflective bodily self-awareness (proprioception, kinesthesia, interoception) - Typically there is a stable background sense of being an embodied subject located in space - In VR this unified self-body-space relationship can become disrupted or "disowned": - Lack of tactile/haptic feedback or sensorimotor contingencies compared to physical interactions - Mismatch between seen movements of avatar and felt movements of physical body - Altered or impossible forms of virtual embodiment (floating, passing through objects, body swapping) > "Agency and ownership are thought to be key aspects of the 'minimal phenomenal self', the basic feeling of being a distinct, embodied entity" (p. 36). - These disruptions bear similarities to neurological disorders of bodily awareness: - Somatoparaphrenia - loss of ownership over a paralyzed limb, often attributing it to someone else - Depersonalization - persistent feeling of being detached from one's body and mental processes - Out-of-body experiences - seeing one's body from an external perspective, feeling dissociated from it - VR may allow controlled induction of these altered states to study their neural bases: - Manipulating visual perspective, sensorimotor synchrony, avatar appearance to modulate body ownership and self-identification - Combining VR with neuroimaging and physiological recording to probe the mechanisms of bodily self-consciousness ### The Flexible Body Schema - The body schema is a dynamic, sensorimotor representation of the body's size, shape, and position in space, updated by multimodal inputs. - Guides reaching, grasping, postural control and other motor actions without explicit awareness - Can incorporate external objects (tools, clothing, vehicles) into its network of body-environment relations > "The inclusion of tools and other noncorporeal objects in the body schema works in such a way that it feels 'as if our own effector (e.g. the hand) were elongated to the tip of the tool'" (Maravita & Iriki 2004, p. 79) - VR experiments reveal the plasticity of the body schema in adapting to novel virtual bodies and environments: - Rubber hand and virtual arm illusions show rapid extension of body schema by synchronous visual-tactile stimulation - "Envelopment" - avatar body becoming transparent or passing through solid walls - Altered size/shape of limbs - elongated arms, giant hands, tails etc. still integrated into body schema if sensorimotor contingencies preserved - Body schema differs from body image, the conscious, attitudes, and emotions related to one's body: - Body image has a weaker effect on virtual embodiment than 1st person perspective and multisensory correlations - Perceived loss of a virtually owned body part (e.g. virtual amputation) can induce strong emotional reactions - Dissociations between body image and body schema demonstrated in eating disorders, body dysmorphia ### Neural Basis of Self-Representation - Neuroimaging studies point to multiple interacting networks underlying different aspects of self-processing: - Default mode network (DMN) - involved in internally-directed, self-referential cognition, mind-wandering, autobiographical memory - Medial prefrontal cortex, posterior cingulate/precuneus, inferior parietal lobule, lateral temporal cortex - Resting-state activity, deactivated during externally focused tasks - Mirror neuron system - activated both when performing an action and observing another perform the same action - Implicated in action understanding, empathy, theory of mind and social cognition more generally - Frontoparietal network including premotor cortex, inferior frontal gyrus, anterior intraparietal sulcus > "Two large-scale neural networks have been implicated in our understanding of other minds: the mirror neuron system, which comprises the parietal and premotor regions, and the network of areas known as the social brain, which includes the medial prefrontal, the temporopolar, the temporoparietal cortices and the amygdala" (Wilms et al. 2010) - Self-specific processing also involves: - Anterior insula - interoceptive awareness, embodied self, emotional self-consciousness - Temporoparietal junction (TPJ) - self-location in space, 1st person perspective, mentalizing - Posterior superior temporal sulcus (pSTS) - biological motion, gaze processing, social attention - These networks continuously interact to generate the self as an integrated mind-body system: - Narrative self linked to autobiographical memory and default mode activity - Minimal/embodied self anchored by multisensory integration in body-centered networks - Social self dependent on frontoparietal mirroring mechanisms and pSTS - Implications of neuroscience frameworks for VR: - Inducing and modulating self-related processes with avatar embodiment, perspective taking, social interactions - Dissociating levels of self by activating body ownership without agency, self-location without self-identification - Linking phenomenology to brain dynamics with neuroimaging, stimulation and recording methods ## Chapter 3: Self and the Virtual Other ### Social Cognition and Intersubjectivity - Social cognition refers to the perception, interpretation and processing of information about other people and social interactions. It enables us to understand and predict others' beliefs, feelings, intentions - "mindreading". - Spans multiple levels from automatic, implicit processes to controlled, refl Here is the continuation of the detailed summary: ### Social Cognition and Intersubjectivity (continued) > "Social cognition includes processes that support our understanding of others and our interaction with them" (Gallotti and Frith 2013). - Key processes in social cognition: - Face and emotion recognition - detecting identity, age, gender, emotional states from facial features - Gaze following and joint attention - sharing attention on an object with others, monitoring their attention - Biological motion perception - recognizing actions, intentions, interactions from movement patterns alone - Action understanding and imitation - linking observed behaviors to own motor repertoire, learning novel skills - Empathy and affect sharing - vicariously feeling what others feel, emotional contagion - Mentalizing/theory of mind - attributing mental states to others to explain and predict their actions - Development of social cognition in infancy: - Preference for facelike stimuli and biological motion from birth - Gaze following and joint attention emerge around 6 months - Imitation of novel actions and understanding of false beliefs by 18 months - Pretend play, deception, and reasoning about others' thoughts by 4-5 years - Two contrasting theories of the cognitive basis of mentalizing: - Theory theory - mental states are inferred using a naive theory of mind, like a scientific theory - Simulation theory - mental states are attributed by imagining oneself in the other's situation > "Mirror neurons are the neural basis of an action observation/execution matching system" (Gallese 2000a) - Discovery of mirror neurons led to a hybrid theory - embodied simulation: - We understand others' actions and emotions by activating our own neural representations of those states - A direct, prereflective mechanism not requiring propositional knowledge or theory-like inference - Supports a fundamental intersubjectivity, an immediate grasp of others' experiences - Disorders of social cognition as a window into its mechanisms: - Autism - deficits in joint attention, imitation, theory of mind, possibly due to a faulty mirror system - Schizophrenia - impaired emotion recognition, overactive mentalizing, confusion of self and other - Psychopathy - lack of empathy and concern for others, but intact or even superior mindreading ### The Social Brain - Social cognition, like other cognitive faculties, is supported by specialized neural systems that process social information. Key regions and networks: - Fusiform gyrus - contains face-selective areas (FFA) that respond to face identity and features - Superior temporal sulcus - processes biological motion, goal-directed actions, eye gaze - Amygdala - detects emotional significance of faces and social stimuli, linked to fear processing - Orbitofrontal cortex - integrates social cues to generate appropriate emotional responses and decisions - Anterior cingulate and anterior insula - self-awareness, empathy, emotional contagion - Medial prefrontal cortex and temporoparietal junction - mentalizing about beliefs, intentions > "Two key networks are consistently implicated in reasoning about others' mental states - the mirror neuron system and the mentalizing system" (Koster-Hale & Saxe 2013) - Different theories propose different divisions of labor and interactions between these regions: - Mirror neuron system supports low-level embodied resonance, mentalizing system handles explicit, symbolic mental state attribution - Mentalizing recruits mirror areas when reasoning about actions/emotions, and other areas for abstract beliefs/knowledge - Amygdala and orbitofrontal cortex generate automatic social-affective responses, while prefrontal areas exert top-down control - The social brain hypothesis proposes that: - Humans have uniquely complex social lives, requiring sophisticated social cognition - These social demands drove the rapid expansion of brain size, especially frontal and temporal areas - Specific adaptations like theory of mind, shared attention, imitation learning provided key fitness advantages > "Humans are 'ultra-social' - our brains are specialized for rapidly learning, remembering and predicting complex social information" (Herrmann et al. 2007) - Comparative studies reveal human-specific social cognitive abilities not shared by other great apes: - Teaching and social learning - attending to others' knowledge states to transmit information - Collaborative problem-solving - coordinating actions and goals to achieve shared intentions - Social norms and institutions - representing collective agreements about behavior, punishing violations - Cumulative cultural evolution - building on and transmitting others' discoveries across generations ### Virtual Selves and Others - VR provides a powerful tool to study social cognition by creating controlled, realistic, interactive social simulations. Key applications and findings: - Dyadic interactions with virtual agents - testing the factors that influence rapport, trust, persuasion - Contingent gaze, mimicry and back channeling from an avatar increases social influence (Bailenson et al. 2004) - Manipulating an avatar's attractiveness and behavior changes interpersonal distance (Yee & Bailenson 2007) - Social skills training - practicing job interviews, public speaking, assertiveness with virtual audiences - Anxious participants show reduced fear of public speaking after rehearsing with a virtual crowd (Slater et al. 1999) - Exposure therapy for social phobia using virtual conversations and audiences (Powers & Emmelkamp 2008) - Inducing and measuring social psychological effects with avatar appearance and behavior - Proteus effect - conforming to expectations based on one's avatar's looks, e.g. attractive avatars act more confidently (Yee & Bailenson 2009) - Virtual body swapping - reducing racial bias by embodying an avatar of a different race (Peck et al. 2013) > "Putting people in avatars of a different race, age or gender can lead to immediate changes in implicit bias and stereotyping" (Bailenson 2018) - At the same time, social interactions in VR face limitations and challenges: - Lack of photorealism, facial expression and nonverbal cues compared to in-person - Mismatch between appearance, behavior and identity of avatars piloted by real humans - Heightened uncertainty about whether an agent is an avatar or artificial intelligence - Glitches and inconsistencies reminding users that the virtual social world is not real - The future of VR as a social medium will depend on: - Improving the expressive realism and behavioral fidelity of virtual humans - Optimally integrating human and AI control to create meaningful interactions - Understanding the similarities and differences between virtual and physical world social cognition - Anticipating the psychological effects and ethical implications of socializing through artificial selves ## Chapter 4: Virtual Embodiment and Action ### Distinguishing Self and Other - A key function of the motor system is to differentiate between self-generated actions and external events, including the actions of other agents. - Efference copy - an internal copy of a movement command used to predict its sensory consequences - Forward models - neural mechanisms that map intended actions to expected outcomes for comparison to actual feedback - When predicted and actual signals match, the action is attributed to self; otherwise it may be attributed to an external cause > "Under normal circumstances, the human motor system efficiently processes the distinction between self-created actions and actions created by others" (Howard et al. 2016, p. 88). - This self-other distinction can be impaired in VR due to: - Altered or absent proprioceptive feedback of one's body and movements - Inconsistency between seen and felt limb positions and action timings - 3rd person avatar perspective decoupling visual and motor representations - Identity cues on avatar suggesting another agent (e.g. opposite gender, race) - Experimental evidence for impaired agency and ownership in VR: - Delayed visual feedback of hand movements makes them feel less voluntary and controlled (Franck et al. 2001) - Watching an avatar copy one's movements induces illusory ownership over the avatar's body (Sanchez-Vives et al. 2010) - Controlling a virtual arm without haptic feedback reduces neural activation in insula and anterior cingulate (Liang et al. 2015) - Dissociations of agency (feeling of initiating and controlling actions) and ownership (feeling that body/limbs belong to self): - Out-of-body illusions - experience ownership of a surrogate body without agency over its actions - Anarchic hand syndrome - experience agency without ownership, as if hand has a mind of its own - Xenomelia (body integrity identity disorder) - disownership of a functional limb, urge to amputate it ### Action Possibilities and Affordances - According to ecological psychology (Gibson 1979), we perceive objects and environments in terms of the potential actions they afford, based on our bodily capabilities. - Affordances are objective action possibilities in the environment relative to an agent's body and skills - E.g. a chair affords sitting for an adult human, but climbing or hiding for a child - Affordances are detected automatically, without internal representations or inferences about the object's identity or qualities > "Behavior affords behavior, and the whole subject matter of psychology and of the social sciences can be thought of as an elaboration of this basic fact" (Gibson 1986, p. 135). - The neuroscience of affordances reveals multiple interacting brain systems: - Canonical neurons in premotor cortex - respond to visual features of graspable objects - Parietal-frontal circuits - extract action-relevant properties like shape, size, orientation, location - Dorsal visual stream - computes sensorimotor transformations for reaching, grasping, manipulation > "Two parietofrontal circuits - the dorso-dorsal stream for online control of action and the ventro-dorsal stream for action planning and representation" (Binkofski & Buxbaum 2013) - VR can modulate action affordances in novel ways: - Altered avatar bodies with non-human morphologies, e.g. elongated arms (Kilteni et al. 2012), extra limbs (Steptoe et al. 2013), tails (Won et al. 2015) - Reduced degrees of freedom and haptic feedback constraining physical action possibilities - Superpowers and physics violations - flying, teleporting, passing through objects, invulnerability to damage > "Thus the match between visual and proprioceptive information about limb position and movement is a powerful driver of the sense of ownership" (Blanke et al. 2015, p. 556) - Designing compelling affordances in VR requires: - Sensorimotor contingencies - visual-motor synchrony, realistic collision dynamics and haptic cues - Audiovisual realism - high-fidelity 3D graphics and spatial sound increasing perceptual believability - Coherent avatar mapping and embodiment - aligning virtual and physical body schemas and action capabilities - Tangible interaction options - grasping, touching, wielding objects with naturalistic resistance and texture ### Vision for Action - The visual system has two distinct but interacting pathways (Goodale & Milner 1992): - Ventral "perception" stream - object recognition, scene understanding, visual awareness - Dorsal "action" stream - real-time visuomotor transformations for reaching, grasping, navigation > "The crucial distinction between the two streams is that the ventral stream transforms visual information into perceptual representations that embody the enduring characteristics of objects and their relations whereas the dorsal stream deals with moment-to-moment information about the location and disposition of objects with respect to the effector being used" (Milner & Goodale 2008, p. 775). - Vision for action has specific properties dissociable from vision for perception: - Egocentric reference frame - represents objects relative to observer/effector, not objective scene structure - Real-time, automatic processing - no memory buffer, continuously updated control signals to muscles - No conscious access - can guide actions without perceptual awareness or recognition of target - Resistance to pictorial illusions - visual size contrast effects do not fool grasping - Evidence from neuropsychology for separate visual pathways: - Visual form agnosia - impaired object recognition with spared visually guided grasping and locomotion - Optic ataxia - impaired reaching and grasping with spared object recognition - Action blindsight - preserved fast pointing to blind visual field stimuli > "The visual processes mediating speeded target-directed behavior are rapid, automatic, and independent of the perceptual interpretation of the target" (Neely et al. 2008, p. 59) - Implications for virtual reality: - Ego perspective and valid stereoscopic cues are essential for accurate visually guided action - Graphics latency, jitter and poor registration impair reaching, grasping and object manipulation - Mismatched haptic and visual feedback (e.g. seeing hand pass through objects) breaks action immersion - Avatar movement should be intuitive and responsive, minimizing perceived control latency ### Virtual Sense of Bodily Self - The experience of a coherent, embodied self depends on multisensory integration of: - Exteroceptive signals - vision, touch, sound, smell, taste - Interoceptive signals - proprioception, kinesthesia, vestibular sensations, pain, hunger, arousal - Efferent signals - motor commands, effort, intentions to act > "The bodily self is a complex construct comprising several simpler components including the senses of ownership, agency, and self-location" (Blanke & Metzinger 2009) - These intermodal bodily cues form an unconscious, dynamic representation - the body schema or body matrix: - Manages the configuration and metrics of body parts in space - Incorporates muscles, joints, tendons, skin, and neural maps - Support motor control, object interaction, defense of personal space - VR can induce illusions of embodiment by exploiting the same multisensory mechanisms: - Rubber hand and virtual arm illusions - synchronous visual and tactile stimulation transfers ownership - Full body ownership - 1st person perspective and visuotactile synchrony yields identification with virtual avatar - Enfacement - seeing one's face morphed with another's face touched in synchrony blurs identity boundaries - Key factors determining the strength and stability of virtual embodiment: - 1st person colocation of real and virtual body - seeing from the avatar's visual perspective - Visuotactile synchrony - temporally matched visual and haptic events on the body - Visuomotor synchrony - close temporal coupling of tracked real and rendered avatar movements - Realistic body continuity and connectivity - virtual body appears attached to 1st person viewpoint - Volumetric body appearance - 3D modeled avatar rather than a flat image or abstract icon - Naturalistic avatar animation - smooth, biologically plausible body shape and movements > "For the illusion to succeed, there needs to be a sense of body continuity, i.e., visual experience that the virtual hand belongs to the body" (p. 25). - The plasticity of body representation allows virtual bodies that deviate from the physical body: - Altered size, shape and appearance - childlike body, elongated limbs, different skin color - Supernumerary limbs - controlling a third arm or a tail with fluent sensorimotor mapping - Cross-modal perceptual extension - "seeing" tactile stimuli on body through visual flashes alone - Functional substitution - using motor actions to "feel" virtual textures or control a visual menu ## Chapter 5: Spatial Cognition in Virtual Reality ### Body and the Space Around It - We perceive space not as an abstract void, but through our situated, embodied interactions with the environment. Key psychological qualities of space: - Extrapersonal (far) vs peripersonal (near) - can be reached and manipulated from current position - Egocentric (self-relative) vs allocentric (world-relative) - defined by observer's perspective or independent spatial relations - Euclidean (metric) vs topological - quantified by distances and angles or qualitative relations like containment and proximity - Isotropic (directionless) vs anisotropic (directional) - equal in all directions or having privileged orientations like verticality > "Nearness is not a purely spatial relation - it always has functional significance. What is near is what is accessible with the body" (Gallagher 1995) - Neuroscientific studies reveal multiple parallel spatial representations in the brain: - Posterior parietal cortex - egocentrically coded space for eye, head and arm movements - Medial temporal lobe (hippocampus and parahippocampus) - allocentric mapping of environment, places and routes - Retrosplenial cortex and precuneus - translation between egocentric and allocentric reference frames > "The parietal lobes appear to be the main neural substrate for egocentric spatial cognition, while the hippocampus and medial temporal lobe support allocentric spatial memory" ### Sense of Place in Virtual Environments - A sense of place is the holistic, meaningful experience of being in a specific environment, comprising (Turner et al. 2003): - The physical features and sensory qualities of the space - The actions and activities supported by the affordances - The personal and social meanings, memories and emotions evoked > "Space not only communicates in the most basic sense, but it also organizes virtually everything in life" (Hall 1990, p. viii). - VR experiences often lack a strong sense of place due to: - Low visual realism and unnatural appearance of objects and scenes - Absence of rich multisensory stimuli - sound, smell, air temperature, wind - Limited interactivity and constrained user actions compared to real places - Difficulty evoking memories, meanings and emotional associations - Factors enhancing sense of place in VR: - Vivid, naturalistic visual and auditory details in the environment - Congruent ambient sensory cues - e.g. sound of wind, warm temperature in a desert scene - Narrative backstory and meaningful context justifying the user's presence - Interactive objects that respond coherently to physical actions - Personally significant content related to background knowledge and interests - Measuring the phenomenology of place in VR with questionnaires and interviews: - Feelings of immersion, involvement and being there rather than in the lab - Subjective realism, coherence and meaningfulness of the virtual place - Emotional engagement and reactions to the people, objects and events - Memories and associations triggered by the environment, sense of familiarity ### Spatial Frames of Reference - Spatial reference frames are coordinate systems used to represent locations and orientations of objects relative to an origin and a set of axes. Multiple types: - Egocentric - self-centered, defined by the observer's eye, head or body - Retinocentric - positions relative to center of gaze - Head-centered - positions relative to interaural axis and facial midline - Trunk-centered - positions relative to torso/shoulder midline - Allocentric - world-centered, defined by features of the external environment - Object-centered - positions relative to reference objects, like a desk or building - Environment-centered - positions in a fixed coordinate system, like a room or map - Peri-personal - near the body, within arm's reach; extra-personal - outside of reach > "To fixate and then reach towards a goal object, it is necessary that the location and motion of that object be specified in egocentric coordinates (that is, coded with respect to the observer)" (Milner & Goodale 1995, p. 41) - Posterior parietal cortex contains multiple egocentric spatial maps: - Lateral intraparietal area (LIP) - eye-centered coding for saccades and attention - Ventral intraparietal area (VIP) - head-centered coding for near-head space - Medial intraparietal area (MIP) - arm/hand-centered coding for reaching - Each map is dynamically updated as eyes, head and body move to maintain stable coordinates - Hippocampus and adjacent regions support allocentric spatial mapping: - Place cells - fire when the animal is at a particular location in the environment - Grid cells - fire in a periodic hexagonal lattice covering the environment - Border/boundary cells - fire in response to edges and barriers in the environment - Head direction cells - indicate the facing angle in the environment using vestibular cues > "Place cells in the hippocampus encode an animal's location in the surrounding environment, while grid cells in the entorhinal cortex provide a metric for distance traveled" (Moser et al. 2008) - Spatial reference frame transformations and coordination: - Posterior parietal cortex combines visual, tactile and postural signals into common egocentric coordinates - Retrosplenial cortex translates between egocentric (parietal) and allocentric (hippocampal) frames - Prefrontal cortex selects and maintains the relevant reference frame for the current task context ### Spatial Cognition in Virtual Reality - VR enables novel manipulations of spatial perception and frames of reference: - Altered scaling of near vs far space - objects can be grabbed at a distance or made to appear closer/farther - Remapping of real to virtual movement axes - walking forward physically moves sideways in VR - Rotated or inverted frames of reference - world turns upside down when head pitched beyond vertical - Warped or non-Euclidean spatial geometries - parallel lines diverge, shortest path is a curve - Dissociation of visual and motor reference frames - 3rd person avatar control from overhead view > "Various perceptual rearrangements can be achieved with virtual reality that are impossible in the physical world" (Klatzky et al. 1998) - Spatial cognitive processes partly transfer from real to virtual environments: - Accurate judgment of egocentric distances to targets less than a few meters away - Systematic underestimation of distances and depths at greater range (Willemsen & Gooch 2002) - Path integration and spatial updating impaired by reduced visual motion and vestibular cues - Switch costs in perspective taking and mental rotation comparable to real world - VR advantages for research on spatial cognition: - Precise control and real-time modification of spatial parameters and cues - Visual isolation from ambient real world while preserving natural input-output coupling - Safe, ethically acceptable simulation of situations impossible or impractical in reality - Neural and physiological recording synchronized with 3D position/orientation tracking - Automated logging of spatial trajectories and interactions, unobtrusive to the user - Applications of VR in spatial cognition: - Diagnosing navigation impairments in aging and neurodegenerative disease (Cushman et al. 2008) - Training spatial skills of surgeons, pilots, architects and other professionals (Kozhevnikov et al. 2013) - Studying the neural basis of spatial memory and hippocampal function (Doeller et al. 2012) - Developing novel spatial user interfaces and interaction techniques (Bowman & McMahan 2007) ### Spatial Presence and Body Memory - Spatial presence is the feeling of physically being in the virtual environment rather than the real world. It depends on: - Immersion - the objective fidelity and extent of sensory information and interactions supported by the VR system - Place illusion - the subjective feeling that the virtual environment is the primary reference space rather than the physical environment - Plausibility - the coherence and credibility of the virtual environment's behavior with respect to user expectations > "Spatial presence occurs when a person perceives the virtual environment as more engaging than the physical surroundings, and this feeling is not broken by incongruent cues or technical limits" (Wirth et al. 2007) - The body schema - an unconscious, dynamic representation of the body's spatial properties - is critical for spatial presence: - Incorporates the avatar body into the proprioceptive and tactile body image - Allows object interactions and impacts to be intuitively mapped onto bodily space - Integrates the real and virtual bodies into a single, transparent locus of sensation and action - Breaks in presence occur when salient cues from the physical environment capture attention and reference the real body, such as: - Haptic cues - touching real objects or experiencing force feedback inconsistent with visual motion - Postural instability - losing balance or colliding with physical obstacles not visible in VR - Interoceptive sensations - noticing breathing, heartbeat, hunger or discomfort anchored in the real body - Equipment cues - feeling the weight of the headset or the cable brushing against skin > "Presence breaks when attention disengages from the virtual environment by salient external cues to the real environment" (Slater et al. 2003) - Spatial memory and navigation also shape spatial presence in VR: - Stronger recall of object locations and spatial layout increases sense of being there (Mania & Chalmers 2001) - Lack of proprioceptive and vestibular feedback impairs path integration and spatial updating (Ruddle et al. 2011) - Large, complex virtual spaces without clear boundaries and landmarks reduce spatial orientation - The body-environment link in spatial memory: - The hippocampus is a hub connecting external (allocentric) and internal (egocentric) space - It receives inputs about the body's motion and location relative to the environment - Damage leads to both spatial disorientation and distorted sense of embodiment (out-of-body experiences, autoscopy) > "Representation of place in the hippocampus is actually a representation of the location of the animal's body in space" (McNaughton et al. 2006, p. 52)