The Fluid Gaze in Virtual Reality

In 2006, in the course of an Artists Residency in Munich, I made a video triptych titled Meditations on the Tiger, in which a story unfolds over three adjacent screens. It was a simple story, of a woman who steps out from home and takes a train journey to a lake, reaching which she is ferried further across to an island. Returning relaxed and refreshed, she prepares for action.

The story is as linear as it can get, but working with three screens I found I could move laterally as well. So while the woman is still seated in the train on one screen, she has already reached her destination on another, and is somewhere else altogether in a third. There were multiple tracks of time running together on that train: the real time of action and event, the hurtling projected time of anticipation and expectation, and the deep reflective time of memory, thought and speech.  Three video timelines synchronized, so we begin to approach image, just as we have learnt to approach sound, with Stereo and Dolby, as highly differentiated. A close up and a long shot, detail and vista  – differences in magnitude or quantitative differentiation – could be shown at the same time; as also stillness and motion –  qualitative differences, opposites even – reconciled  and balanced.  Space, time and mode had all been rendered fluid.

Of course, I knew none of this when I began. The residency was only for two months and the preview was already fixed for the last week. So I had to jump in feet first. What I did carry in with me though, was the ancient Indian notion of the third eye, the eye with which the past, present and future are all viewed with the same sense of  equanimity. This idea  I used  –  as inspiration not application –   to compose and distribute the action variously across the three screens, though never in a fixed way. And when I braided the three timelines together I came to realise that this third eye is not an eye at all, but the mind! The triptych made tangible for me what until then was only an abstract idea.   

Something else I realized, also in retrospect, was that the rectangular frame, inherited from painting and traveling into cinema via the still photograph, had to take wings to move beyond itself. Two screens would have been just arithmetic – division or multiplication, however we choose to see it – leading simply to endless proliferation of self-replicating and/or opposing  binaries. The triad on the other hand, was a prime – and a singularity. A level shift and new order of being. For with three screens, one is not simply extending the viewing arena in one horizontal dimension, one is expanding it across dimensions to create volume. A moving space-time column that became vector not scalar spatially, and straddling real and virtual time besides, became 3^3.  And thus I entered, unbeknownst, the world of exponents and algorithm, from which it is but a single step to Virtual Reality.

This too I realized the hard way. For the longest time after the triptych, all the way from 2007 to about 2015, I kept trying to make a two-screen work, but never felt satisfied. Then I thought I was moving in the wrong direction, and should try four screens instead. But that seemed a very mechanical way of thinking. At some point then I remember noticing how sound had developed since its shift (even before the image) from analog to digital. How audio tracks which had first divided into two with Stereo then leapfrogged into 5.1  / 7.1  with Surround Sound, and thence to the infinitely embedded tracks of Dolby Atmos. In my understanding, that is what we are now doing with image too. Moving from the stereo image – which is the basis of simple 3D –  into the 360° surround image, the frame collapsing entirely to give way to the field. This to me is quintessentially digital thinking.

Why is it so important to trace this route? Is it for personal reasons? Yes. Where one arrives from will determine what one will find, so to speak. But it is also important in a larger sense to see Virtual Reality as both continuity and rupture. Technological ruptures are often embedded in underlying conceptual continuities, which is not to say that there is never anything new. To the Indian mind, this is like waves in the sea. The water itself is in an endless state of motion and transformation – rising and falling, approaching and withdrawing, evaporating and condensing, vanishing and appearing. Precisely because the shift – from analog/sequential to digital/exponential – is paradigmatic, the underlying continuities which may be nothing more than frail thought patterns, effusions of the mind, become even more precious. Like the thread of Ariadne as we enter the labyrinth, the pebble trail of Hansel and Gretel as we enter the unknown.

To go further back into the collective known, I wonder if cave paintings weren’t already immersive. One can imagine the sound of the wind and the waves echoing in the cave chamber to accompany both the creation and reception of this rudimentary form of image-making. The flickering flames of a firestick throwing both light and shadow, veiling and revealing the ephemeral image to create a sense of awe and magic. With chanting, song and dance perhaps, to heighten the sense of communitas.

Temples and churches continue to be pre-eminently immersive spaces, as was the street corner and fairground bio-scope and kaleidoscope where you looked into a box shrouded in black cloth. Even the cinema hall is immersive, and the moving camera and elaborate mise-en-scene have been trying since inception to vault us out of the two dimensional. Which brings me to the idea – is it perhaps the rectangular frame that is the anamoly?

Among my many reservations with the static frame is the fact that it creates an art object, out there in space for the viewer to gaze upon, crave for, fetishize, and seek to possess. The world imaged and imagined thus, not as inhabited but external space, fostered fantasies of world dominion and colonialism. The vanishing point on which perspectival depth was predicated, the false infinity at which receding  parallels seemed to converge, became the graphic node of patriarchal invasion and rape, colonizing the female body.

VR like 3D and regular cinema too before it, begins from this same invasive impulse, more frighteningly, from the foreground. In the very first shot that the Lumiere Brothers projected in Paris, the audience reportedly ran out of the hall as the train entering the platform seemed to come leaping out of the screen into their very midst. Similarly, on the welcome deck of the Oculus Rift (interesting maritime imagery there) we duck instinctively from the approaching dinosaur, and then (in the lower decks) from various other monsters, aliens, ghosts and spirits that either appear in our very faces or loom up from directly behind us. But these are clearly shadow figures, not legitimized and romanticized conquistadores; the stuff of horror movies catering to a particular genre predeliction, neither about heroes nor the norm.

To me they feel like vestigial remains of an atavistic imagination that we can now consciously purge. Sitting in broad daylight within the laser-drawn circle of vision described by the VR sensor, one of the first survival strategies one acquires is the reflex to duck, evade, elude and variously avoid that which one does not wish to participate in. There is no collective or peer pressure to do so. One does not even have to fight over the remote with other members of the family. It is all ours. So when horror masquerades as reason, with a title that reads ‘Face your Fears’, one can laugh in its face, as it begins with first of all inducing a fear – not your own – which it then proceeds/pretends to help you face. For our fears are not any less personal than our dreams, if not they are not really ours. Facing a standardized set of fears creates merely a false sense of bravura, not real courage, and VR is possibly one of the best ways of realizing this. To instill psychological immunity may even be its raison d’etre.

‘Testimony’, a work that premiered at the 2017 Tribeca Film Festival, is part of an ongoing project for survivors of sexual abuse, and is also co-produced by a survivor. So everything about it rings true and is deeply moving. The black screen has never felt so intimate or sacred as we listen to the first person accounts. The stark 360 space, with no props whatsoever, reminds one of a circus tent, perhaps because of the loose ropes that flow through it, linking the figures floating like trapeze artists in midair. It is a telling image, evoking the lightness and emotional dexterity with which these individuals hold their own above the yawning abyss of their past. Only when our gaze actually centres on someone do they move forward and begin to speak, as if a certain quality of attention is a pre-requisite for engagement. So our gaze moves, circling the space again and again to seek out the faces and bear witness to their accounts of abuse and slow healing over the years. They speak with the directness of lived experience but touched by reflection, looking as much within themselves as at us, sometimes breaking down, sometimes angry, and sometimes with humour or levity. A deeply immersive work that makes full and intelligent use of this amazing new medium.

There is a moment in VR, when you have donned the headgear, but the program has not yet begun and you are all alone in the silent cocoon of darkness. This has often been for me a profound moment – as I center and ground myself in my body, feel the reassuring ground beneath my feet and my hands and clutch the remote for some semblance of control before lending myself to the experience. There is an almost primeval privacy here, between universes, between the real and virtual. If you take a peek sideways you can see your sleeve barely ending before the image field begins. It is not just beside and behind you, but also above and below. In fact, when you set up the equipment, you are asked to enter your height, so the virtual ground can be placed realistically. But if you then sit down and watch, the virtual ground begins to yawn way beneath your feet so you really are suspended in this world. And somehow in this anti-gravitational ‘lightness of being’ you are able to empty yourself and allow the virtual world to stream into you as well. As if you were mere outline, and then not even that.

Another work that calls upon a floating perspective very well is Dear Angelica by the Emmy award winning Oculus Story Studio. The software Quill was apparently created specifically to make this work possible, allowing the illustrator Wesley Allsbrook to draw/paint directly into the 360 ° space. So there is a live, kinetic feel to the piece as we watch not just a new story but also a new aesethetic unfurl before our eyes. The storyline is as slender as the line drawings themselves, as it sketches the life of a young girl writing a letter to her actress-mother who is no more, while watching her old movies on a VCR. Fleeting personal memories segue seamlessly into scenes from her mother’s films, and the space which is always only suggestive dissolves from real, remembered and fictive space to finally become purely imaginative as she joins her mother in the heavens. All space is virtual, materially by being created and erased successively by the illustrating hand, and figuratively by becoming affective space. Memory, desire, longing and reconciliation mingle in the swirling whirlpool of time-space enveloping her like a bubble; a bubble within which we are ensconced as well, becoming part of her mental and emotional landscape. Sometimes the flowing lines themselves, sometimes sound – words, music, effects – a horse neighing, a car accelerating, gunshots – and sometimes sheer silence directs our gaze, as we swivel in our seats, eager not to miss a single detail, savour every slightest nuance.  

Above: The author in a virtual experience.

VR really does seem to me the next, possibly irreversible step from traditional cinema, which already among the arts has a unique ability to draw the viewer deep into any situation it proposes. When cinema first arrived on the horizon its many detractors, interestingly enough from the world of painting, blamed its verisimilitude for its lack of ‘art’. But its ‘seeming’ verisimilitude, which often requires the greatest and most subtle artistry to achieve, is indeed its strength. Theatre requires a willing suspension of disbelief. Cinema, on the other hand, requires an often unwilling suspension of belief, and this is only heightened with VR. For the difference between cinema and VR, as Michael Abrash, Chief Scientist at Oculus puts it, is a matter of ‘presence’. (1)

“Presence is an incredibly powerful sensation and its unique to VR; there’s no way to create it in any other medium,” says Abrash, in one of his early public presentations. Simply put it is the sensation of feeling physically present in a scene, underwater or in outer space, at the edge of a cliff, the middle of a battlefield, inside a hut, tent, war trench, prison cell or even inside our own bodies. Abrash goes on to make an important distinction between immersion and presence. The former refers simply to feeling surrounded by the scene, he says. The latter is to feel in the scene.

But the lurching sensation in the pit of the stomach, which is what presence causes at least for the moment, is only partly from our physical vulnerability. The greater part is caused by the mental vulnerability of knowing that the medium can transport you at whim across boundaries. Almost every known border and threshold, both internal and external, has been breached, so part of the sensation is certainly on occasion outrage. Yet, as Abrash says in his sales pitch, “most people find it to be kind of magical, and we think that once people have experienced presence, they’ll want it badly.” Much of VR’s hardware research has thus been centred right here, on the refining of this sensation of presence, for this is the seat of the other uniquely VR sensation, motion sickness or vestibular imbalance. Once again for me motion sickness occurs partly from physical and partly from mental causes – when I am forced to move at a pace and proximity that offends my sense of privacy , or when my gaze is bodily yanked and re-directed. For with VR for the first time we are privy to the development of a technology that ‘engineers’ response rather than simply evoking it. This is its disconcerting – even ethically dubious -aspect. So the second level of detachment one needs to acquire is against its sensorial assault. For the sensorial brings forth not the considered but an automated response, activating what Abrash gleefully calls our ‘lizard brain’.

A classic example is ‘Senza Pezo’, called a mini opera, which traces the imagined passage of a soul after death through alternating zones of limbo and action, heat and cold, light and darkness, silence and music; compositing live action into rich CG environments in the style of fashion photography but pushed to a surreal extreme. Even as one cannot help admiring the virtuosity of the rendering one wonders at the complete lack of the aetherial in this visualization of the soul. There is nothing as material as the sensory we realise, as our eyes feel glutted at this appetitive feast.

‘Fantasynth’ is a gentler journey in the same mode. Here the simple inclusion of natural space, like the sky synthesized with artificially created zones, manages to provide a kind of visual relief. In another work ‘Artall’ we have entirely abstract renditions of both space and time as we journey through zones titled simply: garden, town, frost, spring, cube, tunnel, shaft, and finally city. This is at least intellectually challenging as all minimalist art is. We realize it is not necessary to fill every inch of the 360 image simply because it is there. Negative space goes a long way in preventing visual exhaustion as too much and too rich detail ends up repelling rather than absorbing.

Curiously enough one of the most absorbing VR films I have seen was called ‘Notes on Blindness’ based on the audio diary kept by writer and theologian John Hull when he became blind. Calling it ‘A Note on the nature of Acoustic Space’ he goes on to describe a visit to the park. The sounds of running or walking footsteps with different kinds of footwear, whispered conversations, the chirping of birds, a distant carousel, cars pulling out of the driving lot and receding along the highway, ducks in the lake, a paddleboat, children playing, the breeze rustling through the trees – ‘Every sound is a point of activity’ he muses. ‘Where there is no sound there is no activity’ – and for him, no world. In empathetic simulation the image register submits entirely to the auditory, hearkening to his voice in a nebulous dance of arising and dissolving phenomena like ultra-violet phantom figures in this world of synaesthetic darkness, a darkness shot through with the subtle illumination of his mind. All art is of course a kind of voluntary blindness. In a beautiful, impressionistic film titled ‘Monet in Giverny, the Later Years’, we listen to the painter as we wander through the rose garden in his house at Giverny. The director Stephanie Riggs who calls the piece a sharing of the artistic process rather than the product, tells us that the lines were collected from Monet’s actual interviews and letters but voiced by Broadway veteran Merwin Foard. (2) ‘Try not to see objects,” he says in a rasping voice, “but only a square of blue, an oblong of pink, a streak of yellow…not a tree, house, field or whatever…Paint it just as it looks to you. Not the object isolated as in a test tube but the object enveloped in sunlight and atmosphere…the air around objects…the blue dome of the heavens reflected in the shadows.” Dabs of colour begin to appear on a stretch of canvas and, directed by the voice, slowly and magically become a painting, ‘The Artist’s House from the Rose Garden’.

VR suits the news clip very well, adding a greater touch of authenticity and presence to reportage, making us feel we are actually there at that moment, in Syria, in Palestine, in the Native American reserves of Canada with Justin Trudeau, in Yosemite with the Obamas. In the Aldejar Falls of Iceland with kayaker Galen Volkhawen, in Ethiopia for the Fasika celebrations, in the Cayman Islands to decipher the secret-click language of dolphins and sperm whales, in Valen Reef, an archipelago in Indonesia, in Haiti and Gaza and Krakatoa. The language of the news reel is that of a first person account and often we have children speaking straight into the lens about the effects of war, migration, environmental degradation and climate change. Sometimes these become longer documentary films about the history of Cuban dance, about the making of Akram Khan’s new choreography for Giselle, about a day in the life of a ranger in Garamba National Park (a Nat Geo film directed by Kathryn Bigelow), about Ebola survivors in Liberia (‘Waves of Grace’ by Chris Milk), or about Syrian refugees in the Zaatari refugee camp in Jordan (‘Clouds over Sidra’). This is not a fly-on-the-wall camera, but fly-in-the-very-soup as we enter tents and huts and sheds already crammed to bursting. Yet both adults and children go about their lives with little or no self-consciousness, ignoring the camera or speaking into it with equal ease. It is oddly moving to listen to the diverse accents of English across the globe, Babel averted through global compassion and a pressing need to communicate, understand and empathise.

Two novels, Ready Player One by Ernest Cline, and Snowcrash by Neil Stephenson are often mentioned along with the Matrix films as the inspiration for the current market resurgence of interest in VR. Snowcrash describes a dystopian and apocalyptic world in which Hiro Protagonist tries to understand how language functions in today’s ‘post-rational’ era. The librarian, whose memory is the library itself (his personal RAM is nothing more than a search engine) explains that while monists believe in only the material world, dualists believe in a binary one, a spiritual world in addition to the material. As a hacker, Hiro finds himself leaning towards the binary and comes to understand that a world created/oriented through code can also be destroyed/disoriented through the scrambling of that code. His role as savior of the world involves therefore the protection of this code. Stephenson calls the virtual universe a Metaverse, a term that has come to stay, and both his and Cline’s character’s shift nimbly between real and virtual worlds through the simple agency of their avatars. But Cline’s Oasis is a movement in the opposite direction. It is an ode to the past, being a universe suffused in a nostalgia for the 80s, as an escape from a world gone subsequently to seed. Configured as a multi-player game with keys to be unearthed from cryptic clues, it is structured along a series of level shifts, bonus points and extra lives. The final prize is a virtual Easter Egg, with its promise of a resurrected life through the estate of James Halliday its creator and father figure. When the hero Watts/Perzival finally discovers the Egg he also finds the message Halliday/Anorak leaves behind, warning the winner against becoming too obsessed with the virtual, at the expense of the real world. And so the book ends with the two main avatar-competitors – Percival and Art3mis – coming together in real life. The competitor-collaborator dichotomy has already been breached as the third gate can only be opened by three avatars jointly but the unveiling of the real self behind the avatar is in every case a fraught one. When Watts realizes that his fellow avatar and best buddy Aich, is in fact an African American lesbian woman there is some initial discomfiture. But he finally comes to the conclusion that it doesn’t matter. “We’d known each other for years, in the most intimate way possible. We’d connected on a purely mental level …. None of that had changed or could be changed by anything as inconsequential as her gender, or skin colour, or sexual orientation,” he declares. And with that one radical statement all these ‘virtual’ categories that have been dividing our real world are dismissed with a simple wave of the hand. Like a Houdini rope trick the Oasis treasure hunt becomes an initiation ritual to distinguish between the truly real and simply virtual.

In a similar sleight of hand, Penrose Studios, who began with the delightful ‘The Rose and I’ in early 2015, about an Antoine de Saint-Exupery inspired rose plant clinging tenaciously to life on a coughing, pollution-driven universe, comes up with ‘Alumette’ in 2016, loosely inspired by Hans Christian Anderson’s Story of the Poor Little Match Girl. Eugene Chang, the director, also calls presence the ‘holy grail of VR’ as he traces his team’s process of ‘thinking natively’ in the new medium (3). ‘The traditional tools for doing film just don’t translate well,” adds Jimmy Maidens, the technical director. “We have to use internal tools rather than taking tools that were designed for another art.” The same philosophy holds good for their narrative adaptations, which are more like improvisations on the original theme than literal translations across media. So the mother and daughter arrive in a floating ship across the skies to a little toy village to sell their matchsticks. Like the firesticks we spoke about earlier illuminating cave paintings, here it is the matchstick that opens up windows in both time and space – memories and house interiors. On a bleak winters night, the daughter left alone on a street corner with just three matchsticks, chooses to warm herself not with a physical fire but by lighting up memories of her mother who sacrificed her life rather than letting the village burn. In the ‘light’ of the daughter’s own sacrifice, she is reunited with her mother. A beautiful tale poignantly told, with familiar stick figures and stop motion animation but in an immersive world that intensifies our sense of ‘presence’. The high point of this film, though, is a near insignificant detail. Early in the film when their ship stalls mid air, and the little girl stands looking perplexed, her mother walks in gracefully from her piano and repairs it with a simple flick of her wrist. Technical knowledge is no more seen/imaged as the privy of men. The promise of a new language lies precisely here, in being able to choose what tropes we want to hold/carry across into new worlds, and what we want to drop with nary a plonk into the deep, dark abyss, never to be heard of again.

To return therefore to the notion of presence, I tend to agree that it is the most distinctive feature of a VR experience, but not simply at a sensorial level. What it allows/affords rather is for us to reflect on what being in the world truly means. What is our purpose in all the worlds both real and virtual that we inhabit? In short, what is the purpose of human presence in the Multiverse?

In a charming work-in-progress called ‘Invasion’ by Boabab Studios, a small rabbit faces an alien invasion in our presence, so to speak. We see the rabbit hop out of its den in the snowy field at the centre of which we stand. It approaches us and tries to catch our attention and is then distracted by the arrival of a spaceship. It hides behind us first then steps forward and fights/tricks the invaders away and turns back to wink at us complicitly. Something very interesting happens at that moment. This is co-presence, a notion that bears considerable significance. We are not the aliens in a world of nature. We are both witness and participant, a benign and modest role in the life of the universe. Are we humble enough to accept this and move forward in a spirit of collaboration with other intelligences, natural and beyond ? Increasingly today – most notably in the film ‘Insterstellar’ but really with every attempt at space travel even if it be temporarily disguised as space tourism – we hear plans for abandoning this planet which we have harmed beyond repair and starting afresh on a new planet. To me this is just the same old colonizing impulse – ‘Interstellar’ even has them plant the national flag in the newly discovered planet. But as long as we travel with the self-same mindset, like space rats deserting a sinking planet ship, it is just a matter of time before we abuse the new environment as well.

Can engaging with virtual worlds be a mock-preparation for us to reconfigure our presence in the Multiverse? This is the spiritual question Virtual Reality proposes.

Acknowledgements

Dr Michael Kowalski, Associate Dean, Dodge College of Film and Media Arts, Chapman University

Prof. Madeline Warren, Director, Institute for Creative Reality, Dodge College of Film and Media Arts, Chapman University

¹  Microsoft PowerPoint – Abrash Dev Days 2014  http://media.steampowered.com/apps/abrashblog/Abrash%20Dev%20Days%202014.pdf

²  Riggs, Stephanie – Art in VR: Exploring the Process instead of the Product  http://immersiveriggs.com/art-in-vr-exploring-the-process-not-the-product

³   Moynihan, Tim – The Stunning Alumette  is the first VR Film Masterpiece, Wired,  20.4.16 https://www.wired.com/2016/04/alumette-groundbreaking-vr-film-tribeca

Provenence: Invited submission.

Soudhamini’s works lie in the cusp between cinema and other forms of traditional and vernacular culture in South India including music, poetry, performance and ritual. Soudhamini has a Masters in English Literature from the Stella Maris College, Madras and a Post Graduate Diploma in Film Direction from the Film and Television Institute of India, Pune. She began her professional career assisting avant garde film-maker Mani Kaul, and since then has been working across platforms and genres making short fiction, feature length non-fiction and installation works, receiving commissions from the Government of India as well as International Institutions like the ZDF, Germany and the Prince Claus Fund, Netherlands. Her work has been shown at many International venues including the Oberhausen Film Festival, FID Marseille, Creteil International Women’s Film Festival, Cinema Du Reel, Festival du Popoli, Asia Society, New York, Hawaii International Film Festival and the Leningrad International Documentary Festival, besides regular screenings at the India International Film Festivals (MIFF and IFFI). She has officiated on International Selection Committees and Competition Juries and has been Artist in Residence in Japan (2003), Munich (2006) and Berlin (2010). Soudhamini is committed to a career in critical media practice, writing and publishing her theoretical insights regularly in an effort to share and brainstorm collectively in this time of technological flux. Since 2013 she has also been teaching full time in the film program at the Srishti Institute of Art, Design and Technology, Bengaluru. She has most recently been awarded the Fulbright Nehru Research Fellowship for Academic and Professional Excellence for the year 2017-18 and is in residence at the Dodge College of Film and Media Arts at Chapman University from August 2017 to early May 2018. Her project titled Ariadne’s Thread, An Algorithm for the Digital will study the role of VR as emerging technology and its role in shaping a new cinematic narrative.

Featured Image: “Virtual Reality Headset Prototype” by Pargon is licensed under CC BY 2.0