Exploring Affective Storytelling with→→ VisualDesign NewMedia Technolog Exploring Affective Storytelling with→→ VisualDesign NewMedia Technolog

<Face to Face>
Interactive Video Essay

FORMAT: Interactive Video Installation
TYPE: Self-Initiated/ Research
FIELD: Video & Media/ Interaction
TECHNIC: Facial Recognition
    Face Interaction/ 3D Motion
    AE/ Processing/ MaxMSP
    Thomas Rutgers (Tech)
    Joris Landman (Production)
    Marnix de Nijs (Tech)
    Yin Aiwen (Concept/Research)
TIME: Feb-June, 2019

The Cyberspace is a very unique kind of psychological space, a space “out there” that is an extension and blending of human minds, a space where these minds can experiment with words, images, time, interpersonal connections, self-representation, and influences on the physical body and world never before possible in human history.”

—— John Suler, <Digital Age: Humans Become Electric>

is a self-reflective video essay, which narrates the social-psychological influence and intimate connections between Digital Interaction (biotechnology) and Users (human) in a personal case.

In order to communicate about social anxieties, struggles on presentation, as well as a fascination of digital identity. Through the poetic and fictional visual storytelling, I display a blended character of the “modern cyborg” between “self” and “others” (otherness), “human” and “machine”, “inner space” and “outside space”. The final work is a two-screen video installation, which presents a mirroring effect on the audience by applying real-time facial sensor and emotion detection.

As the Master graduation work, combining with theoretical and interdisciplinary research, including facial recognition technology, cybernetics, compensation and spirituality in the digital world, the whole project is an attempt to provide a personal perspective on how technology reflects us as well as the humanness in relationships to machines. It has also been nominated as "Young Talent Award" to exhibit in "Manifestations", Dutch Design Week 2019.



→ Background
Digital interactions in Cyberspace reflect our social desire or loneliness in real life. Meanwhile, as we are being fed by feedbacks from the machine every day, our behaviors and perceptions are also being reshaped by the virtual world. We are embracing both intimate and conflict relationship with digital technology. The blended identity between offline and online, reality and virtual, makes us kind of a cyborg.
→ Question

What is facial and emotion recognition technology? Where is that used in real life?
How to apply facial emotion sensors with interaction in a Narrative Video?

What are the related philosophical and social-psychological theories on Human-machine interaction and Cyberspace?
→ Highlight
Facial Action Coding System
* Relations between Human, Machine, and Technology
  <A cyborg manifesto>, by Donna Haraway
* Algorithm system as the companion species
* Spiritual Compensation in the digital and virtual world
  Mirroring effect: "Of Other Spaces: Utopias and Heterotopias", by Michel Foucault

→ Conclusion
We are all cyborgs nowadays, as human-technology mixtures in the abstraction;
The Digital technology offers an extension of mental-self, virtual-self while reflecting our daily struggle (desire) for deeper communication and authentic relation.

→ Inspiration
# Zach Blas: <Face Cages>, <Facial Weaponization Suite>
# Coralie Vogelaar, <Interface>
# Hito Steyerl
# Metahaven

︎ For Resarch about Facial Tech.
︎ For Thesis pdf

Thesis Booklet



→ Interaction & Technique 
In the process, I have tested lots of facial interactions with multiple softwares and programs, including Processing, Maxmsp, P5.js, Python. With the application "FaceOCS" and "Affectiva Emotion SDK" to track real-time facial data, the final technic system is mapped as below.

Two-screen Video Installation

→ Exhibition
The immersive space made with curtain creates more intimacy to the viewers; A slit-scan interaction was applied outside to present a poetic feeling of digital technology.

Exhibition Space