perSEAve — Jono Shaw

Jono Shaw

Human Experience Designer
Affective Interaction Explorer
Speculative Art Attempter
Social Robotics Practitioner

Projects
About
Email
LinkedIn
What People Say

perSEAve

A multidimensional body-sensing medium that emulates the dynamics of marine creatures, exemplifying an argument on a new manifestation of tangible interfaces.

# Soma Aesthetics
# Biofeedback and Biomimicry
# Sensory Interaction

Special Thanks to

Dr Tricia Flanagan
Martine Leine Rafteseth

Shot on iPhone

2022 · Interactive Installation

Interactive Art · Tangible Interfaces · Human-Computer Interaction

perSEAve is a metaphorical installation that integrates wearable and bio-sensing technologies to emulate certain dynamic and interactive essence of marine creatures through human body data, including gestures and heartbeats. At its core, it provides a primer for a proposed new genre of tangible interfaces that is fictitious but evolves in real-time, reflecting the nuanced correlations between the participant and the digital environment.

Artifact

Exploring a New Tangible Interface Through Wearables and Body-Focused Interactions

Tangible User Interfaces (TUIs) convey a prospect of Human-Computer Interaction (HCI) embodiment where interactive devices and users engage physically and sensorily through computational resources (Hornecker, 2011). Recent decades have witnessed the industry forming a prejudiced aspiration toward bridging the isolation between physical and digital entities and merging them into a reachable involvement (Colombo, 2016). Since the term “tangible” emphasizes perceivable physical embodiments instead of being purely “touchable” or “reachable”, this prejudiced view highly limits the diversity of human-computer perception and media art practices; the relationship between interactive objects and humans can embrace more sophisticated forms. Meanwhile, contemporary tangible interactions predominately adopt an action-centric approach (Fernaeus et al., 2008) that only focuses on users’ active behaviors, such as touch and movement, rather than passive data for which wearable and bio-sensing technologies have unpacked significant potential. This project conceives an interactive installation perSEAve that incorporates wearable and body-focused interactions (Nunez-Pacheco & Loke, 2014) to present an interactive media artwork while maintaining a moderate relationship between human and digital entities, thus proposing a novel state of tangible interfaces.

Background and Theoretical Foundations

Tangible interaction has undergone biased development. The vision of Tangible Bits raised by the industry pioneer Hiroshi Ishii (1997) brings possible ways to extract digital activities from traditional interfaces to create “seamless interactions”. The idea of being “seamless”, along with many of his later works such as Materiable (Nakagaki et al., 2016) and inDepth (Yoshida et al., 2021), essentially reflects a constant principle of merging digital substances and physical entities to allow close contact interactions. Other emerging mainstreams for tangible interfaces, such as AR, VR, and immersive media art, also tediously follow this philosophy (Israel et al., 2010). These practices can be seen as achieving an “Integrated State” between the human and the computer. If traditional on-screen interactions are similarly conceptualized as an “Isolated State”, an “Intermediate State” between the two can be imagined as a niche embodiment of tangible interaction. In this state, the moderate isolation between the humans and the object promotes a fictitious tangible interface with diverse interactive inputs instead of only touch, enabling multi-layered playfulness.

The Isolated, Intermediate and Integrated (III) Framework for tangible interactions

Explorations of tangible interaction mediums are inseparable from HCI evolution. In the context of HCI beyond ergonomics into a body-centric stance, critique exists that ideation and utilization of wearable technologies are hugely limited with disregard for the human body’s characteristics (Höök, 2013). Recent practices of somatic and body-focused interactions touch upon the generative dimensions of HCI (Loke & Schiphorst, 2018), providing a vision for the fictitious tangible interface in which both active and passive data from the human body are valued. Objects’ actively responding to these generative dimensions echoes Harman’s Object-Oriented Ontology (OOO) philosophy (Bogost, 2012), as they are essentially reinterpreted ontologies that perform human behavioral intentions in artistic ways without orthodox intelligence. Thus, as a pilot exploration, perSEAve utilizes a body-focused interaction design approach to develop a tangible media art. The design of artistic scenes and interactions is driven by Object-Oriented Ontology (OOO) views and provides a primer for intermediate-state tangible interactions.

Case Studies

The discussion above consolidates the distinct features of intermediate-state tangible interactions: 1) interaction embodiments are physical and multidirectional; 2) interactive inputs are both active and passive; 3) humans and devices maintain moderate isolation instead of seamless integration. Artworks matching these exact features are yet to be developed, while some precedents with similar characteristics are worth discussing.

Malaysian architect Jun Ong’s interactive illumination art Bolt leverages human body movement to create a responsive light strip system in the space above people’s heads. The light strips are only activated when human presence is detected in the corresponding space underneath, in which the vacancy allows people’s view and engagement from all directions without any physical contact. Though it is a typical example that integrates contactless interaction into an isolated but body-focused installation, the embodiment has the potential to incorporate more inputs such as sound, warmth, and even higher dimensions like the speed of passing.

Bolt by Jun Ong (Lau, 2018)

Tangible installations reinforced by both bio-sensing and Object-Oriented Ontology (OOO) are rarely practiced in the industry, while BioSensing Garden demonstrates an exemplary exploration. The installation is equipped with a small printer that drips water to the micro garden according to the exercise data retrieved from a Fitbit watch. The amount of exercise that human performs directly links to the volume of water provided to the plant, transforming humans and plants into united “ecologies”, a term emerging from the object-oriented perspective. Future iterations can involve more interconnected activities to enhance the coherency in this ecology.

BioSensing Garden by Joel et al. (OpenLab, 2014)

Design & Methods

Imagined as the intersection of intermediate-state tangible interaction, interactive media art, and wearable & bio-sensing technologies, perSEAve illuminates a miniature scene with the interactive dynamics of marine creatures as a narrative. The installation consists of a physically interactive box built with crepe paper, aluminum foil, thread, wood, and so on, together manipulated by gestures and heartbeats detected by a sensor-embedded glove. This glove, equipped with capacitive touch, gesture recognition, and heart rate sensors, translates user movements into real-time changes within the virtual marine setting, such as wave fluctuations and bioluminescent displays. The design process, grounded in a material-centered approach (Döring et al., 2012), ensured seamless integration of technology with wearable textiles, providing intuitive correlations between human behaviors and physical scenes within the interactive seascape powered by an Arduino platform.

Technical implementation of perSEAve

Discussion

This project envisions a new stance on tangible interfaces by proposing an interactive media art installation using wearable and bio-sensing technologies. The design, evaluation and iterations of perSEAve utilizes a Research through Design (RtD) approach (Zimmerman & Forlizzi, 2014) and remain open to adjustments along with the development. Interactive scenes of perSEAve are powered by physical components without the intention of being touchable or immersive. As a result, the isolation between humans and devices enables participants to control, observe, and interact with this artificial sea as they please. As a parallel demonstration of human behaviors, this body-focused installation reformats humans and devices into equally participatory objects, approaching an object-oriented perception. However, more experiments and evaluations are needed to consolidate the theory and vision of the fictitious interface in intermediate-state tangible interactions.

References

Bogost, I. (2012). Alien phenomenology, or, What it’s like to be a thing. University of Minnesota Press.

Colombo, S. (2016). Merging Digital and Physical: Tangible Interactions. In Dynamic Products: Shaping Information to Engage and Persuade (pp. 21–30). Springer International Publishing. https://doi.org/10.1007/9783319331171_2

Döring, T., Sylvester, A., & Schmidt, A. (2012). Exploring material-centered design concepts for tangible interaction. CHI EA ’12: CHI ’12 Extended Abstracts on Human Factors in Computing Systems, 1523–1528. https://doi.org/10.1145/2212776.2223666

Fernaeus, Y., Tholander, J., & Jonsson, M. (2008). Beyond representations: towards an action-centric perspective on tangible interaction. International Journal of Arts and Technology, 1(3/4), 249. https://doi.org/10.1504/ijart.2008.022362

Höök, K. (2013). Affect and experiential approaches. In S. Price, C. Jewitt, & B. Brown (Eds.), The SAGE Handbook of Digital Technology Research (pp. 174–188). Sage Publications; DiVA. https://doi.org/10.4135/9781446282229.n13

Hornecker, E. (2011). The role of physicality in tangible and embodied interactions. Interactions, 18(2), 19. https://doi.org/10.1145/1925820.1925826

Ishii, H., & Ullmer, B. (1997). Tangible bits: Towards seamless interfaces between people, bits and atoms. CHI ’97: Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, 234–241. https://doi.org/10.1145/258549.258715

Israel, J. H., Belaifa, O., Gispen, A., & Stark, R. (2010). An object-centric interaction framework for tangible interfaces in virtual environments. TEI ’11: Proceedings of the Fifth International Conference on Tangible, Embedded, and Embodied Interaction, 325–332. https://doi.org/10.1145/1935701.1935777

Lau, Y. (2018, March 23). Making the Intangible Tangible: Exploring Jun Ong’s Light Art. The Artling. https://theartling.com/en/artzine/making-the-intangible-tangible-exploring-jun-ongs-light-art/

Loke, L., & Schiphorst, T. (2018). The somatic turn in human-computer interaction. Interactions, 25(5), 54–5863. https://doi.org/10.1145/3236675

Nakagaki, K., Vink, L., Counts, J., Windham, D., Leithinger, D., Follmer, S., & Ishii, H. (2016). Materiable: Rendering dynamic material properties in response to direct physical touch with shape changing interfaces. CHI ’16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 2764–2772. https://doi.org/10.1145/2858036.2858104

Nunez-Pacheco, C., & Loke, L. (2014). Crafting the body-tool: A body-centred perspective on wearable technology. DIS ’14: Proceedings of the 2014 Conference on Designing Interactive Systems, 553–566. https://doi.org/10.1145/2598510.2598546

OpenLab. (2014). OpenLab: BioSensing Garden – OpenLab UCSC. OpenLab UCSC. https://openlabresearch.com/openlab-biosensing-garden

Yoshida, T., Ogawa, J., Choi, K. Y., Bushnaq, S., Nakagaki, K., & Ishii, H. (2021). InDepth: Force-based interaction with objects beyond a physical barrier. TEI ’21: Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction. https://doi.org/10.1145/3430524.3442447

Zimmerman, J., & Forlizzi, J. (2014). Research Through Design in HCI. In J. S. Olson & W. A. Kellogg (Eds.), Ways of Knowing in HCI (pp. 167–189). Springer New York. https://doi.org/10.1007/9781493903788_8