Field notes on organizing a research-led Creative Coding exercise

I am organizing an exercise for design students attending a course on Creative Coding. In my PhD research I am very interested in design practices that focus on making design visible, obstructive and agonistic in order to manifest technological mediations in society. In particular, I think it's an interesting space to explore when it is applied on complex technologies, through prototypes that are used to critically reflect on the role that they have in the design process, and how they could affect society. Additionally, it could be a promising exercise for design student in order to experiment with a material that they have limited experience with (coding) without the limitations of developing a functioning product ready for production.

This approach borrows heavily from critical design practices, and it is a first step at introducing thinking through doing in the teaching context of my research group. Specifically, I'm focusing on how design deploys metaphors to situate technological products, especially the ones that fall under the definition of ubicomp (short for ubiquitous computing).

The goal of the exercise is to develop new metaphors to construct computational objects by reworking existing ones, and to develop them in working prototypes that make data collection processes and technologies visible rather than invisible. An excerpt of the prompt I am writing to kickstart the exercise:

The theme for this year’s group project is “presence”: in a context in which technology used in design products is rendered invisible and intangible, aiming for a quasi-magical experience, we prompt you in exploring the opposite of this metaphor. How can we describe technology that becomes present, tangible, persistent and manifest? How do we conceptualise it, through which metaphors? And which shape does it take when constructed as a functioning artefact?

This framing is influenced by the work of Jonathan Chapman, who collaboratively developed "Critical Technical Practice" (see Sengers 2005, which cites CTP as a methodology to perform Reflective Design), which prompts designers to:

(1) identify the core metaphors of the field, (2) notice what, when working within those metaphors, remains marginalized, (3) invert the dominant metaphors to bring that margin to the center and (4) embody the alternative as a new technology.

Of course, gathering "core metaphors" is a big part of the research, which I don't think I have the material time to perform. Fortunately, some work has been already done, albeit at the beginning of the last decade. Mike Kuniavsky compiled a list of large scope metaphors that describe ubicomp on two levels: "organizational metaphors", or how various devices communicate with each other, and "interaction metaphors", or how we interact with devices. This is an interesting starting point, as it creates a basis from which I can start in organizing data collection technologies of my interest to work with design students.

These are some of the results of images illustrating these metaphors using Midjourney.

Back to nature

Ubiquitous computing returns us to a pre-industrial life. This metaphor implies that technology and nature can coexist in a way that de-emphasizes technology to the point that it becomes indistinguishable from nature.

The factory

The world is a factory. In this metaphor, people own the factory. Their role is to act as overseers, organizing an efficient assembly line of appliances to produce ever more free time. The implication is that automation will automatically make more time for pleasurable activities.

Vapor

Computation is a cloud that surrounds us. This metaphor compares the permanent availability of information and information processing to a vapor enveloping (and possible penetrating) every person and every object.

These cards will act as prompts for design students to start from to imagine alternative trajectories of technological development with the aim of making unfamiliar artifacts that make data and technology "present" — hopefully.

p5.js affordances

The course will teach design students how to use p5.js, a Javascript library based off Processing which allows to quickly sketch computational artefacts using various affordances. The library (with additional libraries) allows to work with:

1. Inputs (touch, keyboard, microphone)
2. Geolocation
3. Sensors (accelerometer, gyroscope)
4. Face, body and hand recognition
5. Speech recognition
6. Servers and connectivity

Slow technology

A similar approach has been developed by Hallnäs and Redström through their definition of "slow technology", or technology that requires effort to be used. In slow technology, time becomes a fundamental aspect of the expressionality of computational objects: dwelling is central in using slow tech, and their design is focused on the long-term. In this framing which, to some extent, incorporates the inversion of some narrative metaphors, technological objects can be grouped into two families of artifacts: informative art and soniture (portmanteau for "sound" and "furniture"). The authors define them accordingly:

With soniture, we mean the more or less movable things in a room that give the room its sounds, the sounds that equip it for living and makes it into the particular room it is. [...] Using the sound as a central property of material amplifies the presence of things and makes learning and understanding slower. Consider the "nightingale floor" of Japanese Shogun castles, a singing floor that was built to warn against intruders.

Posters, pictures, paintings, etc. are often used to furnish the walls of our homes, offices and other places. [...] In more traditional forms of information visualisation, the design problem is how to create a structure that represents the information as efficiently and readable as possible. In informative art, these structures are often more or less given by the conventions of what posters and pictures might be like in order to fit into the desired environment, or by some other set of aesthetical preferences.

It becomes relevant that the focus of the researchers, here, is mainly related to interaction design and product design, the latter not being the focus of this course. This opens up the space of applying similar concepts to digital technologies that can be developed by students using web-based resources.

An example that could be reproduced with p5.js in some capacity is the following, where each time personal information is collected this Chrome extension produces a noise:

I made a very very simple tool that makes some noise every time your computer sends data to Google. Here a demo on the official Dutch government jobs site. The noise starts while typing the domain name already. Code, currently Linux only: https://t.co/ZjKeOSfYff pic.twitter.com/dEr8ktIGdo

— Bert Hubert 🇺🇦 (@bert_hu_bert) August 21, 2022

In summary

To summarize, the four aspects that I introduce in this post come together in building a "playground" of building blocks that could be used to think critically about digital technologies.

Readings

Hallnäs, L., & Redström, J. (2002). From use to presence: On the expressions and aesthetics of everyday computational things. ACM Transactions on Computer-Human Interaction, 9(2), 106–124. https://doi.org/10.1145/513665.513668

Hallnäs, L., & Redström, J. (2001). Slow Technology – Designing for Reflection. Personal and Ubiquitous Computing, 5(3), 201–212. https://doi.org/10.1007/PL00000019

Kuniavsky, M. (2010). Smart things: Ubiquitous computing user experience design. Morgan Kaufmann Publisher.

Pierce, J. (2021). In Tension with Progression: Grasping the Frictional Tendencies of Speculative, Critical, and other Alternative Designs. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1–19. https://doi.org/10.1145/3411764.3445406

Pierce, J., & Paulos, E. (2014). Counterfunctional things: Exploring possibilities in designing digital limitations. Proceedings of the 2014 Conference on Designing Interactive Systems, 375–384. https://doi.org/10.1145/2598510.2598522

Sengers, P., Boehner, K., David, S., & Kaye, J. “Jofish.” (2005). Reflective design. Proceedings of the 4th Decennial Conference on Critical Computing between Sense and Sensibility - CC ’05, 49. https://doi.org/10.1145/1094562.1094569