Final Project

Linguistic Layers

My final project is titled ‘Linguistic Layers’. The project is about exploring the semantic connections between written, physical/gestural and kinetic dimensions of language.

The project is based on recognizing a viewer’s hand gestures, and changing the visual output on screen according to the gesture. The visual output is a continuously running, interactive projection with letter forms animated according to their semantic origins. For instance, the letter ‘A’, and the corresponding ‘fist’ sign, is connected with semantic meanings such as ‘power’ and ‘ox/horns’. Hence, if the viewer shows the program the ASL ‘A’, the corresponding animation attempts to communicate this semantic dimension through meaningful movement. In the case of the ASL ‘A’, the animation mimics a protective, spear-like motion. Currently, the program is trained to recognize the first five letters of the ASL alphabet, as well as a ‘no gesture’ idle pose.

Process

The project was made with a combination of three software programs: TouchDesigner with MediaPipe plugin (an open-source machine learning framework by Google), Wekinator (an open-source software for training ML models, among other functions) and Processing 4.0.

TouchDesigner and the Mediapipe plugin were first used to track hand poses from a laptop webcam. Then, the hand pose data – selected to only include the first 21 x and y values – were sent to Wekinator via OSC message. Wekinator was used to train an ML model for recognizing each of the six total poses, with the data inputs coming in from TouchDesigner. Next, the trained Wekinator model was connected to a Processing sketch through OSC message, using the oscP5 library. Processing was used to parse the Wekinator data for the six different poses, and to code a unique visual for each hand pose.

Reflections

Throughout the course, and through my project, I learnt many things. Mainly, I feel like my eyes have been cracked open – more concretely than before – to the sheer amount of software and possibilities available to people like me, who don’t have extensive coding experience, or a particularly technical background. Seven weeks ago, I would have been very skeptical about my ability to run, let alone comprehend such a project as this, including training my own ML model and managing connections between several different software. Now, I feel more confident and inspired to continue in this realm in the future. As a note, although my project itself isn’t one, I do find the idea of alternative controllers particularly fascinating. Even simple bodily interactions – such as those employed in other course mates’ projects, like jumping or tilting the head – can add so much fun and engagement to very simple concepts.

However, of course, at times during my project, I was also intimidated and frustrated. In the end, I did have to rely on help from the teacher several times. I also had technical difficulties with the software, which meant I wasn’t able to run it on my own laptop on the final showcase day. Somehow, I don’t feel it’s right to say its entirely ‘my’ project, but rather me following in the footsteps of others, and using tools curated and tested by others before I can fully grasp my own. Right now, the project also remains an exploration without a specific ‘purpose’. In the future, it would be interesting to think about hand pose recognition as, for instance, a new narrative game mechanic or alternative controller.

Despite the few setbacks, I can still look back on the last seven weeks and feel a sense of accomplishment, and inspiration for the future.

Video

Additional Materials