Movement and Mindfulness Visualised by Machine Learning – PoseNet & P5.js
Captured Transition is a work which discovers the movement and morphing of the physical body interpreted by a machine. Presented as large-scale prints and contact sheets.
EXPERIMENT: MACHINE LEARNING, PoseNet
Research and Sources
Inspiration
http://www.dear-data.com/theproject
My inspiration for the output of the machine learning PoseNet sketch I hope to create is very much data visualisation but in a very organic subtle and natural way. I was instantly reminded of Giorgia Lupi and Stefanie Posavec dear data project where they would send each other postcards with hand drawn data visualisations. I like the idea of using organic shapes like circles to create an image and I hope to map certain trained yoga poses to a colour meaning there would be a distinction between the poses and I could create a small key just like the ones Giorgia Lupi and Stefanie Posavec used in their postcards. I hope as an outcome for this work I could make 1 image a day for 30 days for example and I could make this into a book its almost like a diary/memory of past practice. Maybe it would even be fun to try and recreate the points by hand possibly in watercolour. I have a lot of ideas for this and I think playing with the visual outputs in code will be very fun, I’m very excited about this. I am happy that I have an idea with an expressive output in mind since initially I was put off trying to make a Yoga Machine Learning Model because most of these are turned into apps and websites and are used more as personal trainers/teachers than having the moment of the body merged with the machine create an expressive artistic output.
Machine Learning and Yoga?
https://cris-maillo.github.io/yogAI/index.html
(A note from the future May 2021) I found an artist who attempted to make new Constellations based on combining yoga and machine learning I felt this was quite interesting to mention here concept wise it isn’t what I am exploring in my own work but I was glad to see more interest around combining yoga and AI especially in the realm of creative coding.
Exploring Visuals
I wanted to make a mock up of what the visuals could possible look like so I took images of poses drew over key points and stacked them on top of each other. Of course this will look different in the future but I am aware that sometimes when I have something in my mind other people may not envision it the same way so I thought I would do this before I start playing with code.
Training a model
Use this sketch: https://editor.p5js.org/AndreasRef/sketches/RLv1QbuLab
name for points I want to draw: key points (look for this in the sketch) line 221/41
Aim: work on one pose at a time change colour if it is held and draw more opaque dots
Tuesday 26th January
I tried to start working on training the model today but honestly I still find the code very confusing and I’m not really sure where I should start. I’m looking at Andreas Refsgaard’s example but when it loads it freezes and nothing works so I might be doing something wrong my aim is to figure this out so I can keep moving forward.
Wednesday 27th January
YAY! I figured it out I was frustrated that I wasn’t understanding why the sketch wouldn’t run and then I had the idea of switching browser and it worked! So I can finally start moving forward with this.
I realise I will need another person to help me if I want to do a few trials with this particular sketch as 1 person will need to do the pose and the second will click. I think there’s a possibility of feeding in a video and training the pose that way or setting a timer before it starts scanning like Dan Shiffman done in his example. So I know I have options but for now just to test things out I might ask my boyfriend for help.
Thursday 28th January
I thought I should watch this video to get a better grasp of the complex code I will be working with, I feel if I watch someone coding it from scratch it might help me understand more of the code better and it might be easier for me to change code so that it does what I want it to. I know I could dive straight in and try to type random code hoping it works but I feel I could get quickly discouraged if I don’t know what I’m doing. I also tried to change some things yesterday and ended up being quite confused.
Coding while watching video
So first thing Dan Shiffman did was drawing the video over the canvas. I actually want to have the canvas without the video so that we can see what is left behind (drawn) by our movement on the canvas and also I commented out the video.hide(); so that we can still see what the webcam sees underneath the canvas and the training buttons.
When I was looking for some more PoseNet tutorials to watch I had a video show up in my recommendations of someone using PoseNet with Yoga! The account posted the code under their video which was really nice, luckily the code is different to what I am hoping to make because from what I gathered a video of someone doing yoga was loaded into P5 and they were drawing certain key points like the wrists, ankles, elbows and knees. I believe there was no training involved. I will link the video below as I think it’s nice other people are interested in representing something slightly similar to me.
I liked this video about regression as it made me consider an idea for a future experiment possibly combining yoga flow movement with sound but this isn’t something I will focus on right now.
Monday 2nd February 2021
I’m still having trouble moving forward with poseNet. I’ve been focused on my physical computing experiment. The example I found for saving and loading model training files doesn’t actually involve poseNet and I’m not sure how to save files using the poseNet example. I’m a little discouraged because I don’t have much experience with P5.js and I feel this is what is stopping my progress.
So I figured out that I just need to change the name of the Jason file to that of what it is named in the code.
Since I understand how to save files now I asked my boyfriend for help in training a few poses. I already encountered a few problems as PoseNet doesn’t seem to recognise some of the poses like downward facing dog or cobra, it seems to be confused and doesn’t detect the legs but I will try this during day time with better lighting these are just my observations from the first attempt.
I trained 3 poses here A- Mountain Pose B- Yogi Squat C- Warrior II.
I’m pretty happy with its ability to detect the different poses now I might start to play around with the visual output maybe starting with signing each pose a different colour and changing A,B and C for the names of the poses in a corner of the screen.
These changes are obviously very basic but it’s always good to start with something simple. I will continue developing this throughout the week.
Wednesday 3rd February
I wanted to begin drawing the poses on the canvas today and start experimenting with the visuals.
The individual images with one colour are the separate poses drawing the key points and skeleton. For the one with 3 colours I turned on all the poses and just let it draw my movements I kind of enjoy how busy it is but I will have to consider the framerate I want it to draw at and also figure out a way for there to be a class where there is no pose because right now it is trying to constantly find one of the 3 shapes. The smaller circle in the middle draws when the probability of the key points is >4 and the transparent dots when it is >2. This isn’t exactly how I want this to work and I need to figure out how to make transparent dots when I am not in the pose and make the dots more opaque when I am in the pose.
Thursday 4th February
I am trying to make a distinction between the key points drawn when the pose is detected compared to when it is not.
In the first image the grey dots where the passive key points and the pink key points represent the mountain pose being detected in the second image the grey was changed to a plum colour. I’m still trying to figure out how to make this look better and whether I should limit the frame count so that less dots are drawn and if I need to fade the background so that the canvas doesn’t become too crowded.
I trained another model A- Forward Fold B- Chair Pose and I wanted to do another but I am still struggling with PoseNet not detecting because of the background. I wanted to train these pauses because they quite naturally fit together and it is easy to move from one to the other. I want to start choosing different colours and experimenting.
I really liked the outcome of this I changed the key points to be significantly smaller and less opaque so that the skeleton lines were more visible. I am actually surprised at how much I liked this since at first I didn’t even want to draw the skeleton but I really enjoyed how structured the outcome became. Since I am portraying what a machine sees when watching someone do yoga it may be interesting to show something that is usually very fluid and flowing (yoga) splayed much more mechanically like the machine would consider it to be but because I get to make the creative choices the lines to me still seem soft and subtle representing the human form quite well in my opinion.
I have been considering that colour is really important to me in this work and I want colours that complement each other nicely so that the transition between the poses looks visually appealing rather than disagreeable. I decided to look at some graphic designer webpages about different colour palettes and picked a few that I liked because I still want this project to be quite personal and I make decisions about which poses I choose to train, how I choose to draw them on the canvas wether it be with lines for the skeleton or key points, the sizing of each thing drawn so I felt like choosing the colours was also quite important and it gives nice verity this was I can have the same code providing many different visual outcomes.
I felt that at this point it’s quite important that I can easily save PNG images of the outcomes so that I don’t have pixelated screenshots as outputs. I managed to figure out how to add a button that downloads a png image to my laptop which I was quite happy about because I’m not really familiar with coding in P5.js I had to write a line in HTML to actually draw the button and it works fine.
.PNG image output from p5.js 
.jpg image output from p5.js
Some screenshots of how I added the button. This makes me feel quite positive as I think it will be quite simple for me to add more classes into one sketch if I wanted to meaning more transitions through varying poses. I also think I should start drawing a larger canvas so that future images will be high quality.
Tuesday 9th February
I have seen ‘Yogasan Charts’ on the internet and thought it would be fun to recreate one with the code I wrote for the chair pose and forward fold. I felt it would be interesting to show a chart that already exists but through the eyes of the machine and also the poses won’t be shown as an instruction instead they will be my personal attempts through the eyes of the machine as I’m not sure I can do all the advanced poses. I see this small exercise/experiment as a fun way to continue exploring visuals rather than building a product or finished piece. I will not be training each of these poses separately instead I hope to just draw the key points and say an image of the pose and then put it together into a visual chart. This reminds me of how I staged my abstract ‘robot drawings’ created with the servo motor with each drawing being put together into a grid format.
This is a visual I seen a lot on when looking for yoga charts however it is mostly posted on pintrest and and I haven’t found any research suggesting its authenticity which is why I think of this more as an exercise to continue exploring visuals.
I decided to tweek the code I was already working with and picked 3 background colours inspired by the chart shown above. I decided to just let the sketch draw the key points and skeleton it sees instead of training each pose separately as I wanted to do 20 which would take quite a lot of time. I’ll place some images from the process below.
THE FINAL CHART OUTCOME
How a machine sees yoga
I’m really happy with how this came out. It is not very easy to see exactly where each part of the body is because I think PoseNet gets confused by some of the poses and doesn’t see the legs but this could be caused by a busy background or the lighting in my living room but I think it adds a certain abstract quality to the images which I enjoy. The outcomes remind me again of last years Physical Computing outcomes from my ‘robot drawings’ and I really enjoyed those so I am quite happy to create something with a similar aesthetic. I am also quite happy I managed to get this done in a day and I enjoyed trying all those different yoga poses and putting the whole chart together in Photoshop.
April 2021
I really wanted to come back to these poseNet drawings as I wanted to have some nice final outcomes to show for the work I put into this particular experiment. I decided to go with black and white showing the transition between two poses and I wanted to export these as .png images so I could potentially experiment with the background later but I am pretty set on blue.
Contact Sheets
I have been considering different ways to present and show some of my posenet outcomes and testing the most impactful staging. I am still deciding but some of my current options are large scale prints, framed prints and Jen mentioned I could possibly try the axi draw. Even though I like some of these options as outcomes I was still curious and excited to try and create contact sheets. I believe prints would be a very impactful way to present my visuals but I enjoy how contact sheets were a way for a photographer to see a lot of their photographs to then make choices, for me it’s presenting a visual timeline in 2D images and I feel that helps to portray my concept of movement and transitions shown as images and it almost feels even more dynamic to me.
After speaking with Paul today during my 1v1 tutorial he mentioned that the outcomes reminded him of Giacometti’s figure drawings and that I could consider presenting the work as a room filled projection like Communion (2020) link. I like this idea but I do feel I would have had to tailor this work to be moving image and I had envisioned it as stills exploring the challenge of only using lines and dots in a very minimalistic colour palette to present movement and the morphing of the physical body.
Captured Transition is a work which discovers the movement and morphing of the physical body interpreted by a machine. PoseNet is a machine learning model which recognises the human skeleton and key points of the body. In P5.js, a JavaScript library for creative coding, I used PoseNet and trained the machine to recognise certain yoga poses and to draw the skeleton and key points of the body on a canvas in the web browser.
The PoseNet model is very good at recognising people however with more complex yoga poses it would sometimes miss limbs in turn creating abstract outcomes and drawings.
The white lines represent the pose which the machine was trained to recognise while the black lines show the transition through to the second pose not taught to the machine. The abstract drawings vary between more delicate and subtle to bolder more obvious outcomes where a human body is more easily identified.
The work is presented as large-scale prints and contact sheets. I created the contact sheets to consider the sketches/outcomes in more detail displayed in a timeline format while the large prints are a human scale display of the drawings.
Marta Zofia Palacz 