Using machine learning object detection model to find shapes in generated images from the yoga model I trained. Inspired by Henri Mattise I used these shapes to create cut-outs and collage. This work allowed me to experiment with bright colours and bold compositions creating a final large scale wall collage and a compositional ‘scrap-book’.
Wednesday 3rd March
When exploring the idea of what a machine sees when we practice yoga and meditation I used Runway object detection to identity what shape of a person it seen in my photographs. I enjoyed how playful these outcome shapes were almost like balloons, not taking into account the complexities and imperfections of the human body and it’s form. My intention before I seen the results was to just create a poster of these but when I seen the outcomes I started to imagine them as an interactive sculpture. At first I instantly imaged them being clear and see through and I though I could use sheets of plastic to make them and then I started to imagine what I would do If I had space and no budget and I envisioned a large scale interactive sculpture. I hope to use different tools to present how I envision this idea and also the outcome is a combination of the previous experiments I have been working and focusing on in Semester 2. Right now I am thinking of using tinkered to create a simple 3D model of what I envision the sculpture to be right now in early stages of figuring it out and possibly Maya. I also hope to make a physical smaller scale prototype probably using cardboard and maybe laminate pouch plastic. I am also considering adding the LED lights with the Arduino but I have a lot of testing and physical experimenting to do as I want to see how the materials will behave and if adding light will be effective. I also want o make a conscious decision of which poses to use what they mean and also what the interaction with the sculpture will be and what it will trigger/affect.
Inspiration
https://yurisuzuki.com/artist/sonicplayground
I felt this sculpture by Yuri Suzuki is quite relevant and could be a source of inspiration due to its playful nature. Sonic Playground consists of 6 sculptures that allows the audience to manipulate acoustics and sound by interacting with the human sized sculptures. I remembered this piece due to the size of it and the playful colours it instantly renovated with me as the yoga poses ‘blob’ shape outcomes have the same kind of visual to me even though the shapes are much different. At first I considered the sculpture I want to make to be made of transparent material but now I am thinking it may be of benefit to play with colour or at least make some experiments and prototypes using it inspired by Suzuki.
Monday 8th March
I wanted to connect some of the research I have done about Chakras in connection to yoga poses with my runway outputs to begin experimenting with what the sculpture could be and I felt like rather than going with the first idea that I think of this time I will make some small experiments and to see if it forms my idea of the ‘sculpture’ or whatever this becomes more clearly with a clear process and making choices through making.
I chose images of yoga poses with correspond with each Chakra. I selected the images from the dataset I used to train my Runway ML model. I used object detection and PoseNet models in runway to get these visual outcomes of ‘what the machine sees’ in these images of yoga poses.
I wanted to make a visual chart to document each of the chakras with the corresponding pose and colour scheme along with the outputs from Runway ML. This is meant to simply show and explain at this stage what my working looks like and as a reference point for any future work. I anticipate and hope some of my outcomes will be very abstract so I feel it is important to include these ‘first steps’ here on the blog as these shapes, names, colours and visuals could inspire future work that won’t be so obviously tied back to the origin. I really want to isolate the PoseNet outcome for the poses as well as the object detection and explore those visuals.
Since I have been thinking about and wanting to make a sculpture and a prototype for it I thought I could try something quite basic, minimal and hands on. I used the outcomes from the object detection to make these paper poses with a coloured outline of the relevant Chakra for each pose. They are lined up in the order of the Chakras. At first I didn’t really like this and I’m starting to question if I want to move forward with these object detection pose ‘blobs’ shapes but I think it is quite beneficial to sometimes take these complicated computer processes out of the screen and play with physical materials such as paper to spark inspiration.
When I was taking some pictures of the paper prototype I was playing with the light source and I had a small lamp. Earlier I had considered cutting out the shapes shown above in paper like a hole in the middle of the page and then shining a light through and projecting onto a wall in a dark room, I haven’t done that but when I was moving the lamp around I noticed the shadows were making really interesting shapes and visuals on the paper. I actually really enjoy these images and the movement element to them as well as the delicate layers that appear to be showing. So making the paper prototype was not a complete waste of time, I think these visuals I really successful and I enjoy them. I really like how they are an outcome of all the process and consideration of using runway and picking certain poses as well creating the image dataset of me doing yoga in the first place.
Tuesday 9th March
Group Tutorial 2:30pm with Gillian, Ailsa and Nomi
Henri Émile Benoît Matisse
La Tristesse du Roi Lithograph by Henri Matisse, 1958 
The Parakeet and the Mermaid. 1952. Gouache on paper, cut and pasted, and charcoal on paper, 11′ 11/16″ x 25′ 2 9/16″ (337 x 768.5 cm)
https://www.moma.org/interactives/exhibitions/2014/matisse/the-cut-outs.html
After the tutorial I was really inspired when Nomi mentioned Henri Matisse and his cut-outs works. I wanted to play around with the code I have previously made implementing an array and trying to create the effect of a decoupage through creative coding.
At first I experimented with the runway output array that I had and I figured out how to implement the random function to affect the size of the images and their placement on the canvas.
I decided this would be a good time to implement the images I want to work with so I worked on adding an array with the runway shapes outputs. This array currently only has 20 .png images but I hope to add more while I continue with this experiment.
Wednesday 10th March
Today I have been thinking of the visual I managed to implement in processing yesterday and I had the idea to play with the transparency of the shapes almost as if it was different kinds of paper and tissue. I think moving forward this might be an interesting area to explore.
I was also considering that these decoupage creative coding ‘drawings’ could be influenced in some way by interacting with the drawing board and I think this will be the next thing I should aim to experiment with.
I began tweeking the code to see if I could somehow connect the balance board to the drawing code. It took some time for it to do exactly what I wanted because The whole array was drawing on top of each other at once and I wanted to pull out 1 image from the array at a time.
In simple terms the position of the human being in the physical world standing on the board determines the placement of the shapes on the processing sketch. The virtualX and virtualY values pulley’s from the board then draw on the x and y values of the sketch.
I think this is really successful at this stage and I see myself working on and polishing these sketches.
Possible things to consider:
- Adding too and amending the dataset of the array. I think I want to use the more abstract shape outcomes than the forms I am currently working with. This means time consuming work in photoshop pulling new shapes.
- Staging the outcomes in the physical world, looking at Matisse’s process of painting, cutting and pinning as a possible way to stage the outcomes. I could use a wall in my house to present the outcomes.
Thursday 11th March
James Whitney
https://lightcone.org/en/filmmaker-341-james-whitney
When I was considering the outcomes and staging I could create for this work one idea was to follow the process of Matisse and physically pin cut-outs to my walls which is an idea that I still like and will experiment with but it also made me think of the equivalent to paper being pinned to a wall in the realm of technology and the tools we have access to now. This made me consider projection and I was instantly excited by the idea so I decided to look for something within my budget and ordered one last night it is expected to arrive today and I hope it will work well.
This then made me think of James Whitney’s work and his bold use of shape and colour in his movies especially Variations on a Circle (1941-1942) and how to me they seem like a combination of a processing sketch and a projection. This made me consider possibly using blendmode in my code as well as possibly adding more movement to the images being draw.
I will now work on creating the dataset I want to use in my code which might be time consuming I will describe my whole process below.
First of all I had to get my generated images from my yoga poses Runway model through object detection here are some results:
I particularly like these two results because cat and cow are 2 very well known poses in yoga, if I end up making a visual image based book I will want to add these two images.
I then began extracting the shapes in photoshop and exported them as .pdf files. This was a very long time consuming process as I used the magic wand to select the shape, I then created a new layer, filled in the shape by selecting a colour that was present within the shape that runway recognised and drew. I managed to get around 70 shapes out of the runway images. These are the images that I will feed into my array and sync up with the wii board to draw the digital collages.
Experimenting with code and visuals using the dataset of shapes shown above.
My new projector arrived today and I was extremely happy to begin experimenting with some of the visuals.
I then began playing around with my code, beginning with implementing my new array of images. I then added rotation because I wanted to add more of that collage effect so that when the images are drawn they will be drawn at different angles, currently this is happening through a random number (angle) between 0 and 360. I also changed some of the mapped numbers to better fill the screen once I made the sketch fullScreen.
The virtual X and visual Y values of the board are controlling where the shapes are being placed, the processing sketch is then projected onto a wall. Essentially the movement for example yoga or even sitting on the board and meditating will draw these collages where the shapes are based on artificial intelligence and machine learning trying to understand yoga poses through pictures of me.
Tuesday 23rd March
As well as exploring the AI generated shapes with projections and Interaction I really wanted to follow Mattise’s technique of his cut-outs to create a non-digital outcome for one of this work. It’s currently spring break and I felt that having some time away from the screen will be a nice balance but digital media is still the origin of this work since machine learning was used to create the shapes and I also use a projector to get accurate tracings.
Firstly I began with choosing a colour palette. I chose 10 colours, I wanted really bright colours staying true to Mattise’s cut-outs work since he always used bright bold colours. I wanted to have some light and some quite dark colours to create a balance and contrast in the final piece. I used Gouache paints just like Mattise.
I then went on to paint A3 paper. I also got some soft box lights to help with lighting so I can have better documentation of my work.
I then used a projector to help me trace the AI generated shapes and cut them out using scissors.
I knew that I would be left with a lot of scraps and I had the idea to do shape compositional studies in a very small sketchbook using the scraps from the cutouts.
6th April
Over spring break I worked on this as I wanted to create a big collage on my wall as well as this I really wanted to make the small ‘scrap book’ with the left overs from the shape cutouts as I felt I could explore making interesting small scale compositional studies and also not wasting the painted paper.
I filmed the progress of this piece starting with painting the paper and tracing the shapes using a projector to cutting them out and pinning of the collage I hope to make these into short time-lapse clips to show the progress of creating this.
Sunday 11th April
The ‘scrap-book’ made with the scraps left over from the large work above. Simple compositional studies each page containing no more than two colours.
I’m currently working on gluing down the cut outs instead of using blu tack so that it will be easier to flip through, the video above shows what it looks like with blu tack and some of the cutouts get struck together making it difficult to flip pages.
Overall, I am very pleased with my collage and scrap-book outcomes. I enjoy the fact I could take something I generated digitally out into the physical world of paper, paints and scissors and have a chance to physically explore the shapes dataset I created. I took the shapes dataset further and created an interactive experience (Part 10 of my learning journal Immersive Projections) but I am happy that I took some time away from the screen to explore these works in the physical realm.
Marta Zofia Palacz 