Sound artist Antoine Bertin on creating ‘Play a Kandinsky’ for Google Arts & Culture

Twentieth century Russian artist, Wassily Kandinsky, said to be one of the founders of abstract art, heard sound in colours and shapes due to a neurological condition called synaesthesia. He associated, for example, trumpets with yellow, the pipe organ with blue, and the violin with red. Google Arts & Culture has captured what Kandinsky might have heard when painting his 1925 masterpiece, titled Yellow-Red-Blue, in an interactive artificial intelligence (AI) tool, Play a Kandinsky.

A project developed by Google Arts & Culture in collaboration with Centre Pompidou in Paris, Play a Kandinsky uses machine learning and Kandinsky’s colour theories to enable experiencing synaesthesia by ‘playing’ the painting Yellow-Red-Blue, which comprises a riot of colours, squares, circles, and triangles along with abstract forms, straight and curved lines, arcs and checkerboards. ‘Playing’ the painting produces complex sounds that evoke various emotions, from heavenliness and rage to restlessness, and stillness. 

Sound artist Antoine Bertin, in collaboration with NSDOS and Studio Crtq, created a musical interpretation of Yellow-Red-Blue in Play a Kandinsky. In an interview with STIR, he elaborates on the process of creating this experience.

Edited excerpts:

Sravasti Datta (SD): Can you tell us how Play a Kandinsky was commissioned to you?

Antoine Bertin (AB): I had worked as a sound designer for a project by artist Cristina Tarquini, entitled Diving into an Acidifying Ocean, commissioned by Google Arts & Culture. This project was the starting point of a series of exchanges with Google Arts & Culture curator, Clare Brooks, around creating web experiences where listening becomes central. The Play a Kandinsky project emerged as an opportunity to explore these sound ideas through the special relationship Kandinsky had with music.

SD: Can you tell us about the research you conducted for Play A Kandinsky?

AB: I plunged into Kandinsky’s writings such as Point and Line to Plane in which he describes in amazing details the nature and role of each element in his compositions; how he composed paintings similar to the ways in which a composer creates music. I also explored the music collection that was found in his artist studio, which belonged to him and is partner, Nina. This record collection, which is now part of the Centre Pompidou collection, became the starting point of my composition and investigation of what he might have heard when painting Yellow-Red-Blue in 1925.

SD: This is intriguing. Can you also tell us a little about synaesthesia and what the person dealing with this disorder goes through?

AB: Synaesthesia is a union between two or more senses. In the case of Kandinsky, his vision and his sense of hearing were associated in such a way that he would hear sounds when seeing colours or shapes, and vice versa. This neurological phenomenon became the starting point of his quest for a more liberated painting, a more ‘musical’ painting that can depict sensations and emotions without the need to be attached to a realistic or tangible depiction of what’s around us.

SD: How did you create music that would be as close as possible to Kandinsky's interpretation?

AB: It is of course impossible to hear Yellow-Red-Blue exactly like Kandinsky composed it based on his own synaesthesia. Play a Kandinsky was thus always going to be part research in the painter’s theory, and part interpretation. A question that interested me is how would have Kandinsky heard his paintings today, taking into account the evolution of music technology from his time to now. I took instruments of the classical orchestra as a starting point to the creation, and opened the musical palette through modern instruments such as synthesisers and digital processing techniques.

But the key to create a present interpretation of Kandinsky’s work that would be as close as possible from his own theories and experience, is the use of machine learning. I worked with a tool provided by Google called Google Transformer. Transformer is a neural network capable of listening to a musical score and propose continuations of this score in the same musical logic and style. I played to Transformer the music that was found in Kandinsky’s artist studio. Then curated the propositions of new score generated by the machine. Because Transformer was trained on thousands of hours of music from a variety of epochs, doing this work with the artificial intelligence becomes similar to the work of an archaeologist, weaving together fragments from the past into a story that can be experienced today. The machine, of course, generates a lot of unusable scores, there has been a huge work in selecting propositions that fits Kandinsky’s theory, while giving the machine some freedom to propose things a human composer would not necessarily find musically relevant.

SD: What, in your opinion, is the objective of Play a Kandinsky?

SD: How do you think this technology can enhance various creative disciplines, such as visual arts, music, and even the subject of mental health?

AB: It has been truly inspiring for me to be given the chance to explore the work of a fellow artist from another time in such details. Creating a continuity between different times and practices in arts and music I think is really key for ideas and tastes to consistently evolve and adapt to the present need. Regarding mental health more specifically, I think the answer is simple - all music is a chance for mind and body to heal and expand. Acquaintances with musical genres may differ from listener to listener, but like Kandinsky’s shows with his painting, art is a way to understand the universe beyond what our senses initially show us of it, similarly to mathematics and science more generally.

SD: Finally, we would love to know a little about your journey as a sound artist?

AB: I like that you selected the word ‘journey’. I have always been interested in listening, to music, of course, but also and most importantly to all kinds of sounds - acoustics, other species, wind, technological sounds, sounds from other planets… the list is infinite. There was an aquarium where I lived when I was little, and I remember spending hours observing aquatic life as I was carried away by the drone of the tank’s filter. I studied physics because I wanted to understand the nature of sound better, before graduating as a sound engineer in France, and later study sound arts in the UK. In this journey, I am currently interested in creating bridges between art and science through sound, which is why my creative process resemble more and more the work of a researcher - collecting sounds on the field, transforming data into sound (a process called sonification) to explore it. My role differs from the one of scientists, however, in that I am not interested in demonstrating an objective truth about the world, but rather to reveal some of the magic we each day discover to the ears of the public. I believe connecting to scientific discoveries through imagination, emotion, physically, socially is just as important as the discoveries themselves.