Sketch Band

Sketch Band is a machine learning music studio that allows people to draw musical instruments and play them with household objects. 


Individuals living in isolation (e.g. COVID-19) often feel a lack of creativity and inspiration, especially when it’s difficult to connect with others. This project explored new ways to use machine learning to connect with others and inspire fun and creative environments. More specifically, the project team was interested in the following question: How might we help those that live alone or with others play music even though they may not have instruments to play with where they are living?

Sketch Band is a machine learning music studio that was developed to allow people to draw instruments and play them with household objects in a fun and understandable way. It’s simple to use. First, individuals can set up the machine learning software and find a pen, paper, and something to play their instrument with. Second, they can draw an instrument (the machine learning software was trained to know a few – drums, guitar, symbol). Lastly, you can play the instrument by placing the drawing in front of the camera and hearing the sound it produces (e.g. drum noise when hitting it with a stick).

In order to develop the concept of Sketch Band, several key techniques were used. The team first needed to train the machine learning software how to recognize different drawings. Then once it could recognize different types of instruments, it was important to teach the machine to play noises when certain movements were made (e.g. hitting a drum pedal with one’s foot). These were key to developing the concept and working prototype, especially when playing songs together as a band.