PhotoSYNTH
The PhotoSYNTHs are a range of multiplayer, collaborative, mixed-reality POP-music factories that grant anyone nearby instant musical superpowers, and transforms any group of individuals into a pop-band! You can find more information about th PhotoSYNTH, including the science and musical theory behind it here.
PhotoSYNTH was developed as part of the DMLab programme at Drake Music, a leading charity working in Disability, music and technology. We’re devoted to using new technologies and ideas to open up access to music for all. Our vision is a world where Disabled and Non-Disabled musicians work together as equals.
Royal Society Summer Science Exhibition 2025
We’re very excited to be exhibiting at the Royal Society Summer Science Exhibition 2025, the Royal Society’s free annual celebration of cutting-edge science. This free event, suitable for all ages, showcases the latest research from across the UK, through interactive exhibits, hands-on activities and talks.
Come along and try out the instruments and explore the future of making music, along with a host of other fascinating cutting edge research programmes.
What you’ll experience:
- The chance to play both multiplayer and holographic versions of the PhotoSynth
- Bespoke accessible instruments developed with the London Philharmonic Orchestra
- MiMU gloves: the world’s most advanced wearable musical instrument
Visitor information:
When: 1st-6th July
Where: The Royal Society
Cost: Free
The Kit
By tracking your motions and emotions in real-time, this magic mirror understands your moods and feelings, converting your everyday familiar movements and expressions into glorious music; where every articulation becomes an audio revelation! It is one of the easiest and most fun musical instruments to play—yet remains highly expressive, explorative and accessible.
Rather than playing a musical instrument, the PhotoSYNTH learns how to play you!
Without any training you can instantly make great music.
Finally, a purpose for your reflection!
Magic mirror on the wall, who’s the fairest of them all?
To make an accessible musical instrument takes many stages; product planning, research, development, testing, implementing adjustments and reiterations…
The PhotoSYNTH was designed from the ground up to be a fair musical instrument. By including disabled people as part of the development cycle, and testing at every stage with real users, has allowed for the natural development of many innovations and unique control mechanisms, driven mainly through feedback and inspiration through experimentation.
We have used the latest in motion capture and augmented reality technology to enable the PhotoSYNTH to understand your movements and emotions and have written custom software so that it knows how to react to them. Full facial landmark tracking, including eye-gaze and blink tracking, as well as eyebrow observations, allows us to create a unique interface for each user based around their needs, converting familiar expressions into musical melodies, such as converting smiles into notes. By tracking multiple people at the same time, each player (or band member!) is responsible for a layer of the music and is assigned a relevant instrument to harmonise with the other players, allowing for music to be collaboratively created.
At the AMIC (Accessible Musical Instrument Collection) there is a Holographic version, designed to be operated by a single player and configurable for use from any angle in any orientation, from sitting down to lying in bed.
This 3D version of the PhotoSYNTH can synchronise with the multiplayer version ensuring that everyone can jam together.
The onboard MIDI ports allow other electronic musical instruments to be plugged in and controlled in the same fashion, as well as any DAW software or even your entire studio!
An important aspect of accessibility is availability, therefore, there is a free home version, available to all. The PhotoSYNTH runs on your personal devices using their built-in webcameras to analyse your movements, converting them into music, as well as controlling any connected digital musical instruments.
All of the PhotoSYNTHs are released alongside their research and code under a very permissive open-source license, so that you can remix them and add your own features and abilities; on the condition that you share your improvements with the world.
The Story
Drake Music’s first Hackathon in 2014 brought together makers from all over the world, including Zenon Olenski, the, “Godfather of biometric music”, who presented his PhotoSYNTH II—a remix of an earlier project that converted light, colours and movements into music.
By meeting users with specific access needs, the team at Drake Music were able to suggest improvements and ideas to make a PhotoSYNTH that was both more accessible to people with disabilities and simple enough for musical beginners. Before long, what was once a simple monophonic musical instrument became a fully-fledged “Pop-music factory”, which was both more enjoyable to play and also lovely to listen to!
As the technology matured and improved, new features were integrated that added more fine-grained expression, fidelity and control. Advancements in motion-capture and neural-networking made them available in real-time on regular home hardware such as laptops and mobile phones, allowing for the development of a new user interface based on observing familiar movements and augmenting abilities. This allowed for smiles to be converted into musical notes, movements into melodies and facial expressions into modulators and audio filters.
In 1999, Zen created a machine that converts dreams into musical soundtracks by recording biometrics—the dreamCapturer—representing real emotions with music, which induce those same emotions when listened back. This research into the emotional connection between music and expression formed the groundwork for the latest PhotoSYNTH—that reads your emotional state and uses it as the basis for your own audio expressions. In this way, the music you create is an extension of your mood, and the expressions on your face become the controls for the music, so rather than having to wrestle with a musical instrument to represent how you feel, instead the PhotoSYNTH will guage your intentions and create sound to represent them.
Over 25 years of research into how emotions are affected by music, and how to control emotions using sound, has resulted in this custom software capable of reading your emotional state and authentically representing it through music and audio expression.
This, “Sympathetic synthesizer”, tries to represent your expression accurately and honestly, simplifying the act of making music that reflects your own emotions. Maximising freedom of expression without limiting the sound possibilities invites exploration beyond your boundaries, resulting in a creative, expressive, and dynamic workflow with sometimes surprising results. Working alongside disabled and differently-abled people has evolved and fine-tuned the PhotoSYNTH so that almost everybody can use it to not only make music, but also to have fun, on their own or with friends!
