Cyber concert

A dark B-flat minor scale from strings, horns and lower woodwinds slowly descends and fills the hall of the Royal Concert Hall in Amsterdam. Most visitors of this concert of the Royal Concertgebouw Orchestra (performed last Saturday) watch and listen in awe as the orchestra starts off with Richard Strauss’ masterpiece Eine Alpensymfonie.

In the back of the hall, however, a few dozen people are fidgeting with their tablets. Their faces are lit up. Usually smart phones are strictly forbidden in this temple of music, let alone tablets. On their little screens, they watch the score move as the music plays. Now and then a message will pop up with information about the music section that is being played.

The thirty or so people are a test panel, testing a program called Phenicx, which is an acronym for Performances as Highly Enriched and Interactive Concert Experiences. Hopes are that this app will attract new visitors to the concert hall.

“The app will help visitors with little knowledge about classical music to better grasp the music”, said project leader Cynthia Liem of the section multimedia computing (EEMCS faculty). “We hope that it will attract more young visitors. Maybe in the future the concert hall will organize special ‘bring your tablet’ events.”

The app is the result of a collaboration between TU Delft, the Royal Concertgebouw Orchestra, an app developer called Videodock and several other European universities including the Austrian Johannes Kepler-universität and the Spanish Universitat Pompeu Fabra. The European Union and the Royal Concerthall Orchestra fund this research.

The pop-ups and the synchronised scores are just the beginning. The idea is to enable users to zoom in on musicians and to amplify their tune. If you are interested in the cellos, for instance, you can zoom in on the right front side of the podium, and the cellists will sound louder. You might even get some information on your screen about the different musicians and the notes they are playing at that moment.

This is ambitious, to say the least. For decades, researchers have been trying to find ways to extract instruments from live music.

In short, the Phenicx project, can be seen as a platform to experiment with lots of different techniques, including audio signal analysis, source separation techniques, gesture modelling, score synchronization and visualization.

But isn’t all this monkeying around with tablets distracting and infesting the serene calm that typifies the concert hall? Yes, it is, says the financial director of the Royal Concerthall Orchestra, David Bazen. He doesn’t expect Phenicx to become beloved among his regular audience. “Distraction is a big issue. If only you could see the faces of the people in the concert hall if someone pulls out his telephone to check an email.”

Bazen is mostly interested in the synchronized scores that run on the tablets, a technology developed by the Austrian Johannes Kepler-universität. This technology can help his organization automate certain aspects of the concert. Currently someone has to follow the scores attentively and give cues to technicians to make sure for instance that parts of the stage are lit at precisely the right moment. If a computer is just as apt at following scores as a person, it could take over this task.

“It is extremely difficult to have a computer follow the score”, said Bazen. “Just counting seconds will not do it for as the duration of a symphony is flexible. Eine Alpensymfonie can last anywhere between 50 and 65 minutes, depending on the conductor. The computer has to compare the score with what he hears. He might hear a mixture of up to fifteen musical groups. It is as if you are listening to fifteen songs at the same time.”

The score technology is thus the primary reason for the orchestra to invest money in the project. Bazen added though that the app might come in handy during specially organised events aimed at music laics. Today’s program could fit into that description. The orchestra played several historically important music pieces and had a commentator give long and lively explanations in between. If it hadn’t been for this particular event with matching public, the test panel probably wouldn’t have gotten out of the hall unscathed.