A set of AI-based tools to automate the digitalisation and cataloguing of historical mediaeval and classical music manuscripts
Building our Future from our Past
The Horizon Europe project REPERTORIUM is creating AI tools that protect Europe’s ancient musical roots and that develop out of this heritage innovative technological solutions to empower the future of the music industry.
REPERTORIUM is a project in which AI can be used not only to generate content and modernise the world but also to drive tools that allow us to learn more about our own past, make discoveries in it, and transform the way we relate to our heritage and culture.
The Linares School of Engineering of the University of Jaén heads the project consortium, comprising participants from 8 countries (Spain, The Netherlands, Italy, Finland, Germany, the United Kingdom, France and Lithuania).
Partners include the University of Oxford, the Complutense Institute of Musical Sciences in Madrid (ICCMU), the Polytechnic of Milan, Tampere University, the University of Alicante, creative marketing firm vist.co kulturmanagement in Hamburg, the cultural technology developer ODRATEK BV, developer Spork Digital in the UK, the French association Musicologie Médiévale, the Hispanic Association for the Study of Gregorian Chant (AHisECGre), and the Lithuanian National Philharmonic Society.
REPERTORIUM stands for Researching and Encouraging the Promulgation of European Repertory through Technologies Operating on Records Interrelated Utilising Machines.
The project operates in two primary spheres: musicology and sound processing. Though the musicological portion is heavily focused on mediaeval and classical period repertoire, the sound processing outcomes leverage these subjects to create tools applicable to all forms of acoustic music.
In musicology, it combines multimodal artificial intelligence and deep learning solutions to perform optical music recognition and music information retrieval across multiple music score image archives. The project will provide a technological platform for preserving European musical heritage through databases of medieval and classical works. These will be enriched through manuscript digitisation, music information retrieval techniques, and audio-to-score technology using AI techniques, allowing automatic data linkage with other relevant existing databases worldwide.
Over 400,000 ancient manuscript images will be scanned and digitised from the Solesmes archive and made available for musicological study through Oxford’s DIAMM portal. Over two million musical works will be correlated for concordance amongst the Solesmes dataset and external datasets such as the Cantus Index network. Approximately 4,000 novel chants are expected to be discovered and prioritised for manual indexing. Thousands of chants will be manually annotated for training the optimal music recognition AI model, and 127.000 will be manually indexed for music information retrieval testing. Approximately 2.200 hours of Gregorian chants sung in the Tridentine forms, corresponding to the manuscript tradition, will be recorded at the Abbey of Sainte Madeleine (France). Between 1.800 and 3.000 handwritten operatic aria manuscripts will be digitised from ICCMU’s DIDONE database.
The sound processing outcomes will build upon these corpora by applying information extracted from musical scores to Deep Learning sound spectral analysis, algorithmic signal processing beamforming techniques (able to “zoom” on the spatial sound of single musicians), and psychoacoustics models. This hybrid combination will empower technologies for state-of-the-art real-time instrument sound source separation explicitly adapted for classical music.
Cost-effective immersive audio recording tools will be built upon AI-based signal processing and spatial audio ambisonics (a 3D-audio encoding technique), leveraging the system’s score knowledge, audio-to-score tracking, phase and spectral analysis results.
The performance of current 3D-audio rendering techniques is limited by the number of expensive Higher-Order Ambisonic microphones used and the high computational costs required. REPERTORIUM will create novel hybrid solutions for sound field reconstruction that will be flexible regarding computational and hardware requirements (combining less expensive linear microphone arrays and data-driven methodologies) and will work in real time.
More than merely virtualising an acoustic space, this outcome enables accurate recreation of the actual acoustic space in which material was recorded, allowing full reconstruction of the sound wavefront at any given location.
Metaverse-ready immersive technologies will be tested through two streamed symphonic concerts, allowing the public to freely “wander about” fully immersive spatial audio sound scenes like an acoustic hologram. Users will select among 360° video hotspots for a complete audio-visual experience.
Dissemination
Performance and recording of the novel Gregorian chants will bring recovered melodies to life not heard in over 1,000 years. An opera from the DIDONE database will also be staged, showcasing music for which the scores were digitally engraved using the project’s optical music recognition technology.
Dedicated promotional activities will be coordinated closely with relevant stakeholders, including multiple technical workshops to show the sound source separation and “minus-one” (karaoke) features in conservatories and music schools. The consortium will attend several international scientific conferences and trade fairs, publish at least 15 scientific papers, and generate additional educational actions and materials for the broader public.
The project outcomes will impact both musicological and sound processing areas. REPERTORIUM will immediately impact musicological scholarship through its digitisation efforts. Four thousand novel Gregorian chants are expected to be highlighted for study. The contents of the Solesmes archive will be accessible online. This impact in preserving and evaluating musical heritage will extend to future digitisation efforts worldwide through the resulting optical music recognition and music information retrieval toolsets.
An API (a protocol that allows sites to share data securely) will make the developed linked data model and AI-assisted semantic enrichment techniques available to other projects.
These tools will aid commercial music publishers interested in digitising back catalogues and archives digitising their collections. The recordings of Abbey of Sainte-Madeleine’s chants will be published through Neumz, an app that offers the complete recordings of Gregorian chants with their scores, texts, and translations.
Regarding sound processing, new commercial products connecting orchestras and labels to the public in the most immersive 3D sonic experience ever created will enhance streaming revenues. Solutions for music students to create real-time, on-demand minus-one (muting an individual instrument for practice purposes) versions of their favourite recordings will revolutionise at-home practice.
Overall, REPERTORIUM will generally widen the accessibility, appeal, and knowledge of classical music heritage among the public and academics.
In addition, the project represents a significant step towards creating a general musical artificial intelligence with machine equivalents for eyes, ears, memory, intellect, and imagination. Doing so is an important contribution to exploring the AI-human values alignment problem by making European cultural heritage available to machine learning solutions. The project is an essential step towards the evolution of artificial intelligence that can authentically reflect human civilisation’s implicit values while achieving its programmers’ explicit objectives.
REPERTORIUM builds our future from our past by preparing the ground for an AI to understand and conform to human intentions, preferences, and goals as embodied in centuries of European civilisation’s traditions and artefacts. REPERTORIUM will serve as a model in shaping our new human-centred digital world.
