The PhD-Project MODAVIS – short for Multimodal Organ Data Analysis and Virtualization Systems – focuses on the development of strategies to virtualize pipe organs and softwares to simplify & standardize this process. Its concept has evolved in the Research Group Digital Organology at the Museum for Musical Instruments of Leipzig University after the digitization project TASTEN and the ongoing research project DISKOS, both that have been funded by the Federal Ministry of Education and Research of Germany. It combines all experiences that have been made within these projects as well as the knowledge of the supervisors Prof. Dr. Josef Focht (Organology, Leipzig University) and Prof. Dr. Rolf Bader (Acoustics, Hamburg University), additionally the concept contains some experiences from maintaining the organological web-database musiXplora and my bachelor thesis about the Sound Synthesis with Artificial Neural Nets (2017) and my master thesis about the Functional Augmentation and Acoustical Virtualization of a Theatre Organ (2020).
TASTEN-Project: Digitizing 36 Historical Key Instruments with Prototypes of Virtual Instruments.
The developed tools and strategies don’t only address recording, audio processing, editing or 3D-Scanning, but also the management, classification, analysis and connection of data and organological information. It will finally result in a new standard format for Virtual Acoustic Objects (VAO), which will include all multimodal data. These VAOs will therefore consist of Audiodata, 3D-Models, Animation & Motion Data, Musical Action Data (Piano Rolls, MIDIs), Measurements, Organological Information and the playable Virtual Instrument itself, everything encoded into a single .vao-file, which can be distributed and updated with proper changelogs. For this distribution, a Platform for Virtual Acoustic Objects will be developed, where it’ll be possible to upload, play, download and discuss about the virtual and the physical instruments for institutions and individuals.
After the development of the capturing and management software, which will constantly be optimized, they will be used for the virtualization of signifiant historical pipe organs in Europe, but they’ll also be provided to other institutions and interested enthusiasts around the world, for example to the Institute for Historical Organs in Oaxaca, Mexico. These international cooperations are another important factor of the project and show a main principle of this project: everything shall be free, open and easy to use, even with simple equipment and without any technical knowledge.
Virtual Acoustic Objects: Slide from the Presentation at the Annual Meeting of the American Musical Instrument Society (AMIS) in June 2022 (Studio Bell, Calgary/Canada)
Another focus is on the optimized creation of photogrammetric models, where a tool for the conversion of the room models for the purpose acoustic simulations will be developed. This tool uses neural networks to classify and segment objects, materials and textures for which a special dataset is being created. The principles and state of development will be presented at the Conference for Cultural Heritage and New Technologies in Vienna.
Since the beginning of this project in April 2022 I’ve been working on achieving a photorealistic quality for instrument models with low-cost equipment to provide a proper result both for remote researches and for AR/VR and Metaverse implementations.
With another technique that is currently under development it’ll be easy to create documentations of recording situations within minutes to enable a threedimensional localization of microphones in its surrounding, which can be used for acoustical simulations and sound field decompositions.
tl;dr
- MODAVIS aims at developing tools & strategies for the virtualization of pipe organs
- Influenced by experiences from previous digitization projects of musical instruments and historical music media
- International cooperations with institutions and individuals around the world
- A new standard format called Virtual Acoustic Objects incorporating multimodal data will be developed
- Tools for acoustical models and simulations with photogrammetric captures will be developed
- A Platfom to share, play & download these Virtual Acoustic Objects will publicly be available
- Focus on an easy usability & everything will be free, open-source and cross-platform!
Latest Update: November 09, 2022