Project MODAVIS – Multimodal Organ Data Analysis and Virtualization Systems

Funding by the German Academic Scholarship Foundation

by Dominik UkolovGeneral September 9, 2022August 12, 2024No Comments

I am incredibly happy to announce that the German Academic Scholarship Foundation (Ger.: Studienstiftung des deutschen Volkes) will support my PhD-Project for the upcoming 2-3 years! It was a long, but steady way to get to this point, and I’ve already worked on increasing my chances for receiving this funding during my M.A., so for about 4 years.

Especially if you come from a working-class family and are the first academic in your family, this support and appreciation means a lot and is very gratefully received. The whole process kept me motivated to follow the intended idea and to dispel my doubts through logic and evidences, e.g. by writing a detailled Exposé with proof of concepts, regular consultations with my supervisors and reflections on my researches so far. The fact that the expert reviewers of the foundation (who are university professors in this field) trust me to realise this project let’s me start with a clear mind, it motivates me even more to pursue a high scientific value and a societal benefit, thank you for this!

At this point I’d like to encourage every indvidual, regardless of their background, to put some effort into applying for a scholarship, as it’s worth it even if the answer is negative! I can provide some experiences with German Study-, Exposé- and PhD-Scholarships, so if you have some questions about this topic or need some help with the application system, feel free to contact me personally!

I decided to publish my Exposé that has been submitted to the foundation, but it is only up to date as of March 01, 2022 and in german, a few aspects have already been updated: Download Exposé (.pdf)

tl;dr

I’ve received a scholarship for the Project MODAVIS for 2-3 years
(outdated) Project Exposé (in German) can be downloaded through the Link provided above
Contact me if you need tips and advice for scholarships in Germany!

Conference: Cultural Heritage and New Technologies (Vienna, 2022)

by Dominik UkolovConferences & Travels August 20, 2022August 12, 20248,430 Comments

As my project focuses on the digitization of musical instruments that are inherent components of cultural heritage sites, the conference Cultural Heritage and New Technologies (CHNT 27) seemed ideal to present one of my works about the acoustical spaces that are surrounding the organ. I’ll propose a system that will detect materials, textures and shapes in a photogrammetric model through artificial neural networks to build a new model for acoustical simulations including a mapping of the derived absorption coefficients. You can download the preview of the Short Paper below. The conference will be held in the City Hall of Vienna between November 10-12, 2022.

Date: November 11, 2022
Session: Cultural Heritage and the Metaverse: Linking the Cyber and the Physical to protect and share Cultural Heritage (Intro)
Chair: Lorenzo Ceccon

Title: Multimodal Digitization Concepts for Acoustic Cultural Heritages. Using Photogrammetric Models for the Preservation and Auralization of Historical Acoustical Spaces
Preview to Short Paper (v0): Download

Historical musical instruments and acoustical heritage sites in the Metaverse deserve much more attention, I’ll probably set a focus on this at the end of the project when the structures become more clear. I’m excited to hear the other contributions at this conference!

Released: midiAuralizer

by Dominik UkolovCodes & Softwares August 12, 2022August 12, 20243,044 Comments

Finally, I’ve decided to publish my first software, the midiAuralizer!
This tool comes with an easy graphical user interface and allows to auralize MIDI-Files through any virtual instrument with a simple click, additionally it automatically logs all instrument parameters into a protocol database, while all generated files can be traced through a cryptographic code (hashcode: MD5), which prevents losing the auralization parameters after renaming an Audiofile at some point of processing or data management.

midiAuralizer Screenshot

At the moment it’s only available as a GUI-Version for Windows, as VST-Instruments are only compatible with this OS, but it will be updated to support SoundFonts and AudioUnits on Linux and Mac OS!

You can download the midiAuralizer here, while some instructions, future changes and the full code can be viewed on the GitHub-Page.

Feel free to write me at any time if you have any suggestions or experience some bugs or problems!

tl;dr

Software to auralize MIDI-Files through VST-Instruments is available to download now
It will be updated to work on Linux and Mac OS with SoundFonts and AudioUnits in near future!

Annual Meeting of the American Musical Instrument Society 2022

by Dominik UkolovConferences & Travels August 12, 2022August 12, 20244,621 Comments

After starting with my project in April 2022, I felt very honored to present my work about Virtual Acoustic Objects in June at the Annual Meeting of the American Musical Instrument Society at the National Music Centre of Canada in Calgary. I was very glad that my colleague Dr. Heike Fricke accompanied me, as she was talking about DISKOS and complemented many aspects of my presentation, also she introduced me to almost everyone she knew from her work in New York (Metropolitan), Washington and Edinburgh – exceptionally nice people with interesting projects! Luckily I received the Frederick R. Selch Award for my presentation, together with Frederick Lapointe, who did a great job in his research about instrument builders in 19th century Montreal.
I was really amazed by the high quality and diversity of the presentations in these days, some of them are still present in my mind and led me to new perspectives on organology in general.

There are lots of Skyscrapers in Calgary, a pretty young and interesting city with some beautiful surrounding nature (the Rocky Mountains and the Banff National Park are nearby). The high number of homeless people and the gentrification in the east of the city were very noticeable. A network of overground-tunnels called Skywalk / +15 also connects many of those skyscrapers, so you can cross the city without walking outside.

The collection of the Studio Bell / NMC and the building itself were pretty impressive, I highly appreciated the collection of rare synthesizers, but especially their fresh ideas and the high competence of the young team at Studio Bell. The concerts on a digital theatre organ, Ondes Martenots and on the TONTO-System (played by Robin Hatch) were pretty special and memorable.

This was actually my first time on the american continent, with all these new impressions and kind people I met it is certain that I won’t forget my first conference as a doctoral researcher! I still take the resonance and the award as an important motivating factor to keep going and to spread the word about the project, but also to build up cooperations and work together with interested specialists all around the world.

AMIS 2022 - Screenshot from Presentation

Thanks to everyone who participated and I hope to see you again soon!

tl;dr

The Annual Meeting of the American Musical Instrument Society (AMIS) was highly motivating and gave me new perspectives on diverse organological topics
The Studio Bell / National Music Centre of Canada has an impressive collection of synthesizers and a great, young team!
Full Presentation: Amplify / YouTube

About MODAVIS

by Dominik UkolovGeneral August 12, 2022August 12, 202421,233 Comments

The PhD-Project MODAVIS – short for Multimodal Organ Data Analysis and Virtualization Systems – focuses on the development of strategies to virtualize pipe organs and softwares to simplify & standardize this process. Its concept has evolved in the Research Group Digital Organology at the Museum for Musical Instruments of Leipzig University after the digitization project TASTEN and the ongoing research project DISKOS, both that have been funded by the Federal Ministry of Education and Research of Germany. It combines all experiences that have been made within these projects as well as the knowledge of the supervisors Prof. Dr. Josef Focht (Organology, Leipzig University) and Prof. Dr. Rolf Bader (Acoustics, Hamburg University), additionally the concept contains some experiences from maintaining the organological web-database musiXplora and my bachelor thesis about the Sound Synthesis with Artificial Neural Nets (2017) and my master thesis about the Functional Augmentation and Acoustical Virtualization of a Theatre Organ (2020).

TASTEN-Project: Digitizing 36 Historical Key Instruments with Prototypes of Virtual Instruments.

The developed tools and strategies don’t only address recording, audio processing, editing or 3D-Scanning, but also the management, classification, analysis and connection of data and organological information. It will finally result in a new standard format for Virtual Acoustic Objects (VAO), which will include all multimodal data. These VAOs will therefore consist of Audiodata, 3D-Models, Animation & Motion Data, Musical Action Data (Piano Rolls, MIDIs), Measurements, Organological Information and the playable Virtual Instrument itself, everything encoded into a single .vao-file, which can be distributed and updated with proper changelogs. For this distribution, a Platform for Virtual Acoustic Objects will be developed, where it’ll be possible to upload, play, download and discuss about the virtual and the physical instruments for institutions and individuals.

After the development of the capturing and management software, which will constantly be optimized, they will be used for the virtualization of signifiant historical pipe organs in Europe, but they’ll also be provided to other institutions and interested enthusiasts around the world, for example to the Institute for Historical Organs in Oaxaca, Mexico. These international cooperations are another important factor of the project and show a main principle of this project: everything shall be free, open and easy to use, even with simple equipment and without any technical knowledge.

Virtual Acoustic Objects

Virtual Acoustic Objects: Slide from the Presentation at the Annual Meeting of the American Musical Instrument Society (AMIS) in June 2022 (Studio Bell, Calgary/Canada)

Another focus is on the optimized creation of photogrammetric models, where a tool for the conversion of the room models for the purpose acoustic simulations will be developed. This tool uses neural networks to classify and segment objects, materials and textures for which a special dataset is being created. The principles and state of development will be presented at the Conference for Cultural Heritage and New Technologies in Vienna.

Since the beginning of this project in April 2022 I’ve been working on achieving a photorealistic quality for instrument models with low-cost equipment to provide a proper result both for remote researches and for AR/VR and Metaverse implementations.

mXp 4010106

With another technique that is currently under development it’ll be easy to create documentations of recording situations within minutes to enable a threedimensional localization of microphones in its surrounding, which can be used for acoustical simulations and sound field decompositions.

tl;dr

MODAVIS aims at developing tools & strategies for the virtualization of pipe organs
Influenced by experiences from previous digitization projects of musical instruments and historical music media
International cooperations with institutions and individuals around the world
A new standard format called Virtual Acoustic Objects incorporating multimodal data will be developed
Tools for acoustical models and simulations with photogrammetric captures will be developed
A Platfom to share, play & download these Virtual Acoustic Objects will publicly be available
Focus on an easy usability & everything will be free, open-source and cross-platform!

Latest Update: November 09, 2022