Kinetic controller driven adaptive and dynamic music composition systems
Principal Investigators: Carlos Guedes, University of Porto and Bruce Pennycook, University of Texas at Austin
Abstract
This joint research project will develop new techniques and strategies for computer-assisted composition in the context of real-time user control with non-standard human interface devices for applications in electronic art and digital entertainment systems. The research team will design and implement real-time software, hardware and specialized human-interfaces that will provide tools and resources for music, dance, theatre, installation artists, interactive kiosks, computer games, internet/web information systems.
Besides incorporating and developing important aspects of recent work in the fields of gestural control of music and real-time automatic music generation, the project will extend the work of Portuguese artist-researchers Carlos Guedes and Tomás Henriques and the work of UT Austin researcher Bruce Pennycook.
The outcome of the project will be the creation of a modular toolbox for real-time dynamic music generation that will allow for easy creation of software applications for the purposes described above. The toolbox will be highly flexible allowing its use both by trained musicians and the general public. Simply by patching together the desired modules for music generation, musical parameters can be seamlessly operated and controlled by gesture driven interface/kinetic controllers, thereby granting the user of the system a very intuitive way of music control and interaction. Moreover, instead of using the usual parameters for musical control such as scale, meter, tempo, etc, a metaphoric type approach will be used based on expressive descriptors such as “faster/slower,” “aggressive/calm,” “loud/quiet,” etc. The musical engine to be developed will be able to generate music in recognizable genres such as Jazz, Rock, R&B, Ambient, etc, as well as generate innovative sound and music systems.
The toolbox will consist of C/C++ external objects and abstractions for Max/MSP and Pd programming environments. These will fall into two categories: the first consists of objects and abstractions for procedural music while the second is a collection of objects and abstractions that map the input of the gestural/kinetic controllers into musical parameters for real time control. Parallel development of C# algorithms and interfaces will be pursued for compatibility with YDreams´ YVision framework. They will ease the development of interactive installations and will function as the musical counterpart to YVision.
INESC-Porto, Universidade Nova de Lisboa, University of Texas at Austin, with industrial affiliates Casa da Música and YDreams as partners make up the consortium for this joint research project.
Casa da Música and YDreams are pivotal partners that will keep the overall focus of the project, the creation of a software toolbox for real time control and generation of music able to be utilized by a broad range of users, into applications meant to be engaging, entertaining and stimulating, due to the quality of the musical output and ease of interaction and manipulation. The applications to be developed will be aimed both at (1) highly specialized users aiming at a standard professional quality for use in products such as inline/offline interactive marketing, computer assisted performance and accompaniment, interactive installations, computer games, etc.; (2) non-specialized users, including people with disabilities, children and the elderly for use in sound based games, interactive music creation and cognitive sound stimulation.
Objectives
The project’s main goal is the creation of a toolbox that allows the development of generative music applications whose parameters can be controlled in real time by simple gestural interfaces such as laptop cameras, several types of games controllers, as well as new digital controllers currently being developed by members of the research team.
This toolbox is aimed at a broad range of users, from the highly specialized to the musical illiterate, and will ultimately serve the creation of procedural music applications, such as musical software for educational and rehabilitation purposes, generative music for games, music for theater and dance performances, computer assisted performance and accompaniment, and musical applications for collaborative music-making on the web. The toolbox be integrated into YDreams’s YVision, a framework for easy development of interactive installations.
By advancing the research in automatic music generation systems this project will add a new body of knowledge through the development of innovative solutions in human computer interaction.