About UsResearchOpportunities
PublicationsResourcesSite Map
Research : Personal Imaging : Mediated Reality : Musically Mediated Reality Plug-in

Musically Mediated Reality Visualization Plugin (MRV)

Screen shot of desktop running xmms and mrv plugin, and me looking at my hand.

What is MRV?

MRV is an XMMS mp3 player plugin which alter's a user's visual perception of the world based on the music. In the same way that the simple spectrum analyzer (or other xmms plug-in) thumps along with the beat, MRV takes a video stream and alters it in time with the music. The above screenshot shows the mrv plugin running in xmms.

Mediated Reality

Mediated reality is to "augment, diminish, or otherwise alter your visual perception of reality." Most readers will be familiar with the concept of virtual reality. Mediated reality is a similar to virtual reality in that both alter the user's perception of their environment, typically by replacing their vision with the output from a head-mounted display. However, virtual and mediated reality differ in that virtual reality seeks to create an entirely new environment for the user, whereas mediated reality alters what you would normally see - i.e. it alters your perception of your actual environment.

Click for full-size
This figure depicts the signal flow in a mediated reality system. The input to the human/computer entity is first processed by the computer, before being presented to the user.

An eyetap is a device which allows all or a portion of a user's vision to be mediated by computer control providing a mediated reality. Typically, this incorporates a camera and display, configured such that the computer is able to tap into the user's vision, alter the images, and then present them to the user again, in real time.

Diagram of an Eyetap device. For more info, see our research/eyetap section.

Some examples of mediated reality currently being developed include advertisement replacement or filtering (visual 'spam' filters), or computer assisted construction where the computer presents visual cues to aid the user in construction tasks.


MRV explores the concept of a muscially mediated reality. MRV alters a user's visual perception based on music to which the user is listening. To understand what MRV does, consider the typical spectrum bar graph analyzer found on many commercial stereo equipment. These are the bars that rise and fall with the music, typically with vertical bars representing the lower freqencies (base, or lower tones) to the left, and increasing frequencies plotted to the right. The image below depicts a typical spectrum analyzer.

Example of a simple spectrum analyzer. These can often be found on the front panels of stereo systems, with the bar heights rising and falling to the music.

In the same way that the bar rise and fall to the beat of the music, MRV alters incoming video from an eyetap and redisplays it to the user. For the wearer, the world appears to pulse to the beat of the music.

Current Implementation

MRV is currently being developed as a plug-in to XMMS, an open source MP3 player. Thus, MRV will analyze and react to any MP3 xmms plays.

In time with the beat, MRV applies the GXand and GXor function operators for XImages. This was a simple way to create a cool "trailing" visual effect. Specular highlights in the scene work especially well to create bright trails, as can be seen in the screenshots below. The GXand function is periodically replaced by the GXor (to prevent the image from saturating to pure white after a while).



This is pre- code.
If you still want to try it out: Test Code(includes complied x86 library)
Untar it in your xmms-1.2.3/Visualization directory, and run the ./build.sh shell script, then copy the library to your Visualization lib dir. It works successfully with a bt848 video card (osprey100), bttv driver, v4l and xmms.

Where I'd like to "see" it go

Thanks to AMD for their donated 1.1GHz Tbird I'm using.
Eyetap.org is running on a donated VA-Linux box