Multimedia Lecture Notes

Most of this course has dealt with the physics (optics and acoustics), math (sampling, coding),  psychology (perception, linguistics) and practice (production, construction) of audio, video, and film. Had we included print, we might have properly called it a multi media course.

In some sense, the media we did examine are all multimedia in that they tend to involve each other. Film uses video. Television uses film. Both use audio and graphics. As we have talked about audio, film, and video we tend to think in terms of producing artifacts (programs, movies, recordings, etc.) that are intended for mass distribution and near simultaneous consumption.

Multimedia is something of a new kid on the block.

The earliest use of the term  usually referred to the simultaneous use of several separate media such as slides, video, cassette tapes, film, and even live performance in the same space. Performances and installations using multiple media became known as multimedia installations. See for example the work of Bill Viola now installed at the Art Institute of Chicago. Video artists were/are often referred to as multimedia artists.

The new meaning of the term is quite different. It refers to a single medium that integrates video, film, graphics, print, audio, etc. The integration is accomplished (mediated) digitally through computers. Multimedia products are characterized by interactivity that, as a minimum, gives choice points and that, at its best, allows user (consumer) input to the product. For the most part the new meaning relates to personal computers of some sort.

Multimedia are produced using many formats, the most common of which at the moment is the CD-ROM used in personal computers. (There are a number of CD formats. Most hold between 500 and 800 Mb of uncompressed data, enough for over an hour of high quality stereo audio).

  • CD-DA (audio)
  • CD-ROM
  • CD-ROM XA (extended architecture)
  • CD-I
  • CD-R (recordable)
  • CD-RW (rewritable)
  • Multimedia are seen in many other formats such as the proprietary game media such as Play Station, arcade games, and simulators of various sorts.

    Newer video formats using sophisticated compression algorithms such as DVD

    Interactivity is the key element distinguishing multimedia from traditional media in which the audience is essentially a passive receiver(s). Interactivity requires an interface or some channel through which the interactivity takes place. There must be a pathway from audience/participant(s) to the medium and a pathway from the medium to the audience/participant(s). Some possibilities are:

  • Keyboard
  • Mouse
  • Joystick or other ergonomic device
  • Touch screens
  • Voice command
  • Eye movement tracking
  • Brain wave monitors
  • Full body transduction suits
  • Multimedia are being used in a variety of ways ranging from the simplest presentation applications to much more complex interactive narrative games.

    At the moment we have the ability to do the basics of multimedia. That is we can provide an interactive environment of sight and sound that is reactive to varying degrees of audience input with variable outcomes. We have media that can provide feedback (corrective or not) to participant input. Newer versions can provide reasonably high quality visual and aural experiences within the confines of the interface (most often a CRT or LCD screen and speakers or headphones). We can do all this but only within the rather extreme limits of current computer technology.

    Remember the quest to accurately reproduce or represent nature or imagination?

    Multimedia in general tend to lag the technological advances of its components when they are considered alone.
    Limitations in processing speed, screen display, audio chains, etc. mean multimedia representations of moving images for example are typically not as realistic as the same images seen in the movies or on hi resolution video. In a real sense multimedia represent a step backward in presentation but a step forward in interactivity and integration of components.

    Limits defined by computer processing power and the quality of the analog devices such as screen display and sound system. Audio, 10 Mb per minute. Video 300 Mb per minute. All of this depends upon the quality of the audio or video. No Free Lunch.

    Multimedia production begins with the same principles we have talked about in other media. Good composition in audio and video is most likely good composition in multimedia as well. Good graphics, text, content, etc. are prerequisite for good multimedia.

    Can we assume a straight transfer of the aesthetic of film and audio as we have developed it over the years into multimedia? We are beginning to see the codification of design principles, at least for the World Wide Web . Of course, it could be a mistake to equate what happens on the web with the whole of multimedia. Nevertheless, some codifications are emerging, even some Do-It-Yourself instructions.

    No free lunch. Multimedia producers must do a balancing act between quality, price, and speed. Higher quality means higher price and lower speed. Higher speed means higher price but lower quality. Speed considerations apply in both the authoring phase and the playback phase.

    Things are being improved considerably by both the increasing speed and storage of personal computers. but also by new data compression techniques, faster transmisson technologies, newer programming languages and more powerful design applications.  Java (try some free apps), Shockwave, Real Audio, MP3, Quicktime, etc

    The main challenge beyond balancing the compromises (That has to be done with any media) lies in creating the conventions of narration, story telling, choice availability, navigation, etc. Multimedia has not been around long enough for us to have well established conventions of construction. We have not established a narrative aesthetic in multimedia. Thus all producers are in effect on the cutting edge.

    Most design is around choice points, decision events that do not exist in ordinary linear media such as movies or novels.

    Multimedia must be designed around multiple flow patterns the specifics of which are chosen by the audience. They make choices that determine the pace, direction, and to varying degrees, the outcome of the experience.

    There is an emerging aesthetic.

  • Choice mechanisms should be intuitive, requiring little training of the audience
  • Interfaces should be simple
  • Make good use of metaphor but be aware of the limitations (e.g. "desktop")
  • Choose smooth and fast over pretty and exact
  • Avoid heavy use of text
  • Avoid disruptive consequences of choice points