Hoogeveen, M.J. (1995). Towards a New Multimedia Paradigm: is Multimedia Assisted Instruction Really Effective? In: ED-MEDIA 95 Proceedings, June, Graz, Austria.

Royal PTT Nederland NV/KPN Research

Groningen, the Netherlands

A strong paradigmatic belief can be noted in the benevolent effects of multimedia for a wide variety of application domains, not in the last place Multimedia Assisted Instruction (MAI) [Heller, 1990]. Multimedia is used here in the sense of the property of a system or object indicating that multiple information types, such as speech, music, text, graphic, still, animation and video are used in an integrated manner. In many studies the learning effectiveness of multimedia is presumed. However, the true experimental foundation of such assumptions and beliefs is incomplete and often weak, and a coherent theoretic basis explaining why multimedia is supposed to work, taking into account the experimental findings, is not (yet) given.

The value of making explicit the multimedia paradigm, and underlying assumptions and hypothesis is that:

• it gives insight into the effectiveness of MAI systems;

• it makes effectiveness hypotheses visible and thus testable;

• hypotheses (insofar confirmed) can be used cautiously as guiding principles when developing MAI systems.

The current multimedia paradigm is formulated in Section 2. The theoretical and experimental support and limitations of this multimedia paradigm are explored in the subsections of Section 3. In conclusion, an attempt is made to formulate a new, more realistic multimedia paradigm [see Section 4].

In literature and multimedia projects [Hoogeveen, 1994] one can recognise a set of convictions, based on observations and experimental findings, with regard to the effectiveness of multimedia which is described below in the form of the multimedia paradigm. The word paradigm stresses the aspect of widespread, underlying convictions which often take the form of a firm belief, rather than just a set of testable hypotheses. This is certainly the case with multimedia.

The multimedia paradigm is the dominant conviction that adding multimedia functionality to information systems (ISs) leads to improved information and knowledge transfer to people. In other words, the multimedia paradigm is about the vision of computers as effective "tools for the mind" [Marmolin, 1991]. Rich communication (multimediality), a high level of interactivity, a high level of congruence of used information types, an adequate usage of reference models (e.g., by data visualisation), and an adequate quality of represented information are presumed to contribute, via a number of interactions, to the improved information and knowledge transfer

The interaction of independent multimedia variables is supposed to lead to a number of psychological responses:

• a high level of stimulation of the senses, at least with regard to the auditory and visual perception systems;

• a high level of involvement, attention, concentration;

• emotional arousal, e.g., fun; the word arousal is used in the psychophysiological sense of emotional, internal arousal, related to arousal of the nervous system;

• strong recognition effects, using mental reference models.

These psychological responses interact in a complex way so that they give people the feeling they experience information instead of acquiring it. The impact of multimedia messages is assumed to be high in comparison to non-multimedia messages by conventional media, and thus lead to a hypothesised improved transfer of information and knowledge to people.

It is believed that, if carefully applied, multimedia improves not only

In support of the multimedia paradigm are notions from Perceptual Psychology. Perception psychologist Gibson [Marmolin, 1991] argues that our senses are constructed to handle the very complex flow of information in natural environments, and that our senses are not constructed to handle simple stimuli. Gibson argues further that we are not passive receivers of information. Instead, our perceptual system is characterised by the pick up of information and by the integration of activities in the different senses. Although these argument are directed at the experimental study of human perception they are also relevant to multimedia computing: ISs that need to support human information processing very effectively, should make full use of human perceptual and cognitive capabilities and therefore should represent natural information flows to users and offer support to process natural information flows. According to [Marmolin, 1991] complex, dynamic and integrated representations of information really are necessary to utilise all the capabilities of the mind.

A number of independent multimedia variables are believed to have a psychological impact: the level of multimediality, the level of man-machine interactivity, the level of congruence of information types used, the usage of reference models and the quality of representation.

1. The level of multimediality can be operationalised in terms of the number of information types used. The assumptions are that the higher the level of multimediality:

• the more stimulation of the senses;

• the more arousal;

• the higher the involvement;

• the stronger the recognition effects.

The idea is that with a high level of multimediality it is more easy, than with a low level of multimediality, to reach a certain perception mass necessary for a conscious processing of information and effective information transfer.

Johansson, following the tradition of Gestalt Psychology, showed that our visual system is developed to handle continuously changing information rather than static pictures [Marmolin, 1991]. This pleads in favour of the inclusion of time-based data, such as motion pictures. [Faber et al., 1991] conducted several experiments to answer the question under which conditions information types combinations including motion pictures, have substantial advantages in comparison to information types combinations without motion pictures. They hypothesised that motion pictures are superior for learning conditions in which spatial or temporal properties of motions must be learnt. With regard to criteria of efficiency, success and time required for successful learning, however, substantial superiority for motion picture presentations was found only with regard to learning a rather complex motion pattern, not for other learning tasks including simpler motion patterns.

In many studies no significant results are found when comparing MAI with conventional courses. However, in support of the learning, effective information transfer, hypothesis are the results of a study of [Nicolson et al., 1991]. Nicolson et al. studied the effectiveness of multimedia courseware for dyslectic children. The children were found to improve their spelling skills significantly by using the multimedia courseware. Alas, the experimental design of this study is weak as it did not include control groups. So, it is not possible to isolate experimentally the independent variables that were responsible for the learning effect. It would have been of interest to make comparisons with traditional courses for dyslectics.

With regard to user satisfaction, ease vs. strain, enjoyability vs. boredom, Faber et al. found that learning by motion pictures is significantly easier and more enjoyable than learning without motion pictures. [Nicolson et al., 1990] also found that their subjects enjoyed multimedia courseware. These findings support the idea that multimedia leads to positive arousal, which contributes positively to learning attitude and motivation; stimulating learning motivation is often an objective in teaching situations. This idea is further supported by the observation that multimedia games, including audio and video, are eliciting a more enthusiastic responses from players, than non-multimedia predecessors.

A finding of [Gale, 1990] in support of the positive arousal (fun) assumption is that audio-visual communication was socially appreciated as positive, 'personal' and 'informal'.

Yet, multimedia is not 'ideal' in many cases. Incorrect use of the multimedia elements can:

• result in negative cognitive side effects (e.g., overstimulation, cognitive overload, distraction, fatigue [Heller, 1990] and

• thus reduce the effectiveness and efficiency of information and knowledge transfer.

As Gibson argues [Marmolin, 1991]: we do not hear, we listen; we do not see, we look around. This means that we are actively exploring our environments. The zapping behaviour of TV 'couch-potatoes' is a current expression of our need for perceptual exploration instead of just passively watching a TV program. One of the visions of multimedia, based on this argument, is that the user of a multimedia system should be enabled to explore the natural multimedia information in an active way. Neither the author of the information nor the designer should decide how the information should be processed, the user should be in control. A high level of interactivity is not new, but present in many computer systems. A high level of interactivity is, however, new for audio and video systems, for example the TV.

I operationalise active exploration of the computer environment in terms of the level of man-machine interactivity. A low level of man-machine interactivity, e.g., only switch a presentation on or off, does not really support our explorative behaviour and will lead to less involvement than systems with a high level of man-machine interactivity, e.g., influence on the course of displayed events, manipulation of objects, and editing content data.

[Schadé, 1993] states that multimedia improves sensory stimulation, particularly due to the inclusion of interactivity. For example, he estimates that reading stimulates about 1% of the sensory capacity of the eye. TV watching stimulates about 25-30% of the eye. If depth (3D images) and interaction with visual objects is added 60% or more of our eye is stimulated. The interactive and gaming elements in addition stimulate motoric behaviour and thus the brain even more, e.g., for making choices or solving problems. The role of sensory stimulation for learning is unclear, but it is assumed that a high level of sensory stimulation facilitates learning more than a low level.

1. The level of congruence is the degree to which different information types are used redundantly to express the same ideas. The basic assumption is that a high level of congruence of information types is far more effective than a low level of congruence for recognition and extending the mental reference models necessary for effective information transfer (learning).

1. [Marmolin, 1991] speaks of the redundant use of information sources. For example, the redundant use of colours has two effects. It facilitates the pick up and processing of information and it results in a more stimulating environment. This applies to the redundant use of sound. A study by Graver using sounding interfaces, where each event is characterised both visually and auditorily supports such a hypothesis [Marmolin, 1991].

1. Disruption is one of the negative cognitive side effects multimedia systems may have if information types are used incongruently. In their review [Hapeshi & Jones, 1992] describe a number of studies that have demonstrated the attention grabbing, sometimes disrupting effect of audio, background speech, noise. Although sounds are generally thought to be useful to deliver warnings and for context switches, [Alty et al., 1993] found opposite results. They found in their experiment that sounds have a detrimental effect on performance. It can be argued that the explanation for this is probably that the sounds were used incongruently with the other information, and thus led to disruptions.

1. With regard to the congruent use of synchronised video and audio Hapeshi & Jones remark that the presence of moving images can serve to enhance comprehension and learning of spoken material. As an example of this, they described an experiment of Hayes, Kelly & Mandel comparing the effectiveness of TV presentations to that of radio presentations of narrative information. Generally, the inclusion, during recall, of inaccurate story content and the distortion of actual story details occurred more often in the auditory only condition than in the aural and visual condition. Hapeshi & Jones further describe studies showing that the presence of an incongruent video presentation significantly reduces recognition memory of audio material, but that showing a congruent visual map results in better recall, particularly if the narrative structure is relatively simple. In the case of a monologue or a dialogue, visual display can facilitate processing of the auditory message if the speaker's face can be seen, because facial expressions, particularly lip movements enhance speech intelligibility. Visual display of text can also enhance speech intelligibility, e.g., to recognise the lyrics of a song. Another example is a study described by Hapeshi & Jones that shows that hearing a colour word improves the naming of a colour patch if the auditory word is congruent with the ink colour, i.e., hearing the word "red" when required to name aloud a red patch.

1. All of this suggests, according to Hapeshi & Jones, that when the visual and auditory channels provide congruent messages, processing is easier.

1. I postulate that an adequate usage of reference models in presented information stimulates recognition and transfer of information to people. A mental reference model is the meaningful organisation of information in our brains. Adequate usage of reference models means that we use, for example, meaningful sounds, pictures and movements to express ideas.

1. [Schadé, 1993] argues that if reference 'pictures' are added to text, people pick up and understand a story about 75% faster than if they are confronted with a text only story. Schadé hypothesises that by using reference models, innate or acquired in our early childhood, text and picture stories are stored faster and more efficiently in our long term memory than text-only stories. He states that people tend to remember 25-35% more of a text and pictures story than of a text only story.

1. The use of reference models is, for example, a well accepted marketing practice to use basic reference pictures and sounds to improve advertisements and commercials. Often family scenes, status symbols, attractive women and responsible men etc. are woven into the marketing material.

1. For these reasons, it seems reasonable to assume that adequate use of reference models is an important variable with regard to effective information transfer.

1. The quality of the representation of information in a multimedia system seems to be an important variable. A more realistic representation of information leads to more natural arousal, e.g., a realistic surgery film can invoke awkward feelings and audio-visual computer games are more fun than character oriented games, more involvement, e.g., aesthetic representations are used to attract attention, and better recognition. The recognition effect is illustrated by the study of [Brooks et al., 1991]. Brooks et al. examined, in an experimental study, whether additional information types added to a tutoring system would enhance effectiveness as measured by speed and accuracy of student performance. They also examined student opinions on the suitability of the system and its information types. Students received a tutorial incorporating either a) text and graphics, b) text, graphics and sound or c) text, photographs and sound. Significant differences were found between the test groups under conditions b and c. Group c performed better on a 'flower recognition' task and an 'object construction puzzle' task. For these tasks it is concluded that photographic representations were superior to simple graphic representations. We can conclude from this that a high quality of a (graphic) representation has a value added if the reality value of represented information is important for the performance of a task.

1. The importance of the quality of information representation is also stressed by a number of classic experiments [Nielsen, 1990] with regard to reading efficiency in relation to text representation on paper and on screen. These comparative experiments indicate that subjects read 25-30% slower from computer screens than from paper and that these subjects also have significantly higher error rates. [Nielsen, 1990] reviews a comparative study of Wilkinson and Robinshaw regarding error frequencies of subjects in two test conditions: proof-reading from screen and proof-reading from paper. During the first ten minutes of the experiment, subjects had about the same error rates in the two conditions (25% vs. 22%), but after they had been proof-reading for 50 minutes, the subjects using computer screens did significantly worse with an error rate of 39% vs. 25% for paper! Nielsen concluded that this experiment shows that users become fatigued fairly quickly when reading from the current generation of computer screens.

1. It is only possible to achieve the same reading speed when the computer screen was high-resolution and used anti-aliased proportional fonts. A side remark is that one can conclude from this that one should be careful with just replacing text books by electronic text books. In the case where one is using electronic text books using multimedia is probably necessary to compensate for the loss of readability of text information using the current generation of computer screens!

Is multimedia effective for instruction or is it just hype? On the basis of the consistent responses of experts and other subjects in the field, it can be concluded that multimedia is perceived consistently to be effective for MAI [Hoogeveen, 1994]. These perceptions are consistent with the multimedia paradigm that adding multimedia to ISs leads to an improved information and knowledge transfer to people. On the basis of these perceptions alone we can conclude that the idea that a multimedia paradigm exists is supported, however, we cannot conclude with certainty that multimedia really is effective. In fact, we can easily falsify the multimedia paradigm, the multimedia paradigm is too optimistic: if multimedia is used wrongly, incongruently etc., it certainly has a negative value added, it certainly does not lead to improved information and knowledge transfer! Thus, a paradigm shift is needed since the current multimedia paradigm is in many cases clearly invalid. In what direction should we look for a new multimedia paradigm?

If we just look at the positive, general attitude with respect to multimedia, we can interpret this positive attitude towards multimedia in four alternative ways, leading to four alternative learning paradigms:

1. multimedia is believed to be effective for information and knowledge transfer, but this is definitely not true, multimedia has hardly any effect at all, it is only a hype;

2. multimedia is believed to be effective for information and knowledge transfer, but this is hardly true, any positive learning effects are based solely on the positive attitude with regard to multimedia, it is a self-fulfilling prophecy;

3. multimedia is believed be effective for information and knowledge transfer, but this is only partially true: it is more than just a self-fulfilling prophecy, it makes learning fun and thus motivates people to learn, but nothing more;

4. multimedia is believed be effective for information and knowledge transfer, and this is true, because it can, if well applied, account also for an improved information and knowledge transfer to people.

Let us look closer at these four alternative interpretations. If we follow the first interpretation sooner or later the multimedia hype will pass, as the results of MAI projects fail completely to meet the high expectancies. If we follow the second interpretation, the minimum value added of multimedia is based on the paradigmatic belief that it is effective, which results in test leader effects or placebo effects. If we follow the third and fourth interpretations, multimedia is no hype at all, it offers real measurable benefits in learning situations; the value added of multimedia, if multimedia is well applied, is based on significant improvements in information and knowledge transfer to people, suppliers and customers and/or on its entertainment value has a positive effect on learning motivation.

Can we falsify some of these four statements on the basis of data gathered? The first two statements, that multimedia is definitely not effective and that it is only a self-fulfilling prophecy, can be falsified on the basis of:

• consistent research indicating that people experience multimedia as fun, as enjoyable, and that multimedia makes it more exciting to learn;

• the observation that multimedia games with audio and video are experienced as more attractive to players, than non-multimedia games.

This brings us to statement three, which is more hard too falsify beyond any doubt, although some results point in the direction of statement four:

• learning experiments made clear that adding multimedia elements - if the information types are used congruently, an adequate use is made of reference models, a high level of interactivity is offered, and a sufficient quality is used for information representation - improves information and knowledge transfer to people in some learning situations, but not always(!);

• multimedia information, presented congruently, is better recalled than monomedia information;

• tasks with an object recognition component are performed better when adequate visual representations are used instead of only descriptions or vague pictures.

Further, indirect evidence for the effectiveness of multimedia is based on the fact that some MAI systems are evaluated as effective in terms of meeting learning objectives [Hoogeveen, 1994].

[Alty et al., 1993] Alty, J. L., Bergan, M., Craufurd, P., & Dolphin, C. (1993). Experiments using Multimedia Interfaces in Process Control: Some Initial Results. Comput. & Graphics, 17(3), 205-218.

[Brooks et al., 1991] Brooks, P., Schmeling, A. &, Byerley, P. F. (1991). Multimedia intelligent tutoring of human-computer interaction procedures: An experimental evaluation. In H.-J. Bullinger (Ed.). Human Aspects in Computing: Design and Use of Interactive Systems and Information Management. Amsterdam, the Netherlands: Elsevier Science Publishers (pp. 959-963).

[Faber et al., 1991] Faber, J., Meiers, T., Ruschin, D., & Seyferth, A. (1991). The Motion Picture in Interactive Information Systems: A Necessary or Facilitating Component? In Bullinger, H.-J. (Ed.). Human Aspects in Computing: Design and Use of Interactive Systems and Work with Terminals. (pp. 485-490). Amsterdam, the Netherlands: Elsevier Science Publishers B.V.

[Gale, 1990] Gale, S. (1990). Human aspects of interactive multimedia communication. Interacting with Computers, 2(2), 175-189.

[Hapeshi & Jones, 1992] Hapeshi, K., & Jones, D. (1992). Interactive Multimedia for Instruction: A Cognitive Analysis of the Role of Audition and Vision. International Journal of Human-Computer Interaction, 4(1), 79-99.

[Heller, 1990] Heller, R. S. (1990). The Role of Hypermedia in Education: A Look at the Research Issues. Journal of Research on Computing in Education, 431-441.

[Hoogeveen, 1994] Hoogeveen, M. J. (1994). The Viability of Multimedia Retrieval Systems for Marketing and Sales. Doctoral dissertation. Leidschendam, the Netherlands: PTT Research.

[Marmolin, 1991] Marmolin, H. (1991). Multimedia from the Perspectives of Psychology. In Kjelldahl, L. (Ed.). Multimedia. Systems, Interactions and Applications. 1st Eurographics Workshop, Stockholm, Sweden April 18-19, 1991. (pp. 301-315). Berlin, Germany: Springer-Verlag.

[Nicolson et al., 1991] Nicolson, R. I., Pickering, S., & Fawcett, A. J. (1991). A Hypercard Spelling Support Environment for Dyslexic Children. Computers Educ., 16(2), 203-209.

[Nielsen, 1990] Nielsen, J. (1990). Evaluating Hypertext Usability. In D. H. Jonassen, & H. Mandl (Eds.). Designing Hypermedia for Learning (pp 147-168). Berlin, Germany: Springer-Verlag.

[Schadé, 1993] Schadé, J. P. (September, 1993). Ons brein en CD-i [Our brain and CD-i]. Janssen Medisch-Wetenschappelijk Nieuws, 276-282.

Acknowledgements: