Ph.D thesis "The Viability of Multimedia Retrieval Systems for Marketing and Sales" by prof dr M. J. Hoogeveen  

Chapter 4 Hypotheses about the Viability of Multimedia Retrieval Systems for Marketing and Sales

4.1. Introduction

The central question in this chapter is: what hypotheses can be formulated about the viability of Multimedia Retrieval Systems (MRSs) for Marketing & Sales (M&S)? Viability is defined here as the state of being able to survive under business conditions. As argued in the introduction, the hypotheses should manifest the ideas, convictions and experiences that Multimedia Retrieval (MR) developers share with regard to the viability, and the variables influencing the viability, of MRSs for M&S. Such a set of related hypotheses is called a theory. As discussed in chapter 1, the term theory is used in the sense of 'grounded theory', and the value of such a theory is that:

• it gives insight into the viability and the factors influencing the viability of MRSs for M&S;

• it makes the hypotheses visible and thus testable;

• its constituent set of hypotheses (insofar confirmed) can be used cautiously as guiding-principles when developing MRSs for M&S.

The experiences with MRSs for M&S, presented in the previous chapter, are used here, together with a survey of business objectives and success/risk factors, to give a more systematic overview of factors influencing the viability of these systems.

In the following sections, I start by explaining the symbols used to represent the hypotheses graphically and discuss my theoretic viewpoints, next I discuss the assumptions and outline of the theory, and in the following sections the outline of this theory is used to elaborate and group hypotheses about the value added of MR, success/risk factors, and MRS effectiveness in terms of meeting business objectives. Finally, as an upbeat to chapter 5 on testing hypotheses, the operationalisation of the dependent variables is discussed and the hypotheses are summarised.

4.2. Explaining the symbols used

Before formulating hypotheses as part of a theory it is necessary to explain the meaning of the symbols used in the simplified graphic representations of the hypotheses and theories.

Figure 32 gives an overview of the graphic elements of a hypothesis or theory. A hypothesis or theory can be represented by variables, theoretic constructs, and the relationships between these variables. The relationships, represented graphically by an arrow, indicate which variables influence or cause effects on other variables. A relation value indicates whether the relationship is positive or negative. Eventually a correlation coefficient can be used to indicate the strength of the relationship.

  Figure 32. Graphic elements of a hypothesis or theory. A variable is an independent variable in a relationship between variables if it is the influencing variable. Independent variables are found at the start of an arrow. A variable is a dependent variable in a relationship between variables if it is the influenced variable. Dependent variables are pointed at by an arrow. (Variables can be simultaneously independent and dependent since variables are always part of a variable chain).

  Figure 33. A general, graphic representation of a hypothesis or simple theory and an example.

In the example given in figure 33, a hypothesis or simple theory is shown that states that the introduction of multimedia (MM) tele-ordering has a positive effect on sales volume.

4.3. Theoretic viewpoints

Before discussing the theory and hypotheses related to the viability issue, I first want to clarify my theoretic viewpoints. This is done on the basis of the three dimensions of theoretical structure proposed by Markus & Robey (1988): beliefs with regard to causal agency, logical formulation of the theoretical argument and level of analysis.

Markus & Robey, in their discussion of theories on IT and organisational change, notice three beliefs with regard to causal agency: external forces cause change (the technological imperative), people act purposeful to accomplish intended objectives (the organisational imperative) or change emerges from the interaction of people and events (the emergent perspective). The emergent perspective holds that the uses and consequences of information technology (IT) emerge unpredictably from complex social interactions.

I agree with these three beliefs to some degree. First, the technological imperative is reflected in the assumption that adding MR functionality leads to changes in the effectiveness of M&S Support Systems. Second, the organisational imperative is reflected in my belief that people act purposeful during a system development project to deliver a system that is effective in terms of meeting business objectives. Third, the emergent perspective is reflected in my belief that a system with a certain level of effectiveness emerges from the interaction between several variables in a system development process, the outcome of which is not really predictable.

Markus & Robey's (1988) second dimension of theoretical structure concerns the logical formulation of the theoretical argument. They adopt Mohr's distinction between variance and process theories. In a variance theory the precursor (the antecedent, independent variable or 'cause') is posited as a necessary and sufficient condition for the outcome (the effect, the dependent variable). In other words, a causal relationship between the precursor and the outcome is assumed in a variance theory. In process theories, however, the precursor is assumed to be insufficient to 'cause' the outcome, but is held to be merely necessary for it to occur, i.e., only a concomitant relationship between precursor and outcome is assumed. The theory presented further on must be viewed as a process theory, as it is based on the observation that no guarantee can be given that an MRS will be viable, however 'ideal' the MR technology that is used seems and however 'ideal' the system development project is performed that results in the MRS, i.e., the precursor is not a sufficient condition.

Markus & Robey's (1988) third dimension of theoretical structure is related to level of analysis. They distinguish between macro level, society, interorganisational; micro level, individual; and mixed level analysis. Sol (1992) also gives the meso level, organisational. My theoretic viewpoint is most akin to the mixed level of analysis since I assume dynamic interplay between individuals, technology, and larger social structures to explain the value added of MR in relation to the viability of MRSs for M&S.

4.4. Basic assumptions and outline of the theory

The basic assumptions and outline of the theory, used in the following sections to formulate and group hypotheses, are discussed in this section. The structure of the theory presented can probably be generalised to other application domains, but this issue is not discussed here.

As already mentioned I adopt a process theory viewpoint: it is my firm belief that, as is the case with all system development projects, implementing an MRS for M&S can easily be a failure, but it requires tremendous effort, creativity, and knowledge to make it a success. Thus, in no way will implementing MRSs lead to guaranteed successes. There do, however, seem to be a number of variables that appear to influence the viability of MRSs for M&S. The relationships between these variables are given in figure 34.

  Figure 34. Graphic outline of the theory of the viability of MRSs for M&S.

The viability of MRSs for M&S depends directly on the perceived effectiveness for the parties involved of an MRS for M&S. The viability depends indirectly on the actual effectiveness of the MRS for M&S in terms of meeting M&S business objectives. The effectiveness of the MRS, and thus indirectly the viability of the MRS, depends on the way with which one is able to benefit from added MR functionality, and to cope with typical success/risk factors.

A number of assumptions can be found in these statements, these are formulated more precisely and reviewed in some more detail below.

A1. The degree to which MRSs for M&S are perceived as effective, perceived system effectiveness, by involved parties has a positive relationship with and direct influence on the viability, the state of being able to survive under business conditions, of these systems.

 

The perceived system effectiveness is the decisive variable with regard to the viability of MRSs. After all, perceptions about the effectiveness of an MRS in meeting its business objectives, determine whether a principal decides to proceed with the system development process or not.

A2. Perceived system effectiveness is positively related to actual system effectiveness, i.e., the degree to which an MRS can meet its M&S business objectives. The relationship is bi-directional: positive perceptions about system effectiveness influence actual effectiveness, and vice versa, observations and measurements of a system's actual effectiveness influence the perceived system effectiveness.

 

In other words, system viability depends indirectly on actual system effectiveness. In fact, we can observe that there is in most cases a large gap between perceived and actual system effectiveness. Even if system effectiveness measurements or calculations are performed this gap is still present, though perhaps smaller; it often is the qualitative interpretation and weighing of costs and benefits that forms the basis of perceptions about system effectiveness.

On the basis of the first two assumptions it can be hypothesised that an MRS for M&S is probably viable when a system meets significant business objectives effectively and is perceived as such by the involved parties. A number of observations support this hypothesis.

Decisions on implementing and accepting an MRS for M&S are often not based on structured and formal effectiveness measurements (see chapter 3), but are instead based on informal and unstructured observations. Often the enthusiastic responses of involved people are highly responsible for the acceptance of a system, and negative perceptions can be a tremendous and decisive threat to system viability. Disincentives for structured and formal effectiveness measurements tests are: lack of knowledge with regard to setting up tests, the cost and effort of setting up tests, justified doubts with regard to the validity and reliability of quasi-experimental tests, and even sometimes avoiding bad test outcomes.

Complicately for effectiveness measurement, is the fact that business processes are very complex, multifactor, socio-economic processes, which are as dynamic and little predictable as other socio-economic processes. It is very hard to isolate the effects of a specific independent variable in complex socio-economic business environments, business systems are in fact open systems. So, the outcomes of formal effectiveness measurements of business systems should, in most cases, be seen as effectiveness indications.

A3. Further, it is assumed that the presence or absence of success/risk factors influences system effectiveness and perceived system effectiveness and thus system viability, and that project management success/risk factors and system success/risk can be distinguished.

 

The better success/risk factors are dealt with during a system development project, the higher the quality of the project, the better the resulting system effectiveness, the more optimistic are perceptions, and the higher the potential system viability. System success/risk factors are aspects of an MRS that are critical for effectiveness and thus the viability of the system. Project management success/risk factors are contextual factors that influence the success with which an MRS development project is carried out.

Two examples can be given based on the summarised experiences in chapter 3. First, an example of a system success/risk factor: if an MM Business Catalogue (MBC) is not equipped with an easy to use user interface, sales people and customers will probably avoid using the MBC. As a consequence the system will not be effective as a sales support tool and will not prove viable. Second, an example of a project management success/risk factor: if there is little familiarity with MM Technology (MT) within an organisation it is necessary to hire expertise to cope with specific MT problems and opportunities; if insufficient expertise is hired all kinds of design mistakes and technical problems may result.

4.5. Hypotheses about the value added of MR functionality
 

In literature and MR projects one can recognise a set of convictions, based on observations and experimental findings, with regard to this value added which is described below in the form of a combination of two paradigms: the multimedia paradigm and the retrieval paradigm. The word 'paradigm' stresses the aspect of widespread, underlying convictions which often take the form of a firm belief, rather than just a set of testable hypotheses. This is particularly the case with multimedia.

4.5.1. Multimedia paradigm

The multimedia paradigm is the dominant conviction that adding MM functionality to information systems (ISs) leads to improved information and knowledge transfer to people. In other words, the MM paradigm is about the vision of computers as effective "tools for the mind" (Marmolin, 1991). Rich communication (multimediality), a high level of interactivity, a high level of congruence of used information types, an adequate usage of reference models (e.g., by data visualisation), and an adequate quality of represented information are presumed to contribute, via a number of interactions, to the improved information and knowledge transfer (see figure 35).

Figure 35. Cognitive model for the value added of MM.

 

The interaction of independent MM variables, depicted at the left side of figure 35, leads to psychological responses:

• a high level of stimulation of the senses, at least with regard to the auditory and visual perception systems;

• a high level of involvement, attention, concentration;

• emotional arousal, e.g., fun;

• strong recognition effects, using mental reference models.

These psychological responses interact in a complex way so that they give people the feeling they experience information instead of acquire it. The impact of MM messages is assumed to be high in comparison to non-multimedia messages by conventional marketing media and ISs, and thus lead to a hypothesised improved transfer of information and knowledge to people.

Yet, MM is not 'ideal' in many cases. Incorrect use of the MM elements can:

• result in negative cognitive side effects (e.g., overstimulation, cognitive overload, distraction, fatigue (Heller, 1990)) and

• thus reduce the effectiveness and efficiency of MM information and knowledge transfer.

It is believed that if one carefully avoids these negative cognitive side effects, MM improves:

• learning (retention, understanding, knowledge acquisition);

• the user friendliness of user interfaces and thus man-machine interaction;

• the entertainment value of systems (i.e., more fun);

• the impact of (marketing) messages, e.g., during business presentations or commercials.

M&S is believed to profit from MM Systems (MSs) for these reasons. Improved learning is important in all situations in which M&S or their customers need to learn something. Improved retention of mark and product names is an important marketing issue, the degree of user-friendliness of information services offered to a private or business customer is an important quality of service issue for M&S organisations, and the entertainment value of systems is an important marketing issue for products for the home entertainment market. Multimedia Promotion Systems are often also enriched with entertainment elements to attract and hold the attention of users. The impact of marketing messages is also a clear marketing issue.

The hypothesised value added of MM for marketing can also be illustrated by mapping the effects of MM on Burnett's (1993) five steps in human information processing related to marketing, exposure, attention, perception, retention, and retrieval and application. MM exposure is a stimulus hypothesised to draw more attention, as it is fun, moves, and responds interactively to our behaviour and, as a consequence, has more perceptual impact than more conventional media. Next, MM is hypothesised to improve the retention of marketing messages. With regard to the retrieval of marketing messages and application in terms of purchasing behaviour, it is also expected that MSs have a positive effect.
 

To conclude, it seems reasonable to hypothesise that:

H1. MM has value added for M&S in situations where effective information and knowledge transfer is needed.

4.5.2. Support of the MM paradigm

In support of the MM paradigm are notions from Perceptual Psychology. Perception psychologist Gibson (MacNab, 1987; Marmolin, 1991) argues that our senses are constructed to handle the very complex flow of information in natural environments, and that our senses are not constructed to handle simple stimuli. Gibson argues further that we are not passive receivers of information. Instead, our perceptual system is characterised by the pick up of information and by the integration of activities in the different senses. Although these argument are directed at the experimental study of human perception they are also relevant to MM computing: ISs that need to support human information processing very effectively, should make full use of human perceptual and cognitive capabilities and therefore should represent natural information flows to users and offer support to process natural information flows. According to Marmolin (1991) complex, dynamic and integrated representations of information really are necessary to utilise all the capabilities of the mind.

Schadé (1993) puts forward a neuropsychological explanation for the relationship between MM and learning. Schadé argues that our brains learn most from MM information flows. A story in pictures is picked up much faster and uses less memory capacity than a text story. According to modern research in cognition MM information makes better use of the brain by appealing to both the left and right hemispheres (see Marmolin, 1991; Schadé, 1993). Both language oriented representations and visual representations are necessary to utilise the cognitive capabilities of the mind.

A number of independent MM variables believed to have a psychological impact, are introduced in figure 35: the level of multimediality, the level of man-machine interactivity, the level of congruence of information types used, the usage of reference models and the quality of representation.

Level of multimediality

The level of multimediality can be operationalised in terms of the number of information types used. The assumptions expressed in figure 35 are that the higher the level of multimediality:

• the more stimulation of the senses;

• the more arousal;

• the higher the involvement;

• the stronger the recognition effects.

The idea is that with a high level of multimediality it is more easy, than with a low level of multimediality, to reach a certain 'perception mass' necessary for a conscious processing of information and effective information transfer.

A similar idea is expressed by the learning pyramid:

  text text + graphics text + graphics + audio text + graphics + audio + video

The idea is that the more information types are included in a course, the more effective the course will be. Many variations of such learning pyramids can be found in the literature, but the evidence is not given! Nevertheless, such pyramids are interesting to mention because they are in accordance with the multimedia paradigm formulated and they are often copied, hardly disputed, and often misused.

The same is the case for the effectiveness statements of the British Audio Visual Society, described by Hapeshi & Jones (1992). This society speculates that we remember:

10% of what we read

20% of what we hear

30% of what we see

50% of what we see and hear

80% of what we say

90% of what we say and do at the same time

The last two lines are beyond multimediality: they plead for interactivity, which is discussed in the following subsection.

Johansson, following the tradition of Gestalt Psychology, showed that our visual system is developed to handle continuously changing information rather than static pictures (Marmolin, 1991). This pleads in favour of the inclusion of time-based data, such as motion pictures. Faber et al. (1991) conducted several experiments to answer the question under which conditions information types combinations including motion pictures, have substantial advantages in comparison to information types combinations without motion pictures. They hypothesised that motion pictures are superior for learning conditions in which spatial or temporal properties of motions must be learnt. With regard to criteria of efficiency, success and time required for successful learning, however, substantial superiority for motion picture presentations was found only with regard to learning a rather complex motion pattern, not for other learning tasks including simpler motion patterns.

In support of the learning, effective information transfer, hypothesis are the results of a study of Nicolson et al. (1991). Nicolson et al. studied the effectiveness of MM courseware for dyslectic children. The children were found to improve their spelling skills significantly by using the MM courseware. Alas, the experimental design of this study is weak as it did not include control groups. So, it is not possible to isolate experimentally the independent variables that were responsible for the learning effect. It would have been of interest to make comparisons with traditional courses for dyslectics.

With regard to user satisfaction, ease vs. strain, enjoyability vs. boredom, Faber et al. found that learning by motion pictures is significantly easier and more enjoyable than learning without motion pictures. Nicolson et al. (1990) also found that their subjects enjoyed multimedia courseware. These findings support the idea that MM leads to positive arousal, which contributes positively to learning attitude and motivation; stimulating learning motivation is often an objective in teaching situations. This idea is further supported by the observation that MM games, including audio and video, are eliciting a more enthusiastic responses from players, than non-MM predecessors.

The value added of MM in terms of office task performance is not incontroversial. In a study Gale (1990) tried to determine the value added to an office system of incorporating audio and video. His study did not show any significant differences between group performance on an information dissemination task, a creative co-operation task and a meeting scheduling task under three conditions: data sharing only, data sharing plus audio, and data sharing plus audio and video. It should be noted, however, that these tasks did not have an audio-visual component. So, we can argue that adding audio and video for these performed tasks is luxury, not a necessity. A finding of Gale in support of the positive arousal (fun) assumption (see figure 35) is that the audio-visual communication was socially appreciated as positive, 'personal' and 'informal'.

So, in contrast to the MM paradigm it can be observed that a higher level of multimediality alone is not sufficient for a better task performance, only for some learning tasks is an effective information transfer noted. Results indicating that MM is fun and enjoyable are more consistent.

Level of man-machine interactivity

As Gibson argues (Marmolin, 1991): we do not hear, we listen; we do not see, we look around. This means that we are actively exploring our environments. The zapping behaviour of TV 'couch-potatoes' is a current expression of our need for perceptual exploration instead of just passively watching a TV program. One of the visions of MM, based on this argument, is that the user of an MS should be enabled to explore the natural MM information in an active way. Neither the author of the information nor the designer should decide how the information should be processed, the user should be in control.

I operationalise active exploration of the computer environment in terms of the level of man-machine interactivity. A low level of man-machine interactivity, e.g., only switch a presentation on or off, does not really support our explorative behaviour and will lead to less involvement than systems with a high level of man-machine interactivity, e.g., influence on the course of displayed events, manipulation of objects, and editing content data.

Schadé (1993) states that MM improves sensory stimulation, particularly due to the inclusion of interactivity. For example, he estimates that reading stimulates about 1% of the sensory capacity of the eye. TV watching stimulates about 25-30% of the eye. If depth (3D images) and interaction with visual objects is added 60% or more of our eye is stimulated. The interactive and gaming elements in addition stimulate motoric behaviour and thus the brain even more, e.g., for making choices or solving problems. The role of sensory stimulation for learning is unclear, but it is assumed by Schadé that a high level of sensory stimulation facilitates learning more than a low level.

Considering these arguments it seems reasonable to assume that the level of interactivity of an MS is an important variable with regard to the effectiveness of information transfer. Yet, we should be careful about drawing further conclusions since the validity of these arguments is unclear.

Level of congruence

The level of congruence is the degree to which different information types are used redundantly to express the same ideas. The basic assumption is that a high level of congruence of information types is far more effective than a low level of congruence for recognition and extending the mental reference models necessary for effective information transfer (learning).

Marmolin (1991) speaks of the redundant use of information sources. For example, the redundant use of colours has two effects. It facilitates the pick up and processing of information and it results in a more stimulating environment. This applies to the redundant use of sound. A study by Graver using sounding interfaces, where each event is characterised both visually and auditorily supports such a hypothesis (Marmolin, 1991).

Disruption is one of the negative cognitive side effects MSs may have if information types are used incongruently. In their review Hapeshi & Jones (1992) describe a number of studies that have demonstrated the attention grabbing, sometimes disrupting effect of audio, background speech, noise. Although sounds are generally thought to be useful to deliver warnings and for context switches, Alty et al. (1993) found opposite results. In their experiment subjects performed a process control task called 'the Crosman Waterbath' tasks. The Crosman Waterbath task involves the control of a simulated thermal hydraulic system that consists of a single tank. A valve on inflow and outflow pipes may be used to regulate the inflow and outflow. A heater, situated immediately underneath the tank, is the third control variable. In some experimental conditions sounds were used as warning signals, in others not. Alty et al. found in their experiment that such sounds have a detrimental effect on performance. It can be argued that the explanation for this is probably that the sounds were used incongruently with the other information, and thus led to disruptions.

With regard to the congruent use of synchronised video and audio Hapeshi & Jones remark that the presence of moving images can serve to enhance comprehension and learning of spoken material. As an example of this, they described an experiment of Hayes, Kelly & Mandel comparing the effectiveness of TV presentations to that of radio presentations of narrative information. Generally, the inclusion, during recall, of inaccurate story content and the distortion of actual story details occurred more often in the auditory only condition than in the aural and visual condition. Hapeshi & Jones further describe studies showing that the presence of an incongruent video presentation significantly reduces recognition memory of audio material, but that showing a congruent visual map results in better recall, particularly if the narrative structure is relatively simple. In the case of a monologue or a dialogue, visual display can facilitate processing of the auditory message if the speaker's face can be seen, because facial expressions, particularly lip movements enhance speech intelligibility. Visual display of text can also enhance speech intelligibility, e.g., to recognise the lyrics of a song. Another example is a study described by Hapeshi & Jones that shows that hearing a colour word improves the naming of a colour patch if the auditory word is congruent with the ink colour, i.e., hearing the word "red" when required to name aloud a red patch.

All of this suggests, according to Hapeshi & Jones, that when the visual and auditory channels provide congruent messages, processing is easier.

Adequate usage of reference models

I postulate that an adequate usage of reference models in presented information stimulates recognition and transfer of information to people. A mental reference model is the meaningful organisation of information in our brains. Adequate usage of reference models means that we use, for example, meaningful sounds, pictures and movements to express ideas.

Schadé (1993) argues that if reference 'pictures' are added to text, people pick up and understand a story about 75% faster than if they are confronted with a text only story. Schadé hypothesises that by using reference models, innate or acquired in our early childhood, text and picture stories are stored faster and more efficiently in our long term memory than text-only stories. He states that people tend to remember 25-35% more of a text and pictures story than of a text only story.

The use of reference models is a well accepted marketing practice to use basic reference pictures and sounds to improve advertisements and commercials. Often family scenes, status symbols, attractive women and responsible men etc. are weaved into the marketing material.

For these reasons, it seems reasonable to assume that adequate use of reference models is an important variable with regard to effective information transfer.

Quality of representation

The quality of the representation of information in an MS seems to be an important variable. A more realistic representation of information leads to more natural arousal (e.g., a realistic surgery film can invoke physical and mental distress and audio-visual computer games are more fun than character oriented games) more involvement (e.g., aesthetic representations are used to attract attention) and better recognition. The recognition effect is illustrated by the study of Brooks et al. (1991). Brooks et al. examined, in an experimental study, whether additional information types added to a tutoring system would enhance effectiveness as measured by speed and accuracy of student performance. They also examined student opinions on the suitability of the system and its information types. Students received a tutorial incorporating either a) text and graphics, b) text, graphics and sound or c) text, photographs and sound. Significant differences were found between the test groups under conditions b and c. Group c performed better on a 'flower recognition' task and an 'object construction puzzle' task. For these tasks it is concluded that photographic representations were superior to simple graphic representations. We can conclude from this that a high quality of a (graphic) representation has a value added if the reality value of represented information is important for the performance of a task.

The importance of the quality of representations is also stressed by a number of classic experiments (Nielsen, 1990) with regard to reading efficiency in relation to text representation on paper and on screen. These comparative experiments indicate that subjects read 25-30% slower from computer screens than from paper and that these subjects also have significantly higher error rates. It is only possible to achieve the same reading speed when the computer screen was high-resolution and used anti-aliased proportional fonts. A side remark is that one can conclude from this that one should be careful with just replacing text books by electronic text books. In the case where one is using electronic text books using MM is probably necessary to compensate for the loss of readability of text information using the current generation of computer screens!

In general, it can be assumed that quality of information representation is an important MM variable for tasks including object recognition and/or reading.

4.5.3. Retrieval paradigm

The retrieval paradigm is the dominant conviction that all tasks or processes which include search and database management (sub)tasks will benefit from electronic retrieval systems. In the case of MM tasks and processes one can read 'MRSs' instead of 'electronic retrieval systems'.

Little controversial is the idea that electronic retrieval mechanisms in comparison to manual retrieval improve:

• search efficiency (how much effort is needed to find what you want to find);

• search effectiveness (finding what you want to find in terms of precision and recall; see chapter 2);

• database management.

In addition, MR is believed to

• effectively support exploration using associative hyperlink mechanisms (e.g., explorative learning, exploring criminal relations, exploring ideas and structuring discussions);

• further improve search performance by a far better presentation of retrieved objects, e.g., in the form of Query By Visual browsing;

• improve database management by offering facilities to manage multimedia databases.

For example in the Tele Sales Assistant (TSA) system and MM Communication Archive (MCA) cases described in chapter 3 it is noted that Query By Visual browsing, in which a searcher selects or ignores retrieved database objects on the basis of associated pictures, is much more efficient and effective than query by text only interfaces (e.g., Query By Example).

When considering the effectiveness of retrieval systems it is important to take notice of the gaps that exist between:

• the actual information need and the formulation of that information need;

• the formulated information need and the formulated query;

• the formulated query and the result set.

The last two gaps are addressed by precision and recall surveys discussed in chapter 2.

In general, the use of computer systems improves search efficiency and search effectiveness, precision and recall, when searching through large databases in comparison to searching manually through databases.

It is a pity that the effectiveness of individual retrieval facilities in office situations is rarely addressed by research. On the basis of experiences with MRSs I hypothesise that:

• complex retrieval facilities, like inexact query languages, are useful only to specialist information workers;

• simple to use facilities are useful for all types of people working with an MRS on a regular basis;

• very simple, low threshold, facilities are useful for inexperienced, ad hoc, or handicapped users;

• MDBMSs are suited to be at the heart of most MRSs.

One of the retrieval facilities that is strongly investigated and has a strong theoretic basis with regard to its cognitive effects is hypertext and hypermedia. Hypertext representations may correspond with the mental associative knowledge representation in the mental reference models of a user. Within a hypertext document a user can trace associative references which correspond with associations with mental reference models in their memory. Discovering new hyperlinks stimulates the extension of these mental reference models. This is akin to Jonassen's (1989) concept of 'web learning', which assumes that information, when learned, is integrated with prior knowledge using a web structure rather than in a linear fashion.

Hyperlinking structures, however, are reported to have a number of negative cognitive side effects. The most important side effects are (Wolswijk & De Kleine-Noorbergen, 1992; Heller, 1990; Van der Meer, 1991):

• disorientation: it can happen that users become lost in a hypertext document;

• cognitive overload: too much choice overstrains the cognitive capacity of a user;

• inefficient searching: hypersearching is sometimes an inefficient way of searching purposefully.

Hypertext textbooks, i.e., textbooks in the form of hyperlinked text fragments, have not yet proven to be a more effective learning tool than sequential, conventional textbooks. One of the alternative explanations for this can be that the current generation of computer screens are less effective than paper as a presentation medium for text.

In conclusion, if we return to the retrieval paradigm discussed at the start of this section, the hypothesis on the impact of retrieval on M&S can be formulated as follows.

H2. Retrieval functionality has value added for M&S situations where search performance and database management performance are important.

4.6. Hypothesised success/risk factors

It can be derived from assumption A3 that to successfully conclude an MRS development project a number of success/risk factors must be coped with. Every success/risk factor alone is potentially sufficient to disqualify project results and threaten the viability of the MRS for M&S. As discussed, I see project management success/risk factors and system success/risk factors as clearly distinct classes of success/risk factors. In most cases these success/risk factors are also applicable to projects and information systems outside the scope of this thesis, but for MRSs for M&S certain important points of gravity can be made. For each and every MRS for M&S project, and for every type of MRS for M&S, a different set of success/risk factors is relevant to a different degree. It is interesting to hypothesise which project management and system success/risk factors are relevant for MRSs for M&S in general, though we must not see any list of such success/risk factors as fixed, as is the case with traditional IT critical success factors (Pollalis & Hanson Frieze, 1993).

In this section an overview is given of the two classes of success/risk factors hypothesised to be relevant. This overview is based on an n=19 survey of MM projects in which respondents were asked to give the business objectives relevant to their project (see section 4.7.), and the most relevant project management success/risk factors (this section) and system success/risk factors (this section). Most of the respondents were employees of PTT Telecom (7) and PTT Research (9), three came from organisations outside Royal PTT Nederland. Respondents working on projects discussed in chapter 3 were included in the survey to introduce experience with the MRSs under investigation. (See also appendix C). The objectives and success/risk factors mentioned were grouped on the basis of similarity. Next, the frequency with which mentioned objectives and success/risk factors were attributed to a group was scored, to give the frequency for which success/risk factor groups were relevant for the respondents (more than one group may be relevant for a respondent).

Project management success/risk factors

In decreasing order of frequency the groups of project management success/risk factors are:

• availability of multidisciplinary expertise (74%);

• insight into target group (47%);

• organisational changes (47%);

• sufficient preparation (37%);

• development and implementation costs (32%);

• time pressure (26%);

• co-operation with other parties/partners (26%);

• insight into business value added (26%);

• system integration problems (21%);

• marketing of project results (16%);

• management commitment (16%);

• training of personnel (11%);

• flexible ways of system development (11%);

• testing the system (5%).

 

Factors related to the availability of multidisciplinary expertise were mentioned most often. The reason for this is probably that MM is still relatively new and requires, as an integration technology, diverse expertises that has not needed to be combined before. If this is true multidisciplinary expertise will become a less important factor over time as MM expertises become more available in the market, and the complexity of MM technology is further reduced by new technological improvements.

The availability of specific MM expertise was seen as indispensable. For example, being at the mercy of an inexperienced MM studio was seen as a tremendous risk for developing sound CD-i's. Specialist expertise is also needed for the selection of adequate retrieval mechanisms, this is relatively scarce if one wants to make use of advanced retrieval facilities.

Factors related to insight into target group were mentioned by about half of the respondents. MM project leaders having a focus on conquering a certain market with an MM product, an MM title for example, in particular stressed the importance of insight into target groups.

With regard to organisational changes it is interesting to note that enthusiasm is believed to be one of the success factors for invoking organisational change, and that MM is believed to help to evoke the enthusiasm of involved project members, principals and user groups: MM is seductive.

In about one third of the projects factors with regard to development and implementation costs were mentioned: too limited budget, high costs of developing MM information (shooting film), and the cost of settling copyrights.

Processing MM information requires the seamless co-operation of multiple types of media, therefore, MRSs, supporting this, are the outcome of a system integration process; system integration problems are typical for MRS projects.

MR technology is evolving rapidly and, partly due to its newness, is often unstable. One effect is that during an MRS development process a number of undocumented bugs in software and hardware, and a number of interoperability problems can be encountered.

System success/risk factors

In decreasing order of frequency the mentioned groups of system success/risk factors are:

• integration with IT infrastructures (47%);

• ease of use (42%);

• flexible user support (37%);

• system and usage costs (32%);

• quality of information (32%);

• motivational system aspects (32%);

• image (21%);

• placing (11%);

• technical reliability (11%).

 

Integration with IT infrastructures is a topical issue related to MRSs. The first generation of MRSs was stand-alone. More and more projects aim at MRSs that are part of the corporate IT infrastructures. Over time, when integration with IT infrastructures becomes normal, this group of factors will probably cease to receive so much attention.

User-interface aspects or usability aspects were often mentioned (42%), which are related to the factor ease of use. Ease of use is of extreme importance for user acceptance and thus viability of a system. MM elements are often introduced to improve the ease of use, however, the use of multiple information types can also complicate the user-interface.

More than one third of the mentioned system factors are related to flexible user support. It is seen as important that MR technology introduces more functionality into ISs. This can improve the support of the different ways of working of different types of users, and human information processing. The flexibility of an MRS increases when it is able to support multiple types of users with multiple ways of working with multiple types of media.

High costs are a clear threat for the viability of any system, this was mentioned by one third of the respondents. MR end products for consumers need, especially, to be competitively priced, and affordable. The price of MR services depends heavily on the level of communication costs. These costs need to be reduced to make MR services affordable.

Factors related to quality of information were mentioned by about one third of the respondents, i.e., completeness and topicality of product databases, understandability of information presented to users, communication and presentation value of marketing information. Too poor quality of audio-visual data was seen, in one case, as a serious risk factor. In the case where instructions or an explanation of services is given to users the reality value of presented information is of extreme importance.

The motivational aspects of marketing media, and thus MRSs if they are used as a marketing medium, are of relevance for marketing . MM touches end users emotions by congruent appeal to cognitive reference models and thus may motivate towards purchasing behaviour. That MM is appealing is seen as a success factor.

Image aspects are mentioned by about 20% of the respondents. For example, in the case of the MM Promotion System (MPS; see chapter 3) the innovative image was seen as an important success factor.

Success/risk factors and MRSs for M&S

To what degree are the groups of factors, listed above, MM specific? I made a qualitative attempt to visualise differences with regard to the MRSs for M&S, presented in chapter 3, in the tables given below.

For example, the project management group of success/risk factors 'multidisciplinary expertise' is extremely relevant for MRSs where complex, multi-party MRSs are developed using relatively innovative, and therefore unstable, technology, e.g., a Virtual Market (VM) application; multidisciplinary expertise is also relevant for the other MRSs for M&S but not so highly relevant.

1 Highly irrelevant

2 Irrelevant

3 Sometimes relevant

4 Relevant

5 Highly relevant Table 6. Relevance estimates of project management variables for MRSs for M&S. The mean gives an indication of project difficulty.  

For example, the group of system success/risk factors 'integration with IT infrastructures', is highly relevant for the VM, in which many different applications and the many public and private information systems need to co-operate; integration with IT infrastructures is not always relevant for an MPS or MM Assisted Instruction (MAI), which need not to be networked, and is just relevant for the other MRSs.

1 Highly irrelevant

2 Irrelevant

3 Sometimes relevant

4 Relevant

5 Highly relevant Table 7. Relevance of system variables for MRSs for M&S. The mean gives an indication of system difficulty.

We can argue that if the relevance estimates of success/risk factors are relatively low the short term viability of a certain type of MRS is relatively high, and, vice versa, we can argue that if the relevance estimates of success/risk factors for a certain type of MRS are relatively high short term viability is relatively low. The latter is certainly the case for my estimations with regard to the VM, expressed in the high means. High investment, many uncertainties, technical complexity and large number of parties involved explains why teleservice providers are so cautious about setting up VM services.

On the basis of experiences with developing MRSs for M&S as described in the previous chapter and as incorporated in these estimations a hypothesis is formulated about the viability of MRSs for M&S.

H3. All the MRSs for M&S are viable except for the VM which will become viable in the intermediate or long term future.

 

With regard to the identified success/risk factors a fourth hypothesis is formulated, in which only MM specific success/risk factors are considered:

H4. The MM specific project management and system success/risk factors are critical for the viability of MRSs for M&S.

 

Some of the MM specific success/risk factors are: the availability of MM and MR expertise, the level of MM development costs, MR system integration bottlenecks, easy to use MM user interfaces, MM communication costs, quality of MM information, motivational effects using MM, and the innovative image of MM.

4.7. Business objectives hypothesised as relevant

As discussed, a basic hypothesis is that adding MR elements and facilities leads to an improved system effectiveness in terms of meeting business objectives. In this section an overview is given of business objectives hypothesised to be relevant for MRS for M&S based on the n=19 survey described in section 4.6.

Before discussing the grouped business objectives I want to stress that the grouping of business objectives is designed to highlight the business objectives that are seen as important for MRSs for M&S. Any grouping of business objectives should be seen as dynamic since business objectives vary with time, from case to case, from firm to firm, and from culture to culture.

Therefore I did not strive for a complete list, the sample was too small for that, however, the list appears to be representative in the sense that the most important business objectives are mentioned, although the grouping is somewhat arbitrary.

In decreasing order of frequency, the mentioned groups of business objectives are:

• improving a market position (84%);

• improving quality of service (63%);

• improving promotion (32%);

• efficiency gains (32%);

• improved productivity (26%);

• improved information and knowledge transfer (21%);

• increased management control (11%).

 

Since, MR services are in their infancy 'improving market position' is seen as a business objective in almost all MM projects surveyed; improving market position is the most often mentioned group; for example by exploring market opportunities for MM services or extending market positions in a growth market. Examples of entering a new market are the selling of exposure on MM Points of Sale and the publishing of CD-ROMs and CD-i's. As MSs become more common, the attention given to exploring market opportunities will probably decrease.

MRSs are sometimes seen as a weapon to defend an old market position by improving competitiveness, by delivering better services, and by lowering sales costs.

MM was seen as an opportunity to improve the quality of service by about two thirds of the respondents, for example, by offering around the clock services by touch screen interfaces on a shop-window at night or by user-friendly shopping systems at home. Another example is that the quality of customer support is believed to improve with the use of MM catalogues instead of dull paper catalogues.

Improving promotion is a typical MM business objective, mentioned by about one third of the respondents. Improving corporate image is a PR objective that was sometimes mentioned. MM means were seen as useful for improving corporate communication. MRSs are, in several cases, used (see also the previous chapter) to make a (brand) name, which means improving name retention. It is believed that manufacturers of consumer products will be better able to communicate their unique selling position using MRSs.

The objectives related to improving information and knowledge transfer, mentioned by about 20% of the respondents, are in compliance with the multimedia paradigm discussed. Examples of this are improved training effectiveness and the resulting improved knowledge level of sales personnel.

To what degree are the business objectives discussed above MM specific? Again, a qualitative attempt is made by me, in the table below, to visualise differences with regard to the MRSs for M&S presented in chapter 3.

For example, improving a market position is the main, and thus a highly relevant, objective of MPSs and for many VM applications; improving a market position is also relevant for the TSA, and MBC, but archives (MCA and Marketing Documentation Archive (MDA)) and training (MAI) are, from the viewpoint of an M&S firm, not primarily directed at improving a market position.

1 Highly irrelevant

2 Irrelevant

3 Sometimes relevant

4 Relevant

5 Highly relevant Table 8. Relevance of business objectives of MRSs for M&S. The mean gives an indication of ambition level in relation to objectives relevant to MM projects.

 

On the basis of the grouped M&S business objectives a number of hypotheses with regard to the effectiveness of MRSs for M&S can be derived, graphically depicted in the figure below. These hypotheses can be reformulated in one sentence:

H5. MRSs for M&S are (perceived to be) effective in terms of meeting M&S business objectives, which are related to one or more of the variables: market position, quality of service, promotion, M&S efficiency, M&S productivity, M&S information and knowledge transfer and management insight.

  Figure 36. Graphic overview of hypotheses with regard to the effectiveness of MRSs for M&S in terms of meeting business objectives.

 

With regard to the relevance estimates we can argue that a higher mean relevance estimate corresponds with a more ambitious MRS, this is clearly the case with the VM.

4.8. Operationalisation of the dependent variables

Three dependent variables are presented in figure 34, effectiveness, perceived effectiveness and viability of MRSs for M&S. The operationalisation of these variables is discussed here.

In their quest for the dependent variable to measure Information Systems success based on an extensive literature survey, DeLone & McLean (1992) classify the operationalisations (empirical measures) of IS success into six categories, which they relate to the classifications of Shannon and Weaver, and Mason. The first three categories are adopted by Delen et al. (1991) in their product quality framework. DeLone & McLean's success categories are categories of empirical measures of:

• system quality, i.e., measurements on a technical level according to Shannon and Weaver and production level according to Mason, like response time, flexibility of system, system reliability and system accessibility;

• information quality, i.e., measurements on a semantic level according to Shannon and Weaver and product level according to Mason, like accuracy, completeness, conciseness, relevance, timeliness, and understandability.

• information systems use, i.e., measurements on the receipt level according to Mason (Shannon and Weaver use the broader term effectiveness or influence level), like percentage of time an IS is used, frequency of use, number of functions used, use of numerical vs. non numerical data.

• user satisfaction, i.e., measurements on the recipient response to the use of the output of an IS (not distinguished by Shannon and Weaver from their effectiveness/influence level or by Mason from his influence on recipient level), like overall user satisfaction, user information satisfaction, enjoyment, top management satisfaction, software satisfaction, and hardware satisfaction.

• individual impact, i.e., measurements on the effect of information on the behaviour of the recipient (not distinguished by Shannon and Weaver or Mason), like time taken to complete a task, decision quality, time to decision, time to solve a problem, user understanding, quality of plans, and personal productivity.

• organisational impact, i.e., measurements on the effect of information on organisational performance (not distinguished by Shannon and Weaver or Mason), like profit performance, innovations, cost-benefit ratio, and Return On Investment.

 

In the outline of the theory, presented in figure 34, we can match the construct system effectiveness (in terms of meeting business objectives) with the individual and organisational impact categories of DeLone & McLean. We can relate user satisfaction to the perception construct, which appears to be an important predictor for system viability. Information systems use may be an operationalisation of system viability in terms of user acceptance. System viability can also be operationalised as past, current and future market acceptance. DeLone & McLean's system quality and information quality are integral elements of the system effectiveness construct insofar system quality and information quality is defined in terms of meeting certain business objectives, like quality standards, etc.

To conclude this section, we can operationalise the dependent variable viability of MRS for M&S as:

• information systems use: the more frequent, and intense and the longer an implementation of a system is used, the more viable it is;

• market acceptance: the more a system is (going to be) accepted, the more viable it is;

• technical feasibility: only then when a system can be implemented, it is potentially viable;

• expert perception of viability: experts perceptions about the viability of a system is an accepted measure of the actual viability, although not completely valid and reliable.

When we consider system effectiveness in terms of meeting business objectives and the perceptions about system effectiveness as important predictors for system viability, we can take these constructs as dependent variables. System effectiveness can be operationalised as:

• individual impact: i.e., the degree to which an MRS for M&S is effective in meeting business objectives in terms of tasks of individuals;

• organisational impact: i.e., the degree to which an MRS for M&S is effective in meeting business objectives in terms of organisational performance;

• system and information quality: the degree to which MM information and MR functionality have value added.

System effectiveness can be measured using IT evaluation techniques such as those discussed by Farbey et al. (1992). When applying an IT evaluation technique we should be conscious of its limitations, the validity of IT evaluation techniques is a matter of concern.

Perceived system effectiveness can be operationalised as:

• user satisfaction: the degree to which user groups in the broadest sense are satisfied with the implementation of an MRS;

• perceptions about meeting business objectives: the degree to which expert groups, managers with the power of decision, and other non-user groups, see the (implementations of) MRSs for M&S as effective for meeting certain previously defined business objectives.

• perceptions about the value added of the MRS: the degree to which it is believed by experts, managers etc. that the MRS has value added in term of meeting certain objectives not met before.

The latter two points are firmly related, and are merely a reformulation of each other.

4.9. Summary

An overview of the hypotheses with regard to the viability, and the factors influencing the viability, of MRSs for M&S is given in this chapter.

It is assumed that the viability of MRSs for M&S depends directly on the effectiveness perceived by involved parties of an MRS for M&S. It is assumed further that viability depends indirectly on the actual effectiveness of an MRS for M&S in terms of meeting M&S business objectives. Finally, it is assumed that the effectiveness of an MRS for M&S, and thus indirectly the viability of an MRS, depends on the way in which one is able to benefit from the added MR functionality, and one is able to cope with typical success/risk factors.

The value added of MR is explained by the MM paradigm, a cognitive model for the value added of MM, and the retrieval paradigm. Research findings tend to confirm that MM has a certain value added for information and knowledge transfer (learning) although negative cognitive side effects of MM and retrieval, e.g., hypertext, may inhibit positive learning effects. Research findings have further made clear that the MM paradigm is too optimistic: an MS is not a panacea; just adding MM functionality does not necessarily leads to better performance of office tasks.

Finally, a first step is made towards operationalising the dependent variables: system viability, system effectiveness and perceived system effectiveness.

The five main hypotheses are summarised below.

H1. MM has value added for M&S in situations where effective information and knowledge transfer is needed.

To realise this value added of MM, the information conveyed needs to make adequate use of interactivity, information types should be used congruently, the way the information is presented should be of a suitable quality and adequate use of reference models should be made.

The level of multimediality rises with the adding of non-textual information types, like stills, graphics, audio and video, to textual information.

H2. Retrieval functionality has value added for M&S situations where search performance and database management performance are important.

To realise search performance benefits, adequate use of retrieval facilities should be made. Complex retrieval interfaces can only be provided for retrieval specialists. Query By Visual browsing is probably an effective retrieval means for non-retrieval specialists.

H3. All MRSs for M&S are viable except for the VM which will become viable in the intermediate or long term future.

Viability aspects can be operationalised as information systems use, market acceptance, technical feasibility and expert perception of viability. With regard to information systems use, the frequency, intensity and duration of use are three viability dimensions.

H4. The MM specific project management and system success/risk factors are critical for the viability of MRSs for M&S.

Some of the MM specific success/risk factors are: availability of MM and MR expertise, level of MM development costs, MR system integration bottlenecks, easy to use MM user interfaces, MM communication costs, quality of MM information, motivational effects of using MM, and the innovative image of MM.

H5. MRSs for M&S are (perceived to be) effective in terms of meeting M&S business objectives, related to the variables: market position, quality of service, promotion, M&S efficiency, M&S productivity, M&S information and knowledge transfer and management insight.

Perceived system effectiveness is assumed (A1) to be positively related to system viability. Perceived system effectiveness can be operationalised as user satisfaction and perceptions about meeting business objectives. System effectiveness can be operationalised in terms of meeting M&S business objectives related to tasks of an individual, organisational processes and system and information quality. The effectiveness of MRSs for M&S can be measured using IT evaluation techniques.

© 1995-2002 Martijn Hoogeveen