Multimodality and the digital reconfiguration of the academy
Multimodality is in its infancy as a research discipline, growing in significance alongside, and as a product of, recent advances in digital technology (Kress and Selander 2012). In our increasingly digitised society, we have at our disposal the means of production to share information in a growing array of ways (Jewitt 2006), allowing for the complex and imaginative representation of academic ideas (Bezemer and Kress 2008, Landow 2006). With years of immersion in digital environments, students are seen to call on a range of technological resources in the communication of knowledge (Bezemer and Kress 2008, Carpenter 2009, Jewitt 2006, Land 2011) and thus contribute to the emergence of a new digital scholarship where established notions of authorship are contested (Fitzpatrick 2011) and literacy itself is reconceptualised (Goodfellow 2011).
As a developing research discipline, much of the literature attends to defining and describing what is understood by the term ‘multimodality’. Two of the most significant voices within this field, Gunther Kress and Carey Jewitt, propose definitions built on earlier work within social semiotic theory of communication by Halliday (1978, 1985). Kress approaches multimodality from a position that all communication is inevitably multimodal and can be conceptualised as an assemblage of meaning-carrying modes within a multimodal ensemble (2005, 2009). Kress sees multimodality as the product of intentional acts of configuration and design. Jewitt meanwhile proposes multimodality as encompassing any action that communicates meaning:
In their early and influential text on multimodality, Kress and van Leeuwen (2001) define modes as semiotic resources that can be realised in more than one medium, and facilitate simultaneous discourses and interaction. A broad variety of semiotic resources are proposed as modes, each with their own ‘special powers and effects’ (Kress 2005, p 7). These include communicational forms as diverse as body language, music, voice, animation and design (see in particular the work of Kress and van Leeuwen (2001) and Jewitt (2006). There does not exist within the literature however a single exhaustive attempt to account for all communicational modes, reflecting different views on what can be understood as a mode, as well as the fact that new modes will emerge alongside technological advances (Bateman 2008).
Reflecting a wider trend towards a more image-based society (Rose 2001) populated by image literate students and tutors (Carpenter 2009), the growing significance of the visual image is seen as fundamental to the emergence of multimodality as an academic concern. Kress has been particularly prolific in exploring how image contributes to meaning-making within multimodal artefacts and suggests that visuality could challenge the dominant position of language-based literacy as the main way of representing ideas (see for example Kress and van Leuwen 2001). According to Kress and van Leeuwen, it is the way that modes are combined - to reinforce each other, to fulfil complimentary roles, hierarchically ordered - that forms the basis of multimodality. Following Kress’s lead, multimodality has come to be conceptualised as the ‘constellation’ of image, language, sound and other forms (Carpenter 2009, Flewit et al 2009, Merchant 2007) that are assembled within a single artefact for the purpose of simultaneously communicating information.
Exposition and authorship in digital composition
On the basis that the proliferation of digital technology – and multimodality in particular – allows for academic ideas to be represented in ‘complex, multi-faceted ways’ (Carpenter 2009, p 146), we are prompted to consider what happens to exposition and authorship through the construction of these artefacts. A popular approach within the literature has been to compare how digital artefacts differ from what are proposed as their conventional text-based predecessors. These studies are driven by the belief that the representation of knowledge and ideas across a range of digital modes unsettles language-based literacies (Bezemer and Kress 2008, Goodfellow 2011, Kress 2005) and challenges the dominance of that ‘bastion of academia’, the essay (Carpenter 2009). Merchant (2007) for instance noted a range of distinct features of digitally composed texts, including greater fluidity due to the lack of page boundaries, greater use of multimodality and overlapping roles of reader and writer as a result of non-linear writing and reading paths. Thus, within the digital assignment the student can abandon the traditional sequence of the traditional essay through the juxtaposition of different modal resources (Carpenter 2009, Landow 2006), or by linking to ‘outside’ material. According to Fitzpatrick, the digital text allows for a ‘remix’ or ‘mash up’ (2011, p 17) where the writer’s own ideas interweave with visual, aural and other web-based resources, thus altering the nature of authorship itself:
The abandonment of a linear structure, use of multimodal juxtaposition and inclusion of hyperlinks to ‘outside’ material has clear implications for the reader - or marker - of the digital artefact. Where a digital artefact has more than one entry point (Jewitt 2005, Kress 2004), an early challenge facing the tutor might concern where to begin reading – or viewing – the artefact. Thereafter, in the absence of a clear sequence, she might navigate her own path through the work (Jewitt ibid). As her journey through the artefact unfolds, the tutor might be simultaneously confronted by an orchestration of words, images and sounds, stimulating a range of senses (Kress and van Leeuwen 2001) and exercising different parts of her brain (Bateman 2008, Jewitt 2006). The inclusion of hyperlinks could see the reader’s journey permeate the ‘fuzzy boundaries’ (Fitzpatrick 2011, p 14) of the work, presenting the challenge of considering the student’s ideas in relation to material created by others (Landow 2006). This in turn provokes a need to consider how a remix of existing material sits in relation to the assessment criteria (Fitzpatrick 2011) and established academic values (Goodfellow and Lea 2013).
As her gaze falls upon particular components displayed across the digital canvas, the tutor might consider how the composer’s intentions are revealed by the photograph’s site of production (Rose 2001), the resources that were available to the student (Norris 2009) as well as her technical proficiency (McKenna and McAvinia 2011). This again has implications for authorship as, according to Kress (2004), the reader is herself reconceptualised as being involved in the co-production of meaning, as she reflects on the significance given to particular modes within the artefact and how what we might describe as intra-modal synergy creates meaning over and above the sum of its component parts (Jewitt 2006). With a greater emphasis on how the tutor constructs meaning from the assembled resources, traditional hierarchical power relations are contested (Merchant 2007, Goodfellow 2011) and the ‘writer-reader’ relationship is reconfigured into what Hemmi et al describe as 'distributed authorship' (2009, p 20).
Multimodality within assessment practice
Having considered the power of digital multimodality to influence and alter the nature of assessment, we will now examine the extent to which these possibilities are realised in practice. To begin, contrary to the enthusiasm surrounding the representational possibilities of the digital form, a number of recent studies have instead identified reluctance amongst students to abandon traditional resources and representational forms. In a study of attitudes towards authorship in higher education, Bayne identified a resistance to digital texts amongst students, with conventional approaches favoured over the ‘dubious’ and ‘disposable’ digital form (2006, p 21). Meanwhile, in their study of the digital practices of students at three UK universities, Lea and Jones (2011) found a reluctance to move away from established notions of presenting knowledge (2011). Elsewhere, in their examination of the use of hypertext essays, McKenna and McAvinia (2011) found a continued attachment to traditional essayistic linearity. While it is tempting to interpret this as the imposition of a traditional representational approach upon a new digital site of production, McKenna and McAvinia instead take a more nuanced approach and suggest that the nature of digital spaces – particularly those under the Web 2.0 banner, such as weblogs – encourage or oblige a linear structure. We should avoid assuming however that the digital form will necessarily be viewed with scepticism or trepidation: Land for instance found that students recognised in wikis the opportunity to write in new ways (2011) while other examples of students taking an imaginative approach in the representation of their ideas are provided by Bayne and Ross (2013) and Hemmi et al (2009).
A number of reasons are proposed for the reticence amongst some students to take a multimodal approach within assessment. Carpenter identifies the dilemma of whether a non-conventional approach will look ‘sufficiently academic’ within an academic setting (2009, p 139), prompting us to consider how the proposed playfulness of the digital form (Land 2011, McKenna and Mcavinia 2011) sits within the context of high-stakes summative assessment. Lea and Jones meanwhile identified an ‘ongoing reliance on the authority of the institution when it comes to accessing and utilising web-based resources for their assignments’ (2011, p 377) and that students were reluctant to depart from conventional ways of communicating their ideas without clear institutional validation. Furthermore, concerns surrounding plagiarism discourage the interweaving of original and external material (Lea and Jones 2011).
This reluctance of students to experiment with the digital form should be seen in the context of an educational landscape where multimodal representation is largely confined to the margins. Fitzpatrick (2011) highlights the pressure and expectation placed upon researchers to publish their work in a conventional text-based format, a situation seen by McKenna and Hughes to be significant considering the influence that the academic’s writing has across the academy (2013). McKenna and Hughes also argue that assessment practices fail to attend to the evolving nature of digital scholarship:
This is supported by research undertaken by Lea (2013) where she found that assessment rubrics rarely made explicit reference to the range of meaning-making and textual practices that students used in the construction and representation of their ideas. The tutor who wishes to set a multimodal coursework assignment might be constrained by the knowledge that literacy within her institution is ‘policed by high-stakes assessment and accountability measures’ (Merchant, 2007, p 243) that allows little room for innovation. In an environment with ‘enormous weight placed upon the quantified outcomes of our writing within academic systems of reward’ (Fitzpatick 2011, p 12), course designers and tutors are encouraged to conform to ‘accepted’ modes of assessment and expression, which in turn stifles the multimodal ambition of students in the classroom. As long as assessment procedures continue to be rooted in a system that privileges words on page or screen (Jewitt 2006) and digital formats continue to be seen as inappropriate within the academy (McKenna and Mcavinia 2011) tutors, like their students, might find it hard to venture beyond the text-based form.
The lack of opportunity and incentive for students to explore the possibilities of the multimodal, digital form is significant. First of all, a continued privileging of the printed form leaves the academy out-of-step with wider communicational changes that are taking place in society (Jewitt 2006, Lea 2013). This ‘neglect of a new pedagogic space caused by the failure to engage with the challenges of the digital form’ (Bayne 2006a, p 5) means that opportunities to exploit the skills that students develop within informal settings (Satchwell et al 2013) are missed. Jewitt meanwhile argues that assessment exercises which preclude a multimodal approach means missing out on much of what has been learned in class, and thus calls for a rethinking of assessment criteria to account for new ways of learning (2006).
The scholarly multimodal tradition and the rise of print
The marginalisation of multimodality within assessment can be partly explained by the longstanding dominance of print within the academy. Before we delve back into the history of scholarship however, it is important to avoid conflating multimodality with the digital.
A wander through the contemporary university reveals an established multimodal tradition that pre-dates recent technological advances, a situation recognised by Jewitt when she describes undertaking multimodal research in the era of chalk and blackboard (2006). As we peer into laboratories, crit rooms and performance spaces, we observe scientists, designers and musicians participating in assessment exercises where ideas and ability are communicated without calling on print. We also glimpse students delivering presentations that call on a combination of orality, text, design and digital graphics, an approach that employs a range of old and new representational forms (Bezemer and Kress, Merchant 2007).
If we were able to extend our survey back in time to the ancient Greek universities we would see a similar emphasis on orality as scholars are exposed to ideas through attendance at lectures, with transcription undertaken for the purpose of later study and record (Ong 1982). However, by travelling forward to the mediaeval monasteries that would become the centres of intellectual life, we would recognise the privileged position afforded to written accounts of ideas and stories, albeit as part of a multimodal assemblage. An examination of these illuminated manuscripts reveals a close attention to colour, font and image which sits comfortably alongside our contemporary understanding of multimodality. In particular we can draw parallels with discussion of the meaning-carrying power of typeface (see for instance Bateman 2008 and Jewitt 2006) on the basis that the visual character of these texts would have been known to the calligraphers and typographers of these early scholarly materials (Jewitt 2005).
The subsequent creation of the first printing presses in the university towns of Bologna, Paris and Oxford saw professors working closely with scribers and rubricators to realise text alongside images in order to share knowledge of an anatomical, naturalist and botanical nature (Febvre and Martin 1979). This is interesting from a multimodal perspective for two reasons. First of all, the significance of image within these artefacts evokes the work of Kress and van Leuwen (2001) and others who recognise the contemporary significance of visuality within the multimodal assemblage. Secondly, we see the creation of these early artefacts less as a task solely of writing, but instead as a process of design (consistent with the position taken by Jewitt 2005, McKenna and McAvinia 2011 and Kress and van Leuwen 2001).
With a further leap forward in time we would note the gradual decline in scholarly significance of oral communication, deprivileged by the proliferation of the printed page (Eistenstein 1979), prompted by a consolidation of language, grammar and spelling as printers strived for consistency (Febvre and Martin 1979). The eminent position that the printed text would attain within ancient universities is reflected, Land suggests, by the positioning of the bound text at the centre of many of the heraldic crests of these ancient seats of learning (2011), while Bayne notes the presence of the bound codex at the heart of countless contemporary university logos, a symbolic anchor of scholarly authority (2006a). Bayne also proposes that ‘academic discourse has historically been, and continues to a large extent to be, deeply ‘papyrocentric’’, (p 5) taking us back to the present where the text-based form continues to sit at the heart of what is largely understood to be the authoritative, trusted mode of scholarly representation.
The technical and rhetorical (il)literacy of the digital student
A debate of significance to multimodality surrounds the assumptions that can be made in relation to the technical and critical communicational skills that students possess. On the one hand it is argued that students spend their lives surrounded by digital technology (Merchant 2007), with exposure to Web 2.0 resources making them electronic multi-taskers (Land 2011) and thus equipped to ‘enter compositional classrooms already possessing technological skills that often surpass those of their teachers’ (Carpenter 2009, p 139). However the proposed existence of a student body with an instinctive ability to call on the digital means of production – a group that Mark Prensky provocatively described as ‘Digital Natives’ (2001) – has little basis in evidence according to Lea and Jones (2011). In their call for a more nuanced understanding of students’ use of technology meanwhile, Bennet and Maton (2010) point to a lack of empirical evidence supporting the idea of a digitally astute student body, alongside a ‘significant lack of consensus over what effects digital technology is actually having on young people’ (p 322). Rather than making broad assumptions surrounding a homogenous student body, Bennett and Maton encourage us to recognise that individuals experience different access and opportunities to exploit technology, reminding us of the significant influence of social context upon acts of communication (Bezemer and Kress 2008, Jewitt 2006, Kress and van Leeuwen 2001).
If we put aside the debate surrounding the proposed technological proficiency of the student body and assume for a moment that exposure to digital material necessarily leads to proficiency with communicational tools, the question arises of whether students know how to do so in a critical way. A consistent theme within the literature is the need for teaching, learning and assessment to be reconfigured to adapt to the social technologies that students are able to call upon (see for instance Merchant 2007 and Land 2011). Carpenter meanwhile discourages a conflation of technical proficiency with critical literacy skills and challenges the notion that the proliferation of digital technology necessarily means that students know how to use these tools in a rhetorical way (2009). We should thus be wary of conflating exposure to digital material, or familiarity with technological communication, with an ability to employ these resources in the production of multimodal artefacts for the purpose of assessment.
Deconstructing, interpreting and assessing multimodal artefacts
In an environment where the growth in digital technology accelerates the ascendancy of the multimodal artefact (Bateman 2008) there is a need to consider how to interpret and understand these emerging forms. Carpenter (2009) points to the value of genre theory in order that we place a greater focus on what multimodal artefacts do and how they are used, rather than being satisfied with what they are. Jewitt meanwhile sees value in observing students as they participate in the creation of multimodal artefacts, on the basis that additional meaning is created in the way that they interact with the screen and with fellow students (2006). An alternative approach is proposed by Bateman (2008) who builds on the earlier work of Kress and van Leeuwen (2001) to propose a ‘modal decomposition’ of the artefact into a series of layers and component parts, before reflecting on the genre of each mode, on the basis that this will offer insights into the creator’s motivation. Carpenter however points to the difference between description and interpretation when he suggests that the task of measuring the features of an electronic text against assessment criteria and standards is not fulfilled simply by an awareness of its component parts.
In their work on multimodal data collection and transcription meanwhile, Flewit et al (2009) identify a number of challenges that would seem to problematise methodologies that intend to dissect multimodal artefacts into their component parts, including the difficulty in unpicking and then interpreting data presented across a range of modes. Jewitt similarly points to the complexity of multimodal artefacts and warns that the decomposition runs the risk of missing the meaning created by interrelationships between modes (2006). With a particular focus on the assessment of multimodal artefacts meanwhile, Kress and Selander (2012) argue that the tutor’s role is to determine and understand the principle of selection that is demonstrated within a multimodal assignment. Jewitt meanwhile proposes that the marker’s role is to relate the student’s work to the information presented within the curriculum, whilst accepting that the representational form will look beyond a traditional linguistic process (2010). As yet however, there has been a lack of research into whether and how this can be achieved, reflecting a wider absence of investigation into the assessment experiences of tutors in practice.
Reflecting on the body of literature
Within this literature review I have critically reviewed literature that offers insights into multimodal assessment. Of particular value to the focus of my dissertation is the work of Kress (2005, 2009) and Jewitt (2006, 2009) concerning the semiotics of digital multimodality. Their influential work has informed my understanding of multimodality, providing a framework that has enabled me to explore assessment practices within the particular contexts of the Digital Education and Architecture at The University of Edinburgh.
With a view to investigating experiences of tutors, the literature is most valuable when theory is supported by observational research, or draws on case studies that explore multimodality in practice. For the most part however, there has been a lack of research that considers multimodal assessment in practice, particularly in the higher education setting. While Jewitt (2006) suggests that many of the fundamental themes that draw on school-based research are sufficiently general to be usefully applied to other levels of education (see for instance Bezemer and Kress 2008, Carpenter 2009 and Jewitt 2006), a specific focus on practice within higher education would allow for more in-depth and context-specific investigation. For instance, potential differences between the high school and higher education environments in terms of access and exposure to digital resources, modes of teaching and assessment, and the nature of the respective student bodies, might yield different results in relation to experiences of multimodality and digital composition.
A more significant gap within the literature, and one that transcends different educational levels, concerns the lack of attention to how multimodality impacts upon tutor practices. There is a tendency to focus on the representational possibilities of multimodality without considering the implications for those responsible with marking these artefacts. Whilst the need to rethink assessment is recognised within the literature (Jewitt 2006, Lea 2013), there has been little attention to what this might entail in practical terms. If we accept that the digitisation of the academy predicates the composition of multimodal artefacts, there is clear value in exploring how multimodality influences, inspires and problematises the assessment practices of tutors.