Literacy is changing. At BYU, we’re retooling first year writing courses and implementing “multimodal literacy.” It boils down to learning how to communicate using media in combination with words, mostly. Literacy is starting to be about design, about the combination of elements, not just text, nor just text + illustrating picture. There is a dynamic at play between the two, and there are different ways one engages audiences through adding image, video, audio, or hyperlinking to one’s communication.
I notice now that I think very differently about reading. I’ve been reading Daniel Boorstin’s book, The Discoverers, a very good layman’s history of famous voyages, inventions, and cultural changes. As I turned the pages, I found myself impatient with the fact that there was no image of the Ptolemaic world maps that he was discussing at length. I quickly pulled up very good images of the various maps he was talking about on my iPhone. Sure, I followed his words, but suddenly his words meant a lot more when I could see the thing he was talking about. How could anyone publish a book today and keep such readily available media from those who could profit from it?
And yet, at the same time, I feel like resisting the allures of non-texts. Yesterday my wife was re-doing her blog design, and this got me playing with mine. I spent two hours finding and cropping and trying out various images. In the end, I got my new blog banner up but (shock!) I didn’t manage to actually update my blog, which has lain dormant for months (as I’ve been writing for Academic Evolution and a bit on this one). Lesson: creative energy and time spent on pictures and design is time subtracted from actual writing. As a writing instructor, I shiver at the prospect of encouraging students toward more superficial elements while the substantial thinking goes undone.
And yet, I see the way that those who use photographs on their blogs invite me into their worlds in ways that not even a finely crafted piece of prose can do. Clearly, the media and design factors have a very consequential social element. My wife involved me in figuring out her new blog look. We debated the message that a given arrangement of colors conveyed. It felt creative, and fun to do together. And then, today, she got her photographer friend to come over and take her picture for the blog. They laughed and laughed, and when the friend sent over the proofs, it became an occasion for all the family members to vote. (Highlight: my returned missionary son summoning others to the computer saying, “Come look at your sexy mom!!”). An ethical discussion ensued about possible Photoshopping out of a wrinkle or two. I’m left with the distinct impression that my wife’s venture in multimodal composition was a distinctly more enjoyable thing for her and for many others than my chiseled paragraphs about oh-so-serious this-and-that that I spend so much time on in my essayistic online installments.
Oranges and apples? I don’t know. I think literacy is becoming more social, that literature can and should be so, and all of this is necessarily mediated through digital and online means. What is sacred about the barren text, untouched by the sensationalism of photography or video? I actually think there might be some very good answers to that question. But be it vanilla print or animation-enhanced anecdotes, I’m definitely in favor of whatever creative medium honors experience authentically. To me, that means honoring the nuances of our thoughts and feelings (often best communicated through the written word), honoring the fully sensuous range of human experience (meaning appeals to the eyes and ears), and honoring the people that truly bring to life our art either as we produce or consume it. Different media, and combinations of media, play to different important modes of experience and of art. I hope we can be patient enough to figure out the newer modes, even while honoring the tried and true.
May I humbly suggest that place to start with multimodal Mormon creativity in relation to textual narrative is illustration and cartooning and graphic novel and narrative or illustrative photography rather than film?
Once you invoke moving images and sound, the experience changes. But static images + text adds up to an experience that is still reading (e.g. not viewing).
The VOOK: http://vook.com/vook.php
I’m not a fan.
Hmmm, lemme clarify that "I’m not a fan" thing because I’m coming at this from three different perspectives: writer, reader, and publisher.
As a writer, I’d love it because I’m a visual person and when I write, it’s like I’m watching a movie in my head. I’m taking dictation, making notes on what parts of the set and props I find important or pretty, trying to weave a soundtrack in as I hear it, describing places and such. I would *love* to be able to impart that in movie form to the reader to get it in a much more visceral way. On the other hand, if it can’t be done exactly the way I see it, then I’d rather the reader make it up in his/her own head so I don’t have to see the distortion of my vision. If I could have nothing else embedded in a book, I’d want the soundtrack because music is so much a part of how I write, how I see the movies in my head, the rhythm of the dialogue and narrative.
As a reader, I’d hate it. It’d pull me out of the text completely. I’d go back to print books. I have also recently run into a bunch of people who say that when they read, they do NOT see movies in their heads. I can’t imagine that.
As a publisher, the only thing I can see is the added cost for no return. To do it right, there would be licensing fees to pay for music and images. There are photographers to pay for original images. There are production people and actors for video portions. I just don’t see how anybody but the most well-funded production companies (notice I didn’t say publishing companies) could do this right.
On the other hand, I can almost guarantee that there are going to be a lot of enhanced ebooks/vooks out there with pirated material (music, images, video). This is a rights issue that’s going to just blow up like crazy.
All that said, the artist in me gets giddy at the thought.
Moriah,
I’m one of those people who doesn’t see pictures/movies in my head when I read a story. Instead, I hear voices saying the words.
I’ve often wondered how different modes of reading impact reading taste. For example, though I blush to admit it, my eye almost automatically skips descriptive passages. Instead, my attention rivets to internal and external dialogue.
The part of me that spends too much time looking at educational research says you could do a study based on this. Recruit a large enough reading base, categorize according to how they describe their reading experience, then see if there’s a correlation to preferences from among a certain set of texts. Perhaps fortunately for everyone, no one’s paying me to conduct quasi-sociological-slash-literary research…
(And by the way, I also think that the Vook would be incredibly distracting. Whatever Gideon’s ideal combined medium would be, this isn’t it, for me at least…)
I love the idea of the book as a supplementary form designed to expand the available market to encompass those who want a richer experience.
Problem is that everyone wants to kill the old form whenever they bring out a new one, and I just don’t see ordinary, text-based media going away any time soon.
Other problem is that you create differently for different media. If I intend to show a color map in a link then I’m not required to add details of location to my narrative–with substantial effects both on the flow of that narrative and the descriptive (and often most vivid and interesting) prose that spurs the reader’s imagination.
Different media for different authors (and audiences).
I find myself wondering about those people who don’t process as much through sight or sound as they do through touch and movement. Multimodal communications that don’t involve tactile and kinesthetic experiences are not going to be any more experiential or useful to them than plain black and white text.
I’d like to recommend a book by a linguist that discusses this: TRY TO FEEL IT MY WAY by Suzette Haden Elgin.
[b]I’m one of those people who doesn’t see pictures/movies in my head when I read a story. Instead, I hear voices saying the words.[/b]
I find that so fascinating. I’m such a visual person and I always say that I write because I can’t paint.
[b]I’ve often wondered how different modes of reading impact reading taste. For example, though I blush to admit it, my eye almost automatically skips descriptive passages.[/b]
And I write a lot of description. 😀 (Because I can’t paint.)
[b](And by the way, I also think that the Vook would be incredibly distracting. Whatever Gideon’s ideal combined medium would be, this isn’t it, for me at least…)[/b]
Even as a visual person, the mixed media would kill me. Maybe the ADD? I find footnotes annoying enough in print, and Wikipedia’s cornucopia of links is even harder to resist. I have to make a concerned effort to ignore them.
So I know what I like as a reader, and I’m sitting here debating the value of enhancing my ebooks as a publisher, what goes, what stays. Hyperlinks within the text would be (to me) incredibly annoying anyway, but some people like those. Popups? I would kill somebody.
On the other hand, putting such things at the back of the book and/or having the ability to hide hyperlinks and disable popups would solve the problem. (I don’t even think there’s a device that can do that yet; I don’t even think the iPhone can if it’s hard-coded.)
On the other other hand, we’ve got digital natives who may not be able to tolerate straight text, and that’s who I have to think about. I really think the multimedia experience is the wave of the future, but that doesn’t mean I have to like it or that I have the answers to how to implement it–or even if I want to.
I see the multimedia experience like a piece of needlework that you work with different stitches, different fibers, adding beads and such here and there to give it sparkle and lift and texture. And there’s such a thing as too much.
There’s part of me that says: If you change the medium, you change the story. "Enhancing" stories originally written for print by adding bells and whistles seems less likely to result in a good product than composing a story from scratch that’s meant to incorporate all those features.
On the other hand… Fantasy and science fiction novels are already famous (or infamous) for maps and assorted supporting paraphernalia for invented worlds. And I actually read all the appendices to Lord of the Rings…
Maybe part of this is whether the multimedia experience interrupts the flow of the story or is designed to be part of it.
Skipping topics somewhat… My 21-year-old son insists that video games can be a storytelling medium. He’s reluctantly gotten me to concede his point in a few cases, most notably the little-known but haunting video game Ico. He and a friend have spent long hours developing plotlines for stories they want to tell in video game format. They’ve worked on everything from plotlines to composing background music. Unfortunately, neither of them are video game designers, computer programmers, etc., so I don’t know how they’ll be able to succeed in getting their stories told. Maybe that will be one of the next killer apps: one that allows ordinary-joe users to develop interactive stories. (Actually, something like that probably already exists out there and I don’t know it…)
Jonathan, the reader part of me agrees with you totally, up to and including the maps and glossaries. The artist part of me is excited by the possibilities.
My brother is a lead programmer at EA and he built a game app for iPhone. Years and years and years ago, just before he went on his mission, he and I collaborated on an RPG. I wrote the storyline, he created the graphics. It didn’t go anywhere because two years in regular time is light years in computer time, but we did it.
I think that electronic books, enhanced or not, need to look to gaming to find its role model, and stop looking at music as its role model. (Not that it’s doing that anyway.)
In the end, I believe that books aren’t competing with books or e-books or enhanced e-books. Books are competing with TV and beer, the internet and World of Warcraft.
Talk about back to the future! In the mid-1980s, I attended a seminar at BYU about this newfangled thing called "hypertext." Text! Pictures! Links! Indexing! The example used was a chapter from a Thomas Hardy novel. One of the first high-tech bubbles was hypertext CD-ROMs. They mostly failed as a market, but then the Internet came along and made the concept work like gangbusters.
I like the idea of simply expanding on the footnote. A great thing about publishing translations online is that I can link to my blog where I can discuss translation decisions and provide illustrations of difficult or unique concepts. Here’s an example (click on "Notes"):
http://www.eugenewoodbury.com/wind/wind_ch01.htm
The one creative talent I truly envy is that of the manga artist. Manga is the one medium where I still prefer the printed page. In Japan, you can buy manga in every imaginable genre, fiction and nonfiction alike.
Another medium popular in Japan but that never succeeded here is the "visual novel." It’s basically the hypertext CD-ROM re-imagined using manga and anime stylistics. I’ve never actually played/read one (I don’t play computer games period), but anime series based on visual novels, such as [i]Kanon,[/i] are some of my favorites.
http://en.wikipedia.org/wiki/Visual_novel
Like Jonathan’s son says, I think the discipline of having the map out the story in advance–pretty much according to McKee rules–really helps. Ultimately, though, the text should be able to stand by itself. If the meat isn’t palatable without being smothered with tenderizer and spices, maybe it wasn’t that good a cut to start with.
Yeah, it’ll remain to be seen if this is more than just CD-ROMs part 2. Although there are a couple of major, important differences. One is that you could carry many such works in one device. Another is that the works themselves could be updated over time.
Complete sidetrack but:
One thing that multi-modal works could help with is criticism. Anybody remember reading film criticism in the ’90s? It was amazing if an article had a few black and white screen captures to reference interspersed throughout the text. I’d love to see book covers and excerpts, audio-interview snippets, charts/timelines and, yes, film clips presented in such a way as to enrich the critical work the text is engaged in.
.
I’m trying to teach a more multimodal literacy at the high school level, but I keep failing. I’m really not quite sure how to do it. (Never seen it done, never been trained in it, don’t know anyone doing it . . . besides Gideon, apparently. Go Gideon!)
Th., think scrapbooks. Narrative text, pix, clippings, mementos, (no DIY scratch’n’sniff possible, tho), art, foldouts (like maps), possibly soundchips if you wanted to get that sophisticated (http://www.amazon.com/Second-Recordable-Sound-Stuffed-Animals/dp/B000JG902O/ref=pd_sim_dbs_misc_4).
So how many people currently buy regular old ebooks, and for what device/reader (Kindle, Sony, iPod, etc.)?
Just curious.