Adaptive Path Blog

A Brief History of the Future

In 1991, Mark Weiser of Xerox PARC wrote an article for Scientific American titled The Computer for the 21st Century (PDF link here). Arguably the father of ubiquitous computing, Weiser predicted a future of computing characterized by an ecosystem of mobile screen-based devices that would seamlessly communicate with one another. In such a system, we would distribute our computation across multiple devices, effortlessly tossing content and activity between them, continually renegotiating these devices and the spaces they occupy based on our changing needs, tasks and contexts.

Weiser’s vision included three device classes, segmented based on their size and intended use. “Boards” would be large screen devices perhaps three feet to a side, “Pads” would approximate the size of a sheet of paper, and “Tabs” would feature a small screen at the scale of a Post-It note. Rather than interacting with a single computer (the dominant interactive paradigm at the dawn of the 90s), Weiser predicted that we would interact with tens or hundreds of these computational devices throughout the day, switching between them depending on activity and context.

Today’s Emerging Device Ecosystem

When we consider the iPhone, iPod Touch and iPad, as well as the new touchscreen iPod nano, it is clear that Weiser was quite accurate when he predicted that we would soon live amongst an ecosystem of mobile screen-based computational devices. Not everything has turned out as he planned, however. There is no widely-available “Board”-sized touchscreen device, and the “seamless” distribution of content and activity between these devices is certainly wanting.

As Apple fleshes out its lineup of iOS (and iOS-inspired) devices, bigger with the iPad and smaller with the iPod nano, it begs the question of what the future holds as these devices get better at communicating and collaborating with one another, and what new uses and contexts emerge as a result of this proliferation of form factors.

What would a near-future iPod nano look like?

The current iteration of the touchscreen iPod nano is impressive, but what really sparked conversation here at Adaptive Path is what the near-future of this device might look like. Upcoming iterations could potentially be outfitted with Bluetooth and Wi-Fi, natively run iOS (rather than an interactive layer that feels like iOS), and ultimately allow users to install apps.

The small screen makes the iPod nano impractical as a device for consuming video, but it offers an interesting extension of the conventional wallet photo. The camera of the previous iPod nano has sadly been eliminated, and text entry into the device would be challenging given its tiny screen, but the multi-touch screen and accelerometer (and potentially a future gyroscope) could open up some interesting possibilities for control or input. The square screen offers a unique symmetry and looseness of orientation that invites design experimentation.

The Wearable, Fashionable nano

With its bright screen and clip-on design, the iPod nano extends the role of the “device as fashion accessory” beyond that of white earbuds and the iPod Shuffle, and the opportunity to “wear” one’s own album art offers a new level of personal expressiveness. Further, the tiny, wearable form factor of the new nano offers a level of unobtrusiveness that is absent from the iPhone and iPod Touch. Hipsters everywhere can now carry their ripped collection of LPs without shoving a device into the tiny pockets of their skinny jeans.

Which isn’t entirely true, because hipsters carry tape players, not iPods.

But anyway.

Connecting and Extending the Device Ecosystem

Short-range wireless communication would allow nearby iPod nanos to communicate with one another in a manner similar to that of Siftables, and the configuration and reconfiguration of multiple nanos could offer a form of play for all those kids these days with their technologies and Facebooks and Beibers. With wireless functionality your iPod nano could also offer a real-world, local access point into your online identity (or, sure, your Ping profile) through an iPhone or iPad app.

The iPod nano already has a radio, which allows what we jokingly refer to as “live podcasting” and “wireless audio streaming.” The radio could also be used to pick up low-power broadcasts from other nearby devices, making the nano a great output device for a remote baby monitor, intercom, thermometer or weather station.

Further alternative uses for the iPod nano could include: bedside clock, alarm, kitchen timer, stopwatch, tiny picture frame, die (as in dice), sketch pad, fake eye (great for pirates), fake fingerprint, bicycle light, and smart car key.

Finally, a future version of the iPod nano that runs iOS and allows user-installed apps would be a great platform for our Charmr concept!

Design work isn’t always glorious. Recently I had the good fortune of being tossed into the middle of a project that had been going on for quite some time. Much of the “designerly” work was finished, and the team had already switched into production mode. They handed me a stack of wireframes and asked me to expand them in some modest directions.

“Great! Can do!” I said.

And then I stared at them, paralyzed, for an hour.

“I do this all the time!” I kept thinking. “Why can’t I figure out what to do with this?”

I began to realize that the reason I didn’t know what to do with the wireframes was because I didn’t know anything about the context in which I was supposed to work. This, despite the fact that the project room was covered in sketches, concepts, Post-It notes, and all sorts of artifacts from the design process. “Why don’t these things mean anything to me?” I wondered.

Design is thinking through doing.

As designers, we think through doing. Design is a reflective practice between the designer and her design materials. When you sketch something and commit it to paper, it moves from being an abstract thought to something that is more concrete and real. Perceiving this concreteness, in turn, influences your thinking, leading to new questions that spawn new ideas. It is this reflective conversation, between a designer and the medium in which she designs, that is central to the design process.

It is the act of creating these design artifacts, rather than the artifacts themselves, that is the most valuable aspect of the design process. These artifacts are the physical manifestations of our thinking. This externalization of thought, this act of making tangible these abstract ideas, is the core of what we do as designers.

And this is why it is so difficult when we find ourselves diving into the middle of a project.

We are surrounded by artifacts of the design process that preceded us, poster boards covered in Post-It notes, sketches of potential layouts, but since we were not actively engaged in their creation, none of it means anything to us. Viewing these artifacts does not grant us the same knowledge as having created them.

If you didn’t do the doing, how do you do the thinking?

The challenge I have been faced with lately is getting rapidly up to speed on a project where most of these design activities have been conducted, and the designers are already engaged in production. They have a rich and meaningful understanding of the project which I currently lack, and yet I need to help wherever I can in producing deliverables. Having not participated in most of the design process, it is difficult to orient myself towards solutions that would be contextually appropriate for this particular project.

All is not lost, however! I have discovered a few valuable survival tips along the way that have proved indispensable in getting up to speed, and productive, when tossed into a design project mid-stream:

Ask questions. Especially dumb ones. – Not only will this help you nail the core principles of the project, the act of explaining it to you will help your fellow team members re-internalize these priorities. Plus, your fresh eyes might identify some tacit assumptions that have not yet been broken open.

When in doubt, sketch it out. – As a kinesthetic learner, I can’t simply look at a set of wireframes and know what’s going on. However, if I grab a Sharpie and spend a few seconds recreating the wireframes on paper, I will suddenly begin to understand them. The trick is to make this activity as absolutely lightweight as possible. No matter if the wireframes are in Fireworks, InDesign, Photoshop or OmniGraffle, I will still deconstruct them on paper. Through this activity I can eventually understand and internalize the other designer’s work well enough to interpret and extend their wireframes in the digital medium of choice.

Make it and talk through it. – You’ve done this before, so give yourself some credit. Don’t be afraid to draft up a proposed solution, present it to a veteran designer on the project, and talk through the reasons you did what you did. Together you may uncover some mistaken assumptions you held about your users, or some core business requirements you were missing. As a designer you already have some pretty good rationale for the decisions you make, and you simply need to adapt them to this particular project. Over time, you too will develop the same intuition that guides the other designers on your team.

Have you discovered any survival mechanisms for when you dive into the middle of a project? If so, please sound off in the comments!

As designers it is often our responsibility to imagine the future possibilities of things. We rarely get to design independent of social and cultural contexts, and we never get to design independent of the perceptual capabilities of our users. You could design a marvelous interface that makes terrific use of “color” outside of the visible spectrum, but it is unlikely that a human would be able to see it. It would be rare indeed to find a visual designer who bemoans the shackles of human perception, which unfairly force her to work entirely within the visible light spectrum.

Recently, Adam Greenfield took Apple to task regarding the design of some of their iOS apps, including iBooks, Compass, Calendar and Notes. His criticism is that these applications are too skeuomorphic; that they feature visual cues from their real-world analogues that insult the future-facing potential of this new generation of touch screen mobile devices:

The iPhone and iPad, as I argued on the launch of the original in 2007, are history’s first full-fledged everyware devices — post-PC interface devices of enormous power and grace — and here somebody in Apple’s UX shop has saddled them with the most awful and mawkish and flat-out tacky visual cues. You can credibly accuse Cupertino of any number of sins over the course of the last thirty years, but tackiness has not ordinarily numbered among them.

Every interface exists within a cultural and perceptual context. Even on touch screen devices, controls are often rendered in a manner that reflects analogues in the physical world. The metaphorical basis of these real-world analogues can be as culturally literal as the bookshelf view in iBooks, or as perceptually abstract as the use of a gradient and drop shadow to communicate three-dimensionality.

The whole concept of affordance depends on there being a coherence between the function of a particular interaction, and the manner in which it is represented visually. If there isn’t enough coherence between an interactive element and the function it performs, the interface will likely be confusing, ambiguous and frustrating. Designers must decide which metaphors to leverage in the design of their interfaces, within the context of existing interactive patterns and real-world experience.

Just as a visual designer must work within the “constraints” of human visual perception, interaction designers must work within the constraints of analogous real-world experiences, existing interaction patterns, and cultural knowledge. These are welcome constraints, however, as they provide a wealth of meaning that can be effectively leveraged by an otherwise new interactive model.

Real-World Experience

Consider this. Is this button sticking out, or is it pressed in?

How about this one?

Technically, there is nothing about either of these images that objectively designates one as concave or the other as convex. However, in the physical world we are used to seeing objects lit from the top rather than from the bottom. As a result we perceive depth in images as though they too are lit from the top.

This may seem obvious, but it illustrates a very important point: on-screen images have meaning only insofar as we have something to associate them with. We are a ways off yet where our every interaction with the world is mediated by a screen, and so the physical world and its properties of light and shadow hold a trove of meaning that we can leverage as designers to communicate a sense of three-dimensionality. Even in the simple world of gradients, bevels and drop shadows, we are leveraging a metaphorical language steeped in our experience with the physical world.

Existing Interactive Patterns

Real-world experiences play a large role in mediating our perception of interfaces, but established patterns for interaction are also incredibly influential. Blue underlined text has no off-screen analogue, but through repetition and familiarization it has become widely recognized as representing a hyperlink. These patterns take time to become established, but when they do they act as a source of meaning that can be leveraged in our interactions.

Interactive patterns are certainly not limited to visual treatment, and with careful intent can quickly establish themselves. If we trace the evolution of Apple’s two-finger trackpad scrolling, to their multi-touch trackpads, to their multi-touch screens, it is clear they are deliberately training us in a new physical vocabulary. As this gestural language becomes familiar it becomes a resource for design, in turn clearing the way for more sophisticated gestures in the future.

Cultural Knowledge

Cultural knowledge and society comprise a final constraint within which we design interactions. Certain artifacts, materials and textures hold cultural significance, which we can leverage in our interfaces to communicate their intended use, or set the stage for a particular experience. A few apps and utilities in iOS, including Calendar, Compass, iBooks, and the ghastly Notes, come under harsh fire from Greenfield for their insultingly literal remediation of cultural ephemera. Compass looks like a brass compass! Your iBooks rest on your wooden iBookshelf! Notes looks like a yellow legal pad! Get it? NOTES?! Like the ones you WRITE?!?!

Now, I agree with Greenfield that the patronizing appearance of these applications is inexcusable, that the literal representation of their real-world analogues demonstrates a surprising lack of elegance and ambition on the part of Apple. However, I disagree with Greenfield on the assertion that cultural metaphors have no place in a future-facing mobile context.

I agree that Notes is an abomination, but it is not an abomination because it leverages cultural meanings, or because it attempts to channel a physical analogue in its form. It is a poor interface because it is a clumsy and inelegant implementation that takes the metaphor of a notepad far too literally. A better effort would be to distill the idea of a “note” to its absolute essence, and to extend it with the unique capabilities afforded by a mobile touch screen device.

The tackiness of these apps is clumsy and insulting, but to claim that screen-based interfaces should exist outside of physical experience, outside of cultural meaning, is nonsense. Real-world analogues are a useful metaphor for structuring the way we think about a possible design, as well as a resource to leverage in shaping how users think about our design. This doesn’t excuse the ham-fisted design of Notes, but it lends some context as to why it fails, and how it could succeed without completely dispensing with metaphor and physicality.

By our talented summer associate Dane Petersen

When you think of a computer, what sort of image pops into your head? Perhaps you see a screen, keyboard and mouse. Maybe you think of a laptop, with its screen, keyboard and touchpad all built into the same unit. If you’re particularly crafty, you might think of a mobile device with a touch screen.

The truth is, none of these objects are actually a computer. They are merely the tools that we use to interact with a computer. We traditionally input data using a keyboard and a pointing device, and the computer in turn outputs information on a screen. A programmable thermostat employs a similar interaction, using buttons for input and an LCD display for feedback. Even on the iPhone, an incredibly innovative mobile computing device, these familiar desktop interactions are still present. Your finger acts as a pointing device, the screen produces visual outputs in response to your input, and you can summon an on-screen keyboard for inputting text.

This desktop paradigm of interaction has been with us for many years, and with the iPhone, Android and Pre it seems to be reproducing well in a mobile setting. However, the “computing” function of a computer, in that it takes in and processes information, requires no particular form factor. As processors get smaller and more powerful, it seems our only limitation is what tools we will use to get information into, and out of, the computer.

Speculation anticipates that the future of computing will involve an ecosystem of mobile devices, touch screen tablets and netbooks, but we need not confine ourselves to this inevitability. To assume that all computer-mediated interactions should be shoehorned into a paradigm that implies a pointing device and screen, is to flagrantly dismiss what is wonderful and unique about our human existence.

Humans are physical beings that exist in a physical world. This “being” is an experience that we all share; a rich, experiential understanding and familiarity with our physical environment. People have an incredible capacity for subtlety and nuance that remains largely untapped in existing digital interactions. Demanding that all our computer-mediated interactions happen through the common screen is an insult to our senses, and an insult to what it means to be a living, thinking, feeling person.

As experience designers it is our duty to celebrate, rather than subvert, the deep, shared experience we have as physical beings. We share a highly-evolved capacity to process an infinite variety of inputs from all of our senses, and are capable of interacting with our surroundings in any number of ways. Our interactions with computers should be no less varied.

Already there is a rich library of interactions in the gaming realm that celebrates the rich, shared tradition of our own physicality. New input mechanisms from Wii Fit to Rock Band all invite unique interactions, and Microsoft’s Project Natal elevates movement-based control to a new level.

These games all involve wonderful physical interactions, but they’ve only scratched the surface of human capability. They break the desktop paradigm in terms of new forms of input, but still depend on a screen for outputting feedback to the player. The use of sound, music and vibrations in gaming helps to enliven the immersive experience, but they remain as supplements to, rather than replacements for, visual feedback.

Moving digital interactions into the physical world can have tremendous value, but developing a compelling experience without the use of a screen can be extremely challenging. In seeking inspiration for these emerging types of analog interactions, it helps to turn an eye to history, to a time long before our interactions were mediated by digital screens.

Based in San Francisco, the Musée Mecanique has an incredible collection of antique coin-operated arcade machines, player pianos and nickelodeons from the late 1800s and early 1900s. That there was tremendous innovation at the turn-of-the-century in these penny arcade games, in step with the tremendous innovation in today’s games, is no coincidence. Entertainment is business, as true then as it is now, and most wonderful things manifest themselves as novelties before they are recognized as something truly useful.

One such device was the Mutoscope, a coin-operated moving picture device built around the turn of the century. As you look through the view port and turn the metal hand crank, it flips through a stack of still photographs like a mechanical flip book. There is something compelling about the materiality of actual physical photographs, and the act of manually animating them is indescribably delightful.

This video features a Marilyn Monroe reel playing on one of the museum’s Mutoscopes. It’s pretty racy by historical standards, in that people are actually kissing one another, but rest assured that it was approved by the New York censors.

This video shows an Englehardt Coin-Operated Orchestrion from 1915. The device reads a perforated paper roll, similar to a punchcard, and translates the bits into notes for each instrument. Even though you are not controlling the device yourself, there is something delightful about real physical instruments controlled by digital punches stored on an analog medium.

The museum has a diverse collection with hundreds of other arcade games, including fortune tellers, music boxes, shooting ranges, car races, and even an animated opium den. While the interactions themselves are quite simple, their experiential qualities are incredibly rich. The coolness of a metal handle, the warmth of a wooden cabinet and the lavish Victorian ornamentation all contribute to an incredibly engaging, tactile experience.

Daniel Rozin’s Wooden Mirror represents a modern interaction that does a wonderful job channeling the delightful, tactile experience of these old analog devices, and giving them new form with invisible computing technology.

The “mirror” is composed of 830 square pieces of wood, each attached to its own motor that controls the angle and brightness of each individual square. A tiny video camera records the viewer, translating their image into pixels that are sent to the mirror in real time. The pieces of wood make a satisfying “clattering” sound as they move into place, further grounding this interaction in the senses.

Expanding on the experiential qualities of this concept, ubiquitous internet connectivity and wireless technology opens all sorts of doors for a distributed system of computers, inputs and outputs. Inspired by the arcade machines of yore, there is limitless potential to augment these traditional analog interactions with networked technology, moving our computing experiences out from behind the screen, and into the world.

Part of the Smart.fm iPhone App Story
By our talented summer associate Dane Petersen

The smart.fm website offers a number of fun learning games that help you master all sorts of world knowledge, from Japanese to French to European birds. Like a sophisticated stack of flash cards, the games learn and adapt to your performance, and constantly tune themselves to providing you with the optimal learning environment. As you progress through the stack, items that you have obviously mastered will automatically appear less frequently than items that you are still trying to learn.

The flash card analogy works well for describing the functionality of the smart.fm learning games, but it provides an impoverished account of their more experiential qualities. In particular, BrainSpeed’s pulsing icons, flapping wings and exploding pufferfish all work together to create a gaming environment that feels face-paced and zany. In designing the look and feel of the learning game portion of the smart.fm iPhone app, we knew we had to create something that would be more engaging than a stack of index cards.

When working on the experience design project for smart.fm, one of our guiding design principles was that the new website would be a fun and open space that invites play. Bringing this concept to the iPhone, we wanted to make the mobile learning game lightweight and playful, easy to start and easy to put away, while still delighting users with fun interactions. People would likely find themselves playing the game during those odd dull minutes of the day, perhaps while waiting for the bus, and we wanted to make sure that these short bursts of play offered a rewarding experience.

In reflecting on these goals, we generated numerous sketches and ideas for ways we could represent the timed, multiple-choice nature of the mobile learning game in a richly experiential manner. We explored metaphors for different ways to show questions and answers, represent time running out, and communicate the user’s progress towards learning an item. We considered the materiality of the game-space, and imagined ways to introduce tangibility through unique interactions. In this video I present a brief walk through my sketchbook, and talk about these explorations:

I took the results of these exploration sessions into a bit more detail, generating a number of sketches that depict potential design directions for the learning game. “Sore” is the Japanese word for “that, that one,” and I oriented this series of sketches around a screen where the user is trying to learn this word, and select its correct response from a series of choices. I talk more about these sketches in the following video:

We thought about an “Advent Calendar” approach, where the user would swipe to open paper doors on multiple-choice items to select their desired response. We also considered a “Scratch-Off” concept, inspired by lottery tickets and scratch-and-sniff stickers, where the user would use their finger to scratch off a response.

Going further afield, we mined the Pogs fad of the 1990s, and cooked up a direction that would involve throwing a “slammer” at an anthropomorphized stack of Pogs in order to select a response. Our interest in Pogs came from a desire to give the user some sort of token as a tangible reward for a correct response. We distilled this concept down into another approach, with Pogs that represent possible responses scattered across a hardwood table. The user would grab the correct response Pog and drag it into a drawer, where they were collecting all of their correct responses.

Finally, we explored a rich metaphor with the natural world, considering a concept where people would interact with a button-based game overlaid on a landscape. As the user answered questions correctly, this world would fill up with small items representing their responses. These items might start as autumn leaves, for example, but as the user answered more questions correctly the world would progress through the seasons, switching to snowflakes, flowers or fireflies. Instead of a conventional timer, the countdown for each individual question would be represented by a rising and setting sun… you’ve run out of time when the moon and stars come out, and another day has passed in your world!

Exploring all possible design directions in these highly generative sessions is an important part of our design process. By keeping the fidelity low and at the sketch level, we are able to entertain a massive number of ideas while still producing a tangible artifact that we can share with other members of the project.