Keep The Noise Down

Below the Surface

I regularly get asked the difference in designing for gesture-based interfaces.  I recently addressed this in an article for .net Magazine.  This may appear somewhat lazy but why rewrite the written? So here’s a reproduction of the article:-

With a wave of gestural mobile devices and gaming systems flooding the market, more and more people are interacting directly with digital content - click is fast being replaced by tap, flick and pinch.

Microsoft recently announced Project Natal, a sensor-based interface that responds to contact-free gestures such as a wave and a nod. And while we eagerly await this technology to come to market there’s plenty to be excited about already with the launch of Microsoft Surface.

Surface incorporates the naturalness of hand-gesture recognition but also blends reality and virtual reality by enabling users to interact with digital content using ‘real’ physical objects. From a design perspective there are a lot of new principles to master. Designing a Surface experience is not the same as designing for traditional websites.

The key is to avoid retrofitting existing experiences and introduce new ways of thinking that enable new interaction models. For those looking to get involved in this new era of interaction here are some pointers:

Make it finger-friendly

Fingers don¹t have the same level of precision as a mouse, so make sure all interactive elements are sized and spaced to optimise gestural interactions.

Encourage active exploration

Don’t make users memorize how to do things. Make the experience discoverable and reward the user for exploring. Always nudge and entice users to do more.

Enhance the user’s sensory experience

Don’t just focus on visual experiences. Surface doesn't just encourage but mandates a fuller sensory. Use sound to reinforce user interactions and create fuller sensory experiences.

Encourage social participation

Surface moves beyond solitary desktop experiences. Design your application to invite people to join in and encourage person-to-person interactions. Consider the challenges of multi-stranger versus multi-friends scenarios and different levels of task coupling.  For example, how do the actions of one user affect the other?  What happens when new people approach the table from a different side?

Make it alive

Ensure every action has an immediate response, regardless of how subtle this feedback is.  Even if no-one is interacting, don’t let the Surface device sit there lifeless.

Make it magical

Add a certain amount of realism but then augment it further with something that doesn’t exist in the real world. For example, use real-world behaviour such as acceleration but then disrupt the physics in a way that only a digital experience can.

Make the interface invisible

In a Surface experience, the interface shouldn’t be visible as the content becomes the interface. Design an interface-free application that enables users to directly manipulate the digital content.

Remember the enjoyment factor

Playfulness can manifest itself in many forms, such as through copy or interaction design, not to forget playfulness through interaction with other people. Don’t ignore this important dimension in designing engaging and immersive experiences.

 

Where do we go from here?

Different gestural interfaces will bring different design philosophies and with this fresh and exciting challenges. Erase your mind to a world without GUIs and design with no preconceptions of interactions models that have been. Don’t underestimate what appear to be simple applications - often the simplicity masks the complexity.

 

For those interested in learning more about Microsoft Surface then I recommend grabbing a copy of the October edition of .net magazine.  In the Microsoft Expression Supplement are some excellent articles including a 'Guided Tour of Expression' by Paul Dawson, a guide to 'Building a Prototype using SketchFlow' and many more interesting articles...

“The ear is totally neglected. Everything is centred on the eye”

I borrowed this quote from Daniel Barenboim’s talk on the Nature and Power of Music. Most technological experiences, especially online, have been primarily visual experiences.  The web gives us a visual workout but few online experiences stimulate our hearing to create fuller sensory experiences.  With the Microsoft Surface interaction paradigm it’s time to prepare for an audio assault as sound is getting the consideration it deserves.  Interaction Designers are digging deep in to their auditory and visual toolkit as sound gets crafted in to the Surface experience. The ear is back in vogue.

OK, so the gaming industry has been treating our senses to full immersive experiences for some time but most other technological experiences have relied heavily on visual alone. 

Surface doesn't just encourage but mandates a fuller sensory experience.  But are agencies ready to build auditory experiences and are clients ready to pay for sound design?

If you can communicate the return on investing in sound design to your clients then they will pay for it. Sound is not there just to titillate the senses.  It is used to enhance a brand, stir emotion, create immersive experiences and communicate information. When users manipulates digital content, information is revealed through sound, visual feedback and movement. Sound is not simply about providing a fuller sensory experience but acts as another information layer.

I recently met with Andrew Diey, Founder & CEO of Radium Audio, a leading music and sound design company that have experience in designing sound solutions for Surface experiences. Does Andrew feel that clients are ready?

Andrew doesn’t see such digital auditory experiences as a huge leap.  Advertisers know the importance of sound to enhance a brand.  This is not about introducing something new but transferring a proven art to a new paradigm.  Andrew believes clients are ready but it’s down to agencies to communicate the value of sound. Andrew suggested agencies establish a manifesto which defines the terms on which they approach immersive experience design.  The manifesto should outline principles, policy and action plans for future work in immersive experiences.  Most agencies have a design manifesto but does it include sound?  If you want your client to buy in to sound design then you need to plan for it from the beginning.  The task of getting your client to understand the value depends on your belief and recognition that sound is critical to the delivery of a full sensory experience.

Are agencies ready?  Yes.  It hasn’t been about not having the sound design know-how but more so about not having an environment where sound design is treated with the same importance as visual design.  Most agencies that specialise is online solutions may not be experienced in sound but with experienced sound designers such as Radium Audio there is no reason for agencies not to be ready. Sound is part of an interaction designers domain to define audio feedback that complements the interaction and visual design. The brief is then given to the sound designer, internal or external, to weave their  magic and create seamless experiences between the audio and interaction.

At EMC Conchango we have in-house sound design capabilities but we are not professional Sound Designers. For superior production and bespoke sound design we look to work with professional sound designers. We employ experienced Art Directors and Creative Designers to achieve our creative vision.  Our approach to audio is the same.  To craft an immersive auditory experience we need to work with the best.

So there is no reason why we should not be entering a new sensory era of compelling and immersive experience that combine sight and sound. It’s now up to us to be heard...

Designing NUI Experiences

A few months back I posted a collection of Microsoft Surface tips from the Twitter hashtag #Surfacetip. The 140 character limit is somewhat challenging so I'm planning to tackle these tips one-by-one in a tad more detail.

I’ll start with a simple tip, but one that resonates with me from recent projects, and that is ensuring that everyone is on the same page when it comes down to a common understanding of what a Surface experience is.

Tip #1 - Kick off every Microsoft Surface project with a half-day Play & Tell Workshop.

A challenge of natural user interface design is that this is a new paradigm in interface design and development. Surface invites a different way of interacting and manipulating content. This is not a subtle change but a significant change in the way we interface with digital content. For many folk this is a significant paradigm shift so don't expect everyone to take to it so easily.  Years of desktop and web applications means some people are understandably institutionalized and will take time to fully down their web way of thinking and enter this brave new world.

Another point is that although multi-touch has been around for several decades we are only now seeing mainstream adoption. The iPhone has accelerated the interest in multi-touch and the mobile market has been busy satisfying consumer demand for multi-touch and gesture-based computing. Although many of us appear familiar with multi-touch, there is still an element of novelty to these gestural interfaces which can mean that other features get overlooked.  And a Surface experience is much more than this. There are many new concepts to understand in Surface design such as multi-friends vs. multi-stranger experiences, collaboration, playful design, social interaction, object recognition, superrealism and 360-degree user interfaces.

And this is where the risk lies. If the team (client and internal project team) don't kick-start the project with a common understanding of the Microsoft Surface philosophy then the risk is that downstream you'll find yourself delivering an application that is not optimised for the Surface computing paradigm. 

The purpose of the Play & Tell Workshop is to inform the team of this new interaction paradigm through play and discussion. It's a great activity to actually kick off the project as the Surface device plays to its strengths of encouraging social interaction. The objective is to ensure that you share the same vision of a Surface experience.  This session is facilitated by a practitioner of multi-touch design. This should be a fun, exploratory session but the practitioner needs to articulate the features of the Surface experience. Some of the subtleties of good design need to be explained as these intuitive, natural and engaging experiences can mask the complexities and thought process that has gone in to them.

Our clients pay us to give them what they want but also for our expertise to guide them and help them make informed decisions. The Play & Tell Workshop kick-starts the project with a common understanding of the principles of Surface computing.  These principles must be kept in mind during the entire project to deliver an experience which is true to the spirit of Microsoft Surface.

 

If anyone wants to suggest further tips then please use the hashtag #Surfacetip

Keep the Gamers Happy

Last week Hungry Hippo came to Microsoft Surface.  As a child I munched my way through many a white plastic marble so I am fascinated to see how this classic children’s board game takes to a digital makeover. A number of puzzles, board games and arcade games have already been given the Surface treatment. Last month the Microsoft Surface Games Pack brought some classics to this interactive table, along with the Tetris-like addiction of the all-new Tiles game. These games demonstrate some of the amazing features of Microsoft Surface; multi-touch, multi-user, gestural experiences which are playful, rich and immersive.

Touch is important as consumers ditch controls in favour of manipulating digital content using natural gestures. It's early days for touch and there is understandably a novelty element around gestural interfaces so despite many of the games being relatively basic, they are still engaging the users. As we become more familiar with multi touch we won't be so easily satisfied and will demand more.

Gestural interfaces are the future, and wherever the future lies you'll always find gamers. Microsoft Surface is not intended as a gaming device but emerging technologies such as Surface give gamers a taste of where gaming can go.  Gamers are demanding types with an appetite for innovation so like many of us they want to squeeze the most out of these technologies. And that takes me to the point of this ramble. During demonstrations when I'm illustrating the features of Microsoft Surface I am regularly asked why none of the games utilize object recognition, why Microsoft Surface’s most compelling feature, and the feature that sets Microsoft Surface apart from other touch screen technologies is not used in any of the games so far?

Multi touch devices recognize multiple simultaneous touch points. Natural user interfaces enables users to interact in a far more natural and intuitive manner.  Yet Microsoft Surface offers more than this.  It houses five cameras inside the Surface unit which reads information from data (domino) tags and triggers events when physical objects are placed on the surface. This is about recognising the object, not just detecting the presence of an object. We have seen object recognition used to good effect in many other Surface applications, but not gaming. Surface has its limitations. It can't detect finger pressure and it doesn’t provide haptic feedback. Surface needs to play to its strengths and one of these is object recognition.

I'm not advocating we implement frivolous object recognition as it needs to enhance the experience. So whether it is as simple as an event occurring on the outcome of a throw of a dice or a playing card being placed on the table the object recognition must add to the experience. Take a look at Identity Mine’s Air Hockey.  This is a great illustration of natural gestures and showcases the magic of blending physical and virtual worlds, not to mention achieving complex events such as collisions and acceleration. The success of this game is in its simplicity and naturalness, but this is a game that clearly wouldn't benefit from object recognition.

Educational games lend themselves well to Surface and we're already seeing some interesting applications in this area.  Object recognition lends itself so well to child development as the technology can provide a richer sensory environment. For non-educational games we’re not seeing the same uptake. The absence of object recognition is not a criticism of Microsoft Surface or the talented designers out there but aren't we missing an opportunity to demonstrate the role of object recognition in interactions of the future. 

I know that there are plenty of creative minds out there so this is a challenge to the Surface community to conceive a gaming experience which benefits from object recognition.

If you have any suggestions on games or gaming events that would benefit from object recognition please comment on this post or post a tweet to #surfacegame.

Surface Computers as Interactive Learning Platform for Kinesthetic Learners

People learn in different ways. We are unique learners with our own learning style. Some learners have a dominant style whilst others display a mix of learning styles which they call upon in different situations. 

 

• Visual learners digest information by reading and observing. 

• Auditory learners process information through talking and listening. 

• Kinesthetic learners are doers and favour a hands-on approach. 
  

 

It's important to know how we process information best and learn 'our way'.  Traditional teaching methods often fail to support the learning style of kinesthetic learners, yet these learners make up somewhere between 5-15% of the population.  They are doers that work best when they're hands on and engaged in motor activity.  Unfortunately, some kinesthetic learners are mislabelled as ADHD. They are seen as hyperactive, difficult and even disruptive when in fact they just learn differently.

 

I'm no behavioural psychologist or education expert but am hopefully suitably informed to recognise the potential of technologies such as Microsoft Surface to offer immersive educational experiences which aid learning for tactile/kinesthetic learners. A typical classroom setting is not great for tactile/ kinesthetic learners.  These multitouch, multiuser interactive learning devices offer a better match and it's only time before they realise their potential and go on to play an important role in education for people who are kinesthetic-oriented?

 

 

What do these Interactive Learning Devices offer for Kinesthetic Learners?

 

Learn by doing: Kinesthetic users can learn from first hand experiences.

 

Learn through touch: Surface computing is a touchy-feely technology.  These natural user interfaces respond to natural gestures such as a tap, flick and pinch. Let the hands and fingers do the talking.  The tactile nature of a Microsoft Surface experience is not limited to the surface display but extends to the touch of physical objects which are recognised when they are placed on the surface.

 

Learn through movement: These active learners have an environment which supports their need to be involved physically. Kinesthetics take in information best when they are moving.  Surface devices don't follow the desk-and-chair paradigm so kinesthetic learners aren't tied to sitting down in one place while they interact. 

 

Learn through engagement: These highly interactive devices incorporate imagery, video, and audio to provide immersive experiences where the learner can interact with and manipulate content.

 

Learn through participation: Kinesthetic learners participate in single user or multi user experiences through digital play.

 

Clearly Microsoft Surface, the SMART Table and X-Desk are devices that provide a learning environment to help kinesthetic learners succeed in a learning system where otherwise they may be left behind. Let the development begin...

 

Related reading: Digital Play as an Alternative to Drugs

Interaction Revolution

Imagine a digital utopia where designers are free from grid-based design, and precision mouse control & clicks are replaced by gestural mechanisms. Where imagination is not exclusively for the mavericks to throw a curve ball at what’s achievable. Where we move away from the chair-and-desk paradigm and the frightening portrait WALL-E depicts of a human race whose only physical movement is to lift a straw to their lips.  This is the new paradigm – a natural, active and social paradise where reality and virtual reality blend seamlessly with magical results.

Well, not quite.  But there is plenty to savour from a paradigm that incorporates naturalness, encourages social interaction, and brings playfulness to the party.  We are living in interesting times.  Consider multi-touch, these devices offer a different way of interacting and manipulating content.  This is not a subtle change but a revolution in the way we interface with digital content.

Many people are getting their first taste of natural user interfaces and there is an understandable buzz.  The Web 2.0 movement harnessed collective intelligence built on a framework of creativity, collaboration, group thinking and sharing. This was a positive step forward for digital experiences.  Web 2.0 made us feel good about ourselves for being more social while all the time we sat staring at a monitor, engaging in less and less face-to-face contact and developing a distorted view of social relationships. So how do we take another step closer to a digital utopia?

It was with several recent Microsoft Surface projects that I had a eureka moment and realised the possibilities of this new interaction paradigm. As the projects threw up questions, it was the subject of these that made me aware of the dramatic changes ahead: -

·         Does the application work for single user, multi-friends and multi-stranger experiences?

·         Do we have the correct balance of playfulness?

·         Is it an exploratory experience?

·         Does the application have a pulse?

·         What is our social design strategy?

·         and so on...

These were fascinating questions to be asking about any application, even more so an enterprise application.

Natural user interfaces have been around for several decades but we are finally seeing mainstream adoption. The rate at which multi-touch applications is forecasted to grow over the next 3 years means that what are currently a small percentage of designers working on multi-touch applications will increase rapidly as there will be an ever-growing appetite for multi-touch solutions. The iPhone has accelerated the interest in multi-touch, no longer niche, and with Windows 7 multi-touch will truly come of age.  If you’re not working with multi-touch devices or NUI’s then does that matter?  No!  It's less about the technology driving our thinking but more about our thinking driving the experiences. The interaction revolution is as much about the evolution of the way we think about digital experiences. It’s exciting to ask these questions of the new breed of digital experiences.   If somehow we can answer some of these questions then we’ll take a huge leap forward in creating compelling and enjoyable experiences.

 

 

Related reading - Playfulness in a Business Context

Breaking down traditional barriers

Isobel has just turned 4 and uses a computer mouse to (attempt to) navigate sites such as CBeebies.  She finds the mouse challenging and frustrating.  At such a young age her motor movement abilities are still developing.  Combine the motor skills required with spatial concepts necessary to effectively use the mouse and it’s easy to understand Bel’s frustration when she tries to click the Charlie and Lola thumbnail but instead finds herself watching Finley the Fire Engine. 

Bel is slowly improving her mouse control but it’s taking time.  While Bel’s thoughts are concentrated on the mastering of the mouse and the precision control required to click the tiny thumbnails or links, her experience of the Cbeebies web site is somewhat reduced.

Interactions such as drag and drop create more frustrations as Bel struggles to keep the pressure on the mouse button.

I thought it would be interesting to see how Isobel finds an interface that she can interact with pretty much as she interacts with everyday objects.  So yesterday Bel came to our Conchango studio so I could see how she handles Microsoft Surface.  I didn’t give her any warning that she was about to become a guinea pig for me to see how she would interact with the tabletop using touch and gesture.  Not a mouse in site.

As Bel stood next to the interactive tabletop I subtly introduced her to the water attract screensaver, being careful not to describe the unit as a computer to see whether the ease of direct digital manipulation would disguise the technology.

Isobel was immediately captivated by the water simulation that creates ripples when touched.  In no time she was hands-on.  There weren’t the usual oohs and aahs as she played with the water but if you remove the technology from this experience then what is so amazing about it?  To me Microsoft Surface is a complex mix of hardware and software, but to a 4 year old the technology is invisible - it’s not much more than a table that she can do some interesting stuff on.

I pointed to one of the access points, Bel reached out and tapped it.   Next, I scrolled through the demo applications.  Instinctively Bel reached out and copied my actions scrolling through the application with surprising ease.  Bel did not display any of the hesitancy she normally displays in digital-situations but much more the confidence of a non-digital situation.

We then spent the next 30 minutes working our way through the photo app, the music app, photo paint, the piano and Identity Mine’s air hockey.

I preloaded the photo application with photos of family & friends and some of Bel’s favourite TV characters. Isobel instinctively started to explore and manipulate the photos.  She found selecting, moving and orientating the photos very intuitive using natural gestures. She took a little guidance to get a handle on photo resizing but by the time she’d played for 15 minutes she had got to grips with it and could shrink and enlarge the photos with relative ease.

 

With the surface piano Bel had no problems playing the usual tuneless racket that she has mastered so well on her grandma’s piano. Air Hockey, a game she isn’t familiar with, required no introduction to as she demonstrated her dad’s competitive streak with a spirited performance against Matt.

Photo paint took a few minutes for Bel to master as her paint strokes caused the canvas to move but soon she got the hang of this as she created her masterpiece.  An amusing observation was how Isobel wiped her painting finger on her trousers each time before she dipped her finger in the virtual paint palette.  This highlights how the direct manipulation and natural gestures of Microsoft Surface can blur the real and virtual world.

 

The experience was certainly memorable for Bel who listed all her friends at nursery that she is going to tell about Microsoft Surface. Isobel found the whole Surface experience fun and easy to use and the touch and natural gestures overcame the challenges of normal PC operations.   I now have some explaining to do as to why Father Christmas can’t fit a Surface unit on his sleigh.

Related posts:

Surface Computers as Interactive Learning Platform for Kinesthetic Learners

Digital Play as an Alternative to Drugs

Read more at Wandy's Blog or connect with me on Twitter

P.S. EMC Consulting are currently recruiting.  If any of my ramblings interest you and you are passionate about User Experience then we'd like to hear from you.  Why not contact me and let's have a chat?