CHI 2013: Autodesk on Data-Driven Design

Written by

Marking Menu: Header

Introducing Marking Menus

Marking, or radial menus are a funny thing. Sometimes seen as a rather proprietary concept, they clearly have their uses and have nonetheless taken strides into the mainstream on touch devices recently.

Autodesk’s Ji-Young Oh set out to present her UX design team’s process used in implementing marking menus in Inventor, their flagship mechanical engineering software. The full presentation is available online, but here’s our thoughts on the work.

Marking Menu: Process

Designing From Data

The initial concept was born from data. This is a common approach we take at Ostmodern - look at existing usage patterns and make a broad set of proposals as to the nature of the product. The trick here is to avoid making assumptions based on data that lock a product down too much.

In the case of Autodesk, their ‘lock-in’ was the marking menu with a single level approach. With data to suggest that the 20-30 top commands take up 80% of usage, it was a fair assumption to make.

The data analysis method is also significant. Using cumulative usage measures, it is possible to extract meaning from what is very noisy but extensive data. However, it’s not without it’s problems.

The data does not represent the entire user workflow.

Workflow is easy to trivialise or simplify, and cannot be analysed on data alone. In our experience with TV, “workflow” means little without context. We’ve seen this unexpected behaviour where favoriting is used in unexpected ways, or as a supplement to someone’s memory.

The initial design based on data must be verified with users.

Based on this common problem, they propose a fair solution that we can all get on board with. Asking users why they did something is far more important than knowing that they did it; it points to a solution, rather than simplifying knowing that something is wrong.

Marking Menu: Graph

Testing real software

Following the initial design of the menus, the team performed informal usability tests every month. With foresight, Autodesk built the menu to support modification using an XML file. This prevented an extra workstream involving the development team during research, and allowed the UX designer to modify the menu directly.

Building a system architecture allowing easy post-launch modifications for research is something that is often overlooked and should be strived for, even in live products. The process of translating ideas from ‘UX thinking’ to ‘build’, and the cost of using development can often rule out changes to products.

However, the team found that there was an inherent overhead in creating production-quality software for research purposes.

Rather costly process since the software at a certain quality had to be delivered to users.

We came across this in our redesign, creating a fully-functioning Django-CMS controlled HTML prototype. There was no doubt that the time spent ensuring the prototype was reliable (and thus believable) was worth it, but it was a costly and high-risk build project.

Risk of disrupting users’ work environment with the work-in-progress software that can cause crashes or loss of user data.

This is a nuanced observation and an important distinction with commercial software. Whilst rapid iterations of design are common in web-based products, traditional software has often yearly iteration cycles.

Where users are paying for a service or - more severely - earning from a product, you change things at your peril. People want new things, but they only want new things if they don’t slow them down. This paradox can make it hard to advance a product, but in this instance, it seems worth it for Autodesk as they note:

Rather than big release each year, deliver smaller chunk of updates more often

Marking Menu: Images

User Feedback

User feedback formed two different branches. In the first, the focus was on pre-beta feedback from a small group of 9 users over a period of time to answer critical questions such as learnability. Our experiences have found that a small number of users will be most reliable when looking for fundamental issues with a product, yet Autodesk found value elsewhere in this user group.

Don’t dismiss the value of testing internal people.

During test period, perform daily ad-hoc meeting to gather “observation nuggets” rather than extensive post-test analysis.

To gather more rounded information, a beta survey followed, with a larger selection of users at 103 respondents. A survey asked users to input on ease of use, layout, learnability, productivity and acceptance. 81% of users preferred the Marking Menu and 65% saw a productivity gain in it.

Despite this feedback, there was a long list of issues, including the steep learning curve, its poor performance compared to hotkeys, and probably most pertinent - to be truly useful, better context sensitivity would be important. This brings to mind a rather pertinent comment in the presentation.

Focus more on number of tests rather than quality of the test itself.

Whilst focusing on this “big finalé” beta test yielded a significant amount of data and feedback, it came in one big lump. This seems counter to Autodesk’s objective to be more agile and perhaps introduces a similar - albeit smaller - problem to the one they highlighted from previous experiences.

Rather than one-off batch delivery of usage data, live and more indepth usage data

In the same way that legacy usage data potentially skewed their UX design backwards to older usage patterns, beta feedback with one final iteration is perhaps too late and too bold.

Rather than Beta test event every year, constantly engage users to receive feedback from the field

A cultural change perhaps for another day, but the Autodesk design team have struck gold. Creating systems that allow persistent gathering of usage behaviour, be it informal 1:1 chats with users, or an analytics framework, could potentially solve many of the pitfalls of this experiment.

The final question - why?

It was important to contextualise the overall aim of the implementation of the menu with the presentation itself. The aim wasn’t to create the ultimate marking menu, but rather to test user response to the existence of a marking menu in the first place.

Talks often unravel into something else entirely at questioning. A burning question came out at the end - why the marking menu in the first place? Well, it turns out, it was a request from corporate.

Marking Menu: Example

At first glance, it feels like big-company bureaucracy. It’s fair to say that a contextual menu could live and die on just that - how contextual it really is. The struggle Autodesk faced was whether there was any logic to developing a complex contextual algorithm when they weren’t even sure if people liked radial menus. That is Agile with a capital A.

Credit once again to Autodesk, Ji-Young Oh, and a reminder that the full presentation is available online.

Ready to talk to us? Interested in visiting?
Contact or
call us on +44 (0)203 215 0041