Development and testing for public labs: “Fake it before you make it”
www.nesta.org.uk/blog/development-and-testing-for-public-labs-fake-it-before-you-make-it/
www.nesta.org.uk/blog/development-and-testing-for-public-labs-fake-it-before-you-make-it/
Issue 7 of Tech4Labs looks at how public labs can develop and test new ideas
During my 8 years at Google as a product manager (PM), I interviewed more than 400 PM candidates. One of my favourite interview questions was about product development and testing.
'Say you are working for McDonald's and are considering a new menu item called McSpaghetti. How do you figure out if people would buy it?'
Most candidates talk about bringing on board a consulting firm for some extensive market research. Way too expensive. Some suggest to pick a few stores and start offering the meal. Still too expensive. Very few consider a more radical approach: add the item on the menu and see what happens; and when people ask for the McSpaghetti, just tell them you are sold out with a big smile and a free basket of french fries.
In this column, we look at tools and methodologies to let your lab develop and test its ideas. In particular we present the concept of ‘pretotyping’ and show it can apply to public sector situations.
There are good examples of public sector initiatives that have been informed by testing to ensure proof of concept before large-scale roll out. For instance, in the UK, a programme to increase the number of people joining the NHS Organ Donor Register used randomised control trials and behavioural economics to test various versions of the marketing message for effectiveness [8].
In the US, the Consumer Financial Protection Bureau (CFPB) used an intensive user-centric approach and A/B testing when designing their loan disclosure form for the Know Before You Owe programme [1, 15].
While it’s important not to conflate private wants (the McSpaghetti example above) with public needs, there are infamous examples that capture the potential value of adopting the pretotyping concept in the public sector. The most recent US example is probably Healthcare.gov [13], where the lack of testing and the initial lack of focus on the user experience almost killed the Affordable Care Act (aka ObamaCare). Likewise, the Aristotle Expert Network - positioned inside the Air Force Research Laboratory, was shut down five years after being launched in 2008 because of budget constraints [7]. In both cases, early testing and cheaper development would likely have identified issues sooner.
No, this is not a typo. The term and concept of pretotyping was originally developed by Alberto Savoia in 2009 while he was working at Google as Engineering Director and Innovation Agitator. The essence of pretotyping is to make sure you are building the right it, before you are building it right.
This sounds obvious but this is often hard because:
We don't want to be criticised
We like to add features
We like to delay launching a product until we think it is perfect
The more time invested on a project, the harder to throw it away – even if it’s failing
The goal of a prototype is to convince ourselves that it can be built and that once built it will work as expected. But this is often the wrong question. What really matters is whether we should build the product/service in the first place and if people will use it if we build it.
The difference between a pretotype, a prototype and a product is illustrated in the Figure 1, taking the Palm Pilot as an example.
Figure 1; source: pretotpying.org
Pretotyping Prototyping Would people be interested in it? Can we build it?
Will people use it as expected?
Will it work as expected? Will people continue to use it? How cheaply can we build it? Will people pay for it? How fast can we make it?The pretotyping 'method' introduces tools to let you fake it before you make it. We present a few (from 17]) and try to provide some concrete examples how they could applied in a public labs context.
A Fake Door is a marketing entry point for something that has not been developed yet [17]. In the digital world, this can be a link on a page, a form, or an online ad. But it can also be a poster, a flyer, a business card pointing to a site, a phone number. The goal is to encourage people to take an action based on what you have to offer. The McSpaghetti is such an example.
In the context of public labs, the Fake Door can be used to advertise a service that is not ready yet and measure interest from users, e.g. a new process to renew a license, a new expert system to consolidate social services, etc. The Nesta LabWorks Practitioner Workshop started as a Fake Door pretotype to gauge interest from participants.
For easy creation of Fake Door websites, you can try KickOffLabs or LaunchRock. For telephony services (voice and/or text), Twilio or Plivo work well. Kickstarter and IndieGogo are good platforms where Fake Door pretotypes are pitched and featured on video. Fake Doors are also good opportunities to crowdsource some elements for your products, via feedback forms or surveys.
A Pinocchio pretotype is one in which a fake artifact acts as a proxy for the real thing [17]. The most famous example is the wood model of the Palm Pilot mentioned above.
Pinocchio pretotypes are the perfect conduit for role play design [23] to encourage people to test your product or service, but also to explore other ways that the "it" you are building is tailored to their needs.
In the context of public labs, the Pinocchio could be a mobile phone app simulating a device that does not exist yet or a situation. At the 2013 Feast on Good Hackathon, a team of innovators used the Pinocchio pretotype to create a 'New Civilian Starter Kit' as part of their Vets for Success project. The Transport for London (TfL) team used cardboard smart phones to test their idea for a new website with end users.
Source = Computer Week
For such pretotypes, 3D printing tools like 3D slash and cardboard prototyping [21, 22] offer easy and cheap approaches.
A Mechanical Turk pretotype simulates a sophisticated technology that would be costly or time-consuming to build from scratch, using human power to substitute for the technology [17]. The story of IBM testing yet-to-be-implemented speech recognition technology by having hidden typists is such an example [24].
Very often public labs will be tasked with redesigning public services which have an in-built expert system or "personal advisor" dimension, such as:
for financial empowerment and advice, e.g. financial aid, student loan, debt refinancing
for career orientation, e.g. education opportunities for veterans, certification
for access to social benefits, e.g. housing, healthcare
Algorithmic-based solutions are needed to reach scale while keeping the cost low. But before finding the right "advisor" to build, a Mechanical Turk approach can be used to experiment, where human experts can hide behind an online form or a SMS-based application.
The One Night Stand pretotype is a model in which an interactive product or service experience is presented in a full-featured fashion, but without the infrastructure that a permanent solution would require [17].
In the context of public labs, this could mean a pilot restricted to a given area or a given population. The goal here is to reduce cost or to target a well-known population. In the spirit of failing fast, if the One Night Stand pretotype applied to a friendly region and receptive population does not succeed, then you know that you are probably building the wrong "it".
Drones (e.g. Parrot), portable Wifi access points (e.g. BRCK), portable solar panels (e.g. PWRstation) can be really cheap ways to simulate "service coverage" (in a broad sense). The TU Delft Ambulance drone is a good example.
An Impersonator pretotype is one where an existing product or service gets a new wrapper or “skin” in order to pose as the new offer under test [17]. A famous example is Tesla who used the body of the Lotus Elise and replaced the engine by its electric engine to go and test it with people.
In the context of public labs, this could mean revamping an existing service by providing a different front-end on top of an existing API.
Capturing which core components of the pretotype testing are most effective (and reflective of users’ needs), will help you hone in on your An MVP - the key feature of the MVP is that the artifact is the simplest possible prototype, stripped down to the bare minimum required to accomplish the live test [17]. Doing so signals is the transition from pretotyping to the prototyping of the eventual product. The key feature of the MVP is that the artifact is the simplest possible prototype, stripped down to the bare minimum required to accomplish the live test [17]. This is often the next step of product development and testing where real interaction with a real prototype is required.
We have already covered the MVP concept in more detail in our previous issue, 'Canvassing' the value proposition of your public lab project’.
Testing and fast development is not restricted to pretotyping. Here are some other useful tools .
For tracking user behaviours, Google Analytics, MixPanel, ChartBeat or Flurry are solid solutions, be it for website or mobile apps.
For a more in-depth view of how users are using your product or service, tools like UserTesting and FullStory let you record their interaction as a video. GoPro cameras are also a good solution.
A/B experiments can be performed using tools like Optimizely or Unbounce.
No matter what tool you use, make sure that you identify the right metrics to measure and do the testing. See [20] for the set of metrics Google is using for its products.
Most of the tools and techniques mentioned above originated in the private sector and should be applicable to the public sector space. While most can be applied without issue, there are a few caveats worth mentioning:
Unlike some consumer products (pick your favourite cat picture search application), public service product or services have real consequences on people. Therefore, when using the Fake Door pretotype, make sure you correctly manage expectations.
Beware of the 80/20 rule. As Chris Mears notes: 'Edge cases at a scale of the whole population quickly do not become edge cases any more. The 80/20 rule still applies, but you really have to think about how to deal with that 20 in a way you may not in the private sector.'
Consumer products often use and abuse non-intuitive language, whether in the form or legalese or filler text (e.g. Lorem Ipsum, being replaced by Minions Ipsum). For public service product, content matters. “Content is always important, but in the public sector the impacts of bad content can be severe.” as Chris Mears reminds us. MindLab found language played a significant barrier to service uptake for the Danish Customs and Tax Administration’s online digital platform, when citizens were faced with terms such as “employer-administered pension capital”. Effective public pretotypes must therefore place a premium on creating comprehensible content built around intuitive language.
Implicit in the development of many consumer products is the assumption that the user will have time to enjoy and use the developed products. For some, this is not the case. Even free stuff is never free and it takes some time to consume and digest, aka the bandwidth tax. This is one of the key messages from "Scarcity: Why Having Too Little Means So Much".
'While education is undoubtedly a good thing, we treat it as if it comes with no price tag for the poor. [...] But in fact, bandwidth comes at a high cost: either the person will not focus, and our effort will have been in vain, or he will focus, but then there is a bandwidth tax to pay.'
- Scarcity: Why Having Too Little Means So Much
As Geoff Mulgan from Nesta has said, 'No plans survive their first encounters with reality intact.' In the context of your public lab, your challenge is to shorten the time it takes for that encounter to take place – thereby minimising the cost and future proofing your idea.
While we have acknowledged the inherent caveats in treating public and private sector product development and testing as exactly the same, the public sector can learn a lot from the private sector.
Introducing earlier testing into the design and development of innovative solutions can help uncover underlying issues sooner, and demonstrate proof of concept that your idea will work and therefore should be built.
For the public lab, this can make or break how successful the lab is at addressing a given public problem. And it is the point where labs can and should seize their licence to take risks, being bold enough to put something out there for people to start using, even if it’s not polished, not totally ready. This is not something we usually do willingly.
Alberto Savoia (father of pretotyping) and Reid Hoffman (founder of LinkedIn) formulate this ego-element way better than I would.
'The toughest thing about pretotyping is not developing pretotypes, that’s the fun part. The tough part is getting over our compulsion for premature perfectionism and our desire to add more features, or content, before releasing the first version. The tough part is getting our pretotypes in front of people, where they will be judged, criticised and – possibly – rejected.'
- Alberto Savoia
'If you are not embarrassed by the first version of your product, you’ve launched too late.'
- Reid Hoffman, LinkedIn
To continue the conversation, and to offer feedback and suggestions on the tools above:
Share your comments
On Twitter using #Tech4Labs
Collaboration for the greater good (slides), CFPB.
How to do UX in Government Without Losing Your Freaking Mind (slides), GSA.
http://www.usability.gov
User testing tools and services: Usertesting.com and Fullstory.com
Why you should give public sector UX a try, by Chris Mears, March 2015.
Bridging the Knowledge Gap: In Search of Expertise, by Beth Noveck, 2014.
Applying Behavioural Insights to Organ Donation, UK Cabinet Office, Behavioural Insights Team.
Why governments need guinea pigs for policies, The Guardian, March 2015.
Testing GOV.UK with real users, GDS, Oct 2012.
Scarcity: Why Having Too Little Means So Much, by Sendhil Mullainathan and Eldar Shafir, 2013.
Why do projects fail, website.
The HealthCare.gov Problem That Could Have Been Avoided: The User Experience, Oct 2013.
Result Of Improper Implementation Of UX Methodology In Projects, Sep 2013.
Know Before You Owe, CFPB.
Pretotype It, by Alberto Savoia. (PDF)
Pretotyping@Work, by Jeremy Clark. (PDF)
Pretotyping in the UK, http://www.pretotype.co.uk.
Role play design tool, Service Design Tools web site.
Human-computer Interaction and Management Information Systems: Foundations, Yahong Zhang, Dennis F. Galletta, 2014. (book)
Photo Credit: Samuel Mann via Compfight cc
<!--/*--><![CDATA[/* ><!--*/ #mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; } /* Add your own MailChimp form style overrides in your site stylesheet or in this style block. We recommend moving this block and the preceding CSS link to the HEAD of your HTML file. */ /*--><!]]>*/
* indicates required
Email Address *
First Name
Last Name
Organisation
Position
Country
Email Format
