View on GitHub

Build-Low-Cost-User-Research-Lab

Why did we build the user research studio?

The main reasons we built the user research studio was to increase the amount of research with participants and reduce the amount of spending on commercial research studios.

What are the benefits of having our own studio?

The main benefits of building our own research studio are:

Research about research studios!

We contacted and visited a number of government and commercial research studios. These ranged from small ad-hoc labs to academic labs to high-end expensive studios in Internet companies.

We learned a lot from all of these labs and are very grateful to the people who shared their expertise with us.

What did we learn - our options

After carrying out this research, we learned a lot of things. We had 5 main options, ranging in facilities, and cost.

Option 1: Do nothing

This would mean continuing to use external research studios for our usability testing sessions - a mix of commercial studio and GDS research studio time.

Costs of this approach

This would have avoided spending on any equipment and building space. This could be seen as the main benefit.

Drawbacks of this approach

However the main drawback would be, it would have meant continuing to pay for commercial space, and trying to get time in the very busy GDS facility. The cost for this would increase as more research with participants was being carried out.

Another considerable drawback would be projects would still need to find budget for ongoing research. Providing an internal studio would mean providing facilities at lower cost.

Another would be at busy times it would have meant possibly not finding availability in the already busy studios.

Option 2: Build a basic lab

This would mean building our own lab, probably in a Home Office building somewhere, or maybe on another departments buildings, if we explore the collaboration option.

Costs of this approach

From research into modern commercial off the shelf equipment, the cost here is under £10,000. This would heavily reduce our dependence on external labs, allow us to carry out more research and save on expenses.

The philosophy of this approach would be the researchers doing the research would manage the lab themselves. This may not be possible for all, and so may need someone to manage the lab.

This would allow us to avoid most, or possibly our entire annual commercial lab spends.

Main benefits:

Drawbacks of this approach

Option 3: An Intermediate Lab

This level of lab would be a more sophisticated setup, incorporating dedicated hardware audio and video mixers, a number of cameras and screens and supporting ancillary equipment.

While not as high spec as some of the very expensive labs, this option should do most of what researchers need, most of the time and should allow for the use of external, commercial labs to be all but eliminated.

However, it would have higher associated setup and running costs and may be technically more difficult to use than the simpler solutions or using existing lab space.

Costs of this approach

From carrying out research with commercial studios and other government studios this approach could cost approx. £40,000. It would also require ongoing support.

Benefits of this approach

Option 4: High quality, professional equipment Lab

A commercially built professional studio can cost more than £100,000 (as of may 2016) http://rikwilliams.net/ux/user-research/user-research-labs-designing-building-managing/

https://gds.blog.gov.uk/2014/12/03/our-gds-user-research-lab-is-6-months-old/

This solution represents the most highly specified lab, it will have multiple tilt and pan cameras,

HD footage, large viewers, different devices, and likely storage and editing facilities. At the

lower end is the Google style bespoke solution, still built around dedicated hardware mixers and

HD equipment, ranging up to the DMS full solution, which is on G-Cloud. Although this is likely

to be as much lab as anyone in government could ever want, there must be some debate about

whether it is too feature rich for the cost. In addition, setting up streaming and support is likely to

be very expensive for these options. The actual cost will likely range from between £80,000 to

£250,000 but realistically we’d probably expect to be paying c. £160,000 plus support and

streaming costs for this.

Please note, on the financial modelling tables, I have divided this into two categories, an

advanced lab, which is approximately equal to what Google have, i.e. mostly a hardware

solution, and a de luxe lab, which is essentially the latest DMS offer as at DVLA / Tesco and

The Guardian. We can see that this affects how quickly the investment can repay itself.

Main benefits:

Main drawbacks

Option 5: Collaboration

This works with any option between 2 and 4, but is obviously particularly well suited to options 3 and 4, it would reduce many of the risks, give us more accommodation options and allow us to share the costs.

On the downside, if we collaborate, we reduce our dedicated lab time.

However, it may just be too long and slow to go down this route if we need the lab within the relevant time.

What we did

Concerning the two viable options there are a few remarks to be made about recommending one or the other. In essence, it is a conflict between features and affordability.

Option 2 will give us the minimum specification, it will fulfill the lowest specification requirements and because it is fairly cheap, it will give some security over the financial envelope should, for whatever unforeseen reason, the amount of costs being avoided by in-sourcing lab tests fail to materialise in the real world.

It therefore poses the least risk and as an additional benefit, it will be less complex and more user friendly than the other do something options and may obviate the need to recruit a full-time studio manager.

On the downside, the level of facilities that this can offer is barely above what we might be able to get from just setting up a couple of laptops and cameras on an extemporary basis and has considerable technical risks around drivers and stability issues. Also, this would leave little budget for important other features like large screens in the observation area which is so critical for engaging stakeholders, leadership teams and project workers. Moreover, the quality of the footage captured is likely to be lower than other options and run the risk of not providing the best quality research to support developing and improving services. There must be some concerns too, over whether the quality produced by this solution would be sufficient to allow projects to avoid using external labs. If it is not then we lost a key justification for pursuing this option.

Option 3 will give us good quality feeds and the ability to install more cameras and sources. It should provide adequate quality for most researcher needs and yet still be highly affordable and pay for itself within the likely lifetime of the equipment. It would also allow us to have some of the other useful equipment in the observation area like large screen TVs, storage and other facilities as well as being more generous in terms of allowing for other spending on tools like accessibility solutions.

As a more highly specified lab, this would also be more capable of appealing to external users, such as other Government departments and agencies. This sort of arrangement might be amenable to recharging, so this configuration could also generate a better income stream, in principle. It also gives us a chance to prove that a lower specified lab can provide most of what user researchers require at a lower price point, which could showcase a way of saving money for other agencies.

There are drawbacks, it clearly won’t pay for itself as quickly as the cheaper option and represents a higher initial cost of investment. It’s not as much of a one stop shop solution as the larger lab specs provide and this still might introduce the risks of having to go external on some occasions, though this risk is much smaller than with the previous option.

As a more sophisticated lab it is likely that it would require more technical support at greater expense and also that it may require more personnel intervention, for instance, if no lab supervisor, we might need some volunteer(s) to support the lab in a local expert role.