UX Case Study

Play Testing Pipeline Setup

Goal

A play testing pipeline for game developer Mighty Kingdom to help integrate player feedback into the development cycle. The goal was also to help the team get the answers they needed to drive their decision-making, while creating a scalable play testing blueprint.

My Role

I spearheaded the creation and setup of the new play testing pipeline to bring structure to the data collection methods used. I was responsible for the testing of the process, planning, iteration and workshopping of changes required. I also helped to define the steps to make it reusable for other titles in our catalogue.

Star Trek Lower Decks: The Badgey Directive was one of our mobile games that used our play testing pipeline

Research & Setup

To create the most value we collaborated with everyone from QA to design. Through those connections and the team’s requirements, we helped to introduce new functions.

Facilitation

We introduced a clear role for a facilitator in moderated play tests. Especially for multiplayer games this was important to ensure sessions ran smoothly as wrangling multiple players is no mean feat.

Player screening and recruitment processes

External play testers were not yet being used when I arrived. So part of the work involved setting up the recruitment portion of the pipeline. Connecting with play testers, screening them and scheduling them into play testing appointments was all part of this step. Collaborating with the talent team we were able to set up a reliable testing pool to draw from.

Play testing objectives

Establishing a purpose for the play tests was an important step and something that the design teams had never done in the past. Collaborating with the game designers, we were able to uncover their needs and questions and prioritise them based on risk. This process gave us a way of measuring the outcomes of the play testing tasks.

QA Accountability

Ensuring that QA-approved builds were provided well in advance, along with any bug reports that might affect the outcome of the play test. This helped us form a close bond with QA and clear visibility on build stability leading into play tests.

Observation channel setup

A variety of channels were set up to observe players. Discord was used for multiplayer games to watch multiple players at the same time. Game designers or programmers could join inconspicuously to see first hand what the players were doing and saying.

Data collation

Data collected from play testing sessions was gathered using Miro boards. With clearly defined objectives for play testing, any noise within the feedback was easier to identify and discard. Trends were also much easier to spot, leading to better data quality.

Play testing retros

After completing play tests the UX team held retrospectives to see what we were doing well and what we could improve on. It was a great way of iteratively improving on our play testing pipeline.

Tech support

Despite the QA support for our builds, there was always room for error. Having a support team member on standby to assist players was imperative to ensuring tests were not derailed when anything went wrong.

A multi-platform testing blueprint

Versatility

We tested everything from PC to mobile to wearables. Including the Google FitBit Ace for kids. Our testing solution allowed us to gather data in a robust way, no matter what platform we were creating for.

We play tested games for Google’s FitBit Ace wearable for kids.

Featured Challenge

Our largest project was a 16 person multiplayer session

Our new process was tested to its limits. We were required to recruit a large number of external players, all of whom were remote and were required to overcome any tech issues in real time. There was an exponential risk of play tests failing due to the number of steps involved: multiple players’ punctuality, machines with a variety of specs running builds remotely, etc. However these play tests often ran smoothly due to our contingency planning of having internal players ready on standby to jump in for anyone who was having issues. This allowed tests to be completed and value to still be gleaned from the sessions.

Our original title Hyper Rush was the multiplayer game we tested.

Failures

Learning from our mistakes

There were multiple failures that helped us improve our play testing pipeline.

  1. Tech problems. The previous play testing system was plagued by tech issues. Sometimes causing play tests to completely fail, losing the opportunity to gather any data. With the testing of a 16-player multiplayer game there was always going to be scope for errors. We had issues with multiplayer games connecting to servers reliably as well as the infamous “Plan C” episode that required us to use our 3rd choice machine to get the test going.

  2. Team participation. As one of our games was multiplayer, this required real players in lieu of bots (as these were yet to be developed). With the regular requirement for 7-15 internal players to join, this was heavy on resources and the internal team did end up getting “play test fatigue” due to the regular cadence.

Power Rangers: Mighty Force for mobile was testing using our play testing system

Outcome

We carried out more than 100 play tests over a 3 year period

Outcomes

Better quality data and a reusable blueprint

Our process ultimately yielded excellent results for the team including:

  1. More accurate diagnosis of player issues for game designers. The team could now pinpoint the exact cause of their issues and think of solutions to measure in the next play testing session.

  2. Actionable outcomes that inform next steps. All play tests resulted in clear pathways on how to improve the game.

  3. Increased accountability for game dev teams with regular play testing cadence. Teams knew when the game was next going to be tested due to the play testing schedule we compiled. This gave a firm date by which changes needed to be made if they were to be tested.

  4. Scalable framework for future play testing at the company. All future play tests were set up and executed at a much faster pace due to the foundational planning work carried out early in the process.

  5. Increased ability to showcase games via live play test sessions to potential partners. Potential partners/publishers were able to use this setup and get involved by playing games with the team. This led to a more engaging product showcase and a gave them a clearer idea of what to expect.

  6. Increased visibility of UX value to the team. A nice by-product of this was that we were able to highlight the value we brought to the development cycle. This in turn communicated the value of UX to the team and allowed us to get involved at a much early stage of development.