Developer Integration Experience - Skillz

MAKE THE TECH EASY - REVAMPING THE SDK INTEGRATION EXPERIENCE

About Skillz

Skillz is the leading mobile esports platform, hosting casual esports tournaments for 30 million mobile players worldwide and distributing over $60 million in prizes each month. While our consumer arm is the largest and most visible component of the business, Skillz does not create the game content. Instead we rely on 3rd party developers to integrate the free Skillz SDK into their games and gain access to our suite of out-of-the-box features and unique monetization models. As a PM on the Developer Experience team, my job (alongside my team) was to build products that enticed and supported developers to launch games with our technology and ultimately create successful businesses of their own.

What We Did

We revamped the end to end experience of integrating our SDK. Replacing a workflow described as “tedious and confusing” with a modular and semi-open experience. The final product we shipped enabled our users to take the product at their own pace, removed roadblocks to their integration, and ultimately resulted in both higher satisfaction score and throughput. Read on for details

The Problem & Goal

The integration experience of adding the Skillz SDK to your game was outdated and barely functional. User research identified that developers found the process too rigid and forced them through a specific workflow that often didn’t match their preexisting game design processes. The goal of this project was to alleviate these issues and remove the product friction that was causing users to abort their integration.

Discovery & Ideation

Given that the old experience was failing to deliver results, we kicked off this project with a collaborative whiteboard exercise across engineering, design and product. All ideas were welcome at the table as we split into small groups and each tackled one core component of the problem space (customer support, tech integration, game management, documentation, analytics & revenue). Within these small groups I helped facilitate each team to mockup their own vision for what a great experience would look like, and then we presented these sitemaps/concept maps to rest of the group to discuss what pieces worked well and what missed the mark.

This exercise was successful in many ways

1. It helped us identify the Big Ideas and Key Design Principles that served as our North Star for the duration of this year long product revamp

2. In just a few hours we had 100 new feature ideas, some were great and some were pretty questionable, but all worth discussing

3. It brought the entire team together under a single product vision, noticeably bolstering the “sense of ownership” across the young team that is incredibly important before undertaking large structural changes to a product

Big Ideas and Key Design Principles

Following our white boarding session and later user research sessions, I worked with the design and research team to identify that the initial integration experience was the root cause of most of our pain points. Focusing in on this problem space, we then aligned on the core concept that the integration process would be separated into two distinct experiences.

1. Integration Guide - a linear experience that quickly shepherds the user through the process of setting up their game within the Skillz Sandbox Environment

2. Game Management - post sandbox setup, game management should be modular and non-linear. Developers can choose what and when to take any actions, such as creating custom themes or publishing their games to the store.

Following this revelation, the rest of the design process seemed to fall in place naturally, and the next couple weeks were spent hard at work joyously iterating on the design directions that this new ethos spawned. This yielded the following key design principles

• Minimize the work required to setup the Skillz sandbox environment so that developers can experience the product first hand from within their app

• Flood the user with positive feedback and encouragement

• Reduce the noise

  • Focus on explaining the core concepts of Skillz and getting users to adopt the correct conceptual model of how our tech works

  • Technical details will live primarily within documentation, which is accessible throughout the integration guide via smart links

• Default to allowing user choice while simultaneously providing a recommendation on what do next

Prototype & Testing

But before we built out this complicated system, it was time to make sure this was the right solution to address the pain points and goals outlines above. I worked with the Design team to develop and run a remote usability test using a prototype on usertesting.com. What we found was that while users generally understood the overall workflow, the signifiers we initially had put in place were not effectively communicating user progress. However, a key win from this study was that despite progress being unclear, user’s were not getting lost and found completing the prototype easy to do (100% completion rate). Why??
‍
We discovered that users repeatedly referred back to a Next Steps system, which was a dynamic recommendation widget that we placed on the core Game Overview page. Even when inside a different page, users remembered this widget and would organically refer back to the Overview page any time they felt stuck or uncertain what do next. This kind of discovery is why user testing is invaluable prior to development, as we then proceeded to cut the more complicated progress bar idea and instead promote and flesh out the Next Steps system. The result (as you’ll see in the final section) was a much smoother user experience that improved our integration completion rate.

Getting into Development

A project this large cannot be developed overnight. In fact, my engineering manager estimated approximately 6 months of effort to complete the project as spec-ed. However, the business can never afford to wait 6 months for the first hope of results. So we opted to break up the project into milestones and iteratively release product improvements in phases.

1. Logically, we attacked the top of funnel first (since we knew the biggest dropoff occurred at the beginning of our old integration flow) and build a componentized version of the new integration process and then retrofitted it into the old tech. Immediately we saw a lift in user’s completing the early stages of our integration and becoming “Sandbox Ready”

   • “Sandbox Ready” is a term we coined to represent that a game was running in our development environment and developers could test out SDK functionality. This new integration guide tripled the number of games reaching this state

2. We then launched the new core Game Management experience and formalized the Game Object as the Core Value Unit of the project

3. After the Game Management MVP was launched, we expanded the functionality that branched off of this point piece by piece. With each component of the new product experience rolling out to users as they became ready

After 5 months of eng effort, the product officially launched as V2 in Feb of 2020.

End Results

By the end of Q2, we had increased the number of self-serve games (new Skillz-powered mobile games that launched on the app store that were NOT the outcome of a Skillz<>Developer publishing contract) launching on Skillz by 600% from ~7 per quarter to over 40. The numbers spoke for themselves, but I personally appreciate the user feedback that came straight from our top developers.

"I now feel like I can do everything that I want on here"