THE DOTA EXPERIMENT USER RESEARCH
My Role: User Researcher
Project Duration: Six Months (Over two summers)
Methods: Surveys/User Interviews, Quantitative Analysis, Research Report
Output: Two Research Reports
Synopsis: After graduating from the University of Arizona in 2012, I wanted to run a research report about my favorite game, Dota 2. To do this, I created a 90+ point questionnaire and released it through Reddit and other popular Dota 2 channels. I released the questionnaire over two summers and compared the results as the community grew from 100,000 to over 1 million users. My findings suggested not only an incredible passion for Dota 2 from the community, but also a heavy-handed elitism that was preventing new players from enjoying the Dota 2 experience. I delivered my results to Valve* at the Dota 2 International 2 & 3 in the format of two printed research journals, and provided the community with the results as well in the format of interactive PDF's.
*This project was not commissioned by Valve Software
This community is growing fast!
Dota 2 is a hard game for newcomers. The variety of written and unwritten cultural rules they must understand to compete can be intimidating. My goal for this project was to understand the behaviors and attitudes of new and old players when it comes to Dota 2, and to identify pathways to improving Dota 2's user experience- especially for new players.
I designed an online questionnaire code-named The Dota Experiment Questionnaire (TDEQ). This questionnaire included sections related to demographics, preferences in online communities, in-game attitudes and behaviors, preferences for Dota 2 characters, and questions based around real in-game scenarios.
This questionnaire was released over two summers, with the first questionnaire getting over 1,600 unique responses from around the world, and the second getting over 3,200.
Quantitative Data Analysis
Using a premium Sogo Survey's account, I was able to cross-compare data sets to identify patterns and trends in user behavior. I was also able to present these data in the forms of charts and graphs to help better communicate my results.
While analyzing data from the second questionnaire, I was able to compare the results with the first study to look for changes in attitudes or behaviors over the past year where Valve implemented new tools for changing community behavior, such as a punishment system.
All of these data combined helped me form a large number of hypotheses that were communicated to Valve in the final report.
Qualitative Data Analysis
I extracted qualitative data through the questionnaire in the form of open-ended questions. From this I was able to identify some attitudes and behaviors that were less pronounced through the quantitative data itself. Many of these resulted in cultural issues, thoughts and feelings from new players, as well as a deep passion for Dota 2 itself.
Perhaps my favorite part of this project were my 25+ user interviews over both books. In each chapter of my reports, I included a unique interview with a Dota 2 community member, whether they were an artist, community leader, or newcomer. These interviews provided the final report with a little extra personality, and a deeper dive into the attitudes of Dota 2 community members.
Two research reports
I hand-delivered The Dota Experiment (2012), and The Dota Experiment 2 (2013) to Valve in the form of spiral-bound research reports. I released interactive PDF versions of the reports through Reddit and the other channels where I released the questionnaire, where they have since received over 20,000 combined views.
Links to the full reports:
Lessons and Feedback
After releasing the results, I received a ton of positive and negative feedback. While people loved contributing to the project, I was made aware that by not making the questionnaire and results available in multiple languages, I was cutting a large chunk of the international community. Though I believe this is a lot to ask for from a one-man team, I believe that if I were to do this project over again I would include translation services.
A second piece of feedback I got was related to the questionnaire itself. It was really long, with over 90 questions. Even though I got over 5,000 combined responses between both questionnaires, I believe that a lot of participants experienced burnout which may have skewed the results. I did include questions in my questionnaire to help weed out people who may have spammed answers, and removed them from my results.
All things considered, I loved this project, and am very proud of it. It was my first time dabbling in user research before I even knew what user experience design even was.