As of yesterday, the GovTribe private beta is closed. As this is our first of such events, I wanted to reflect a bit on the process, the tools, and the results. All in, I was pretty pleased with the way it turned out. As GovTribe grows, I expect we will continue to use this approach to early testing with actual users. In general, it gave us a nice mix of input from our target customers, as well as feedback from informed iPhone app consumers that may not be in the business of government contracting.

The Process

As we were testing an iPhone application, there were a few constraints we faced with respect to process. For one, getting an application onto an iPhone outside of the App Store can be a bit cumbersome. In any event, the basic process consisted of the following steps:

  1. Solicit private beta testers through the GovTribe website.
  2. Invite testers to join TestFlight.
  3. Implore testers to install the TestFlight profile on their device.
  4. Add tester's devices to our ad hoc distribution profile.
  5. Notify testers of an available build.
  6. Solicit feedback both in-app as well as through the GovTribe website.

While I was happy with the TestFlight platform, I would like to improve as much as possible on the app distribution process. Most of our testers had never participated in a private beta before and found the first five steps above to be starkly more complicated than their typical single-step App Store experience. Like five times more complicated. When basically asking people to volunteer their valuable time, this can drastically reduce participation or artificially color the resulting feedback. Some of this is just the nature of the iOS beast, but I feel we could improve on our communications and tester-shepherding a bit.

The Tools

Prior to our private beta, we looked around the Interwebs a bit to understand our options. Best I can tell, the market of platforms that specifically support iOS private beta testing outside of the App Store is relatively nascent. TestFlight, Hockey, and iBetaTest were three up for consideration. After a bit of hemming and hawing, TestFlight emerged as the platform for us.

In addition to TestFlight, we have a few other tools that aren't specifically designed to support our beta, but provided important capability nonetheless. In no particular order the following tools helped the process:

  1. New Relic - Backend performance monitoring.
  2. MailChimp - Email management.
  3. TestFlight SDK - Another product from TestFlight, providing insight into app usage.

The Results

The GovTribe team is a big proponent of the Lean Startup approach and as such, takes user interaction and feedback pretty seriously. We started the beta with a few hypotheses. Some of which were proved and some of which were not. Throughout the beta, we were adapting the production release of our first product to reflect these results. GovTribe believes our products and their capabilities are guilty until proven innocent. Sure, we have ideas about what people want, but if the data proves otherwise, we will quickly adapt and move forward, regardless of the perceived sacredness of our binary bovines.

All in all, we were quite pleased with the overall result of our first beta. There are places to improve, but for the most part it was a process that worked. Please feel free to share your own thoughts in the comments section about beta testing you have conducted. What did we miss? What lessons did you learn?

P.S. Thanks again to all of our beta testers!