I've been struggling recently with how to price Reviewable. I tried reading up on the subject of pricing SaaS offerings and talked to a bunch of people but discovered that there's no consensus, and no universally applicable advice. I'm hesitant to add my voice to the cacophony but I figure it'll help me, and—who knows—perhaps somebody else as well.

My goal is to come up with a pricing scheme for reviewing PRs from private GitHub repos on Reviewable. I ended up with the following list of desirable properties in rough descending order of importance. A pricing scheme should be:

1. Proportional to user value: the price users pay should vary depending on how much value they're getting out of the tool, to be fair to both the users and to me. A corollary is that if only a subset of an organization uses the tool, they should pay a lower price than the organization as a whole would. (This should also help with incremental adoption in larger organizations.)
2. Incentivizing use: there should be no perverse incentives to use the tool less to save money. This is mainly an issue for metered schemes, and may be directly opposed to the first property. The trick appears to be to measure some resource that is related to value but that cannot be effectively controlled by users.
3. Simple to understand: users should be able to quickly grasp what causes prices to go up or down. This means that prices should vary across no more than one, or at most two dimensions, and the resource being measured should be something familiar.
4. Predictable over time: users should be able to predict the monthly cost of subscribing to the service. No nasty “overage” surprises.
5. Proportional to costs: the price should vary based on the resources needed to provide the service. This doesn't matter to the users, but it's hard to keep a service running if you can't cover the costs.

So, given these desiderata, what are some pricing schemes we can come up with and how do they stack up?

Fixed price

We could charge a fixed price per organization (and perhaps a separate one for personal repos); Huboard and Bronson take this approach, for example. This is clearly simple to understand, eminently predictable, and incentivizes use, but isn't proportional to either value or costs. A low price will leave a lot of money on the table (and may have trouble covering costs), whereas a high price will exclude a lot of potential customers. Given how simple this scheme is, though, it may be a good choice for getting off the ground quickly and gathering data to inform a more complex approach later.

Per repo price

We could follow in GitHub's footsteps and charge per repo connected to Reviewable. Again, this is extremely simple as the principle is necessarily familiar to GitHub subscribers, predictable, and incentivizes use since once you've connected a repo you might as well take advantage of Reviewable for all its PRs. It's only somewhat proportional to cost (which is mostly driven by the number of commits and simultaneous users), but that's a minor quibble.

Whether this scheme is proportional to value depends on the details. If we only allow an entire organization to subscribe for a price similar to what they already pay for GitHub hosting (like GerritHub does), then yes, in practice the total number of private repos in an organization is somewhat correlated with the number of members and overall level of development activity. However, this is not helpful if only a subset of an organization (e.g., a team) wants to subscribe. And if we allow a team to sign up for a fixed number of repos (e.g., just one) at a set price, then the relationship breaks down: while larger organizations have more repos, typically most of the activity takes place in just one or two of them, so the scheme reduces down to “fixed price”.

Per review price

Since we're not having much luck so far making the price proportional to value, let's try turning to the resource at the center of user activity: the reviews themselves. If we charge based on the maximum number of reviews in a month we'll be more in line with value and costs, and the metric should be familiar to the users. Predictability suffers since the number of reviews can vary widely month-to-month—especially in smaller teams—but it ought to be manageable.

But things really go downhill when we look at precisely how we count reviews. If we count reviews opened using Reviewable then we introduce a perverse incentive into the system: clicking on a review link may cause your subscription cost to go up! Now users will be second-guessing whether a PR can be adequately reviewed in GitHub instead of getting in the habit of using Reviewable; this is definitely not what we want. We can address this problem by counting all PRs instead, whether they're opened in Reviewable or not. The perverse incentive goes away since creating PRs is unavoidable, but now the connection to value becomes much looser, especially if not all contributors to a repo have adopted Reviewable.

With both variants there's also the issue that not all reviews have the same value: more complex reviews have greater value, but there doesn't seem to be a way to measure this complexity (e.g., by number of comments) without introducing further perverse incentives. And teams that tend to produce many smaller PRs will pay disproportionately more for the same value received.

Per user price

Finally, we could do what the vast majority of SaaS applications do and charge per user. Typically, “user” means a person who is allowed to access the app. This can be a dynamic per-person price (like Gitter) or bucketed into tiers (like Review Board Commons or GitColony). It's simple and predictable (especially the bucketed variant), reasonably proportional to both cost and value since the amount of activity per user averages out quickly, and incentivizes use to “get your money's worth”.

So what's not to like? First, this scheme usually requires the subscriber to perform some kind of additional user management to maintain an ACL for the app. This isn't critical but it's an onboarding speedbump and an ongoing headache as people join and leave a team or organization. Second, it blocks people outside the subscriber list from accessing reviews until they're added by the admin (if at all). This is a concern since reviews occasionally need input from outside the circle of usual suspects and Reviewable reviews don't mesh well with GitHub PR comments. Additionally, it's off-putting that the links to Reviewable from the PR won't work for everyone, inhibiting the viral spread of the app within an organization.

The least worst option

None of the alternatives so far scream “choose me”, but perhaps we've learned enough to put together something better. There are two key insights to apply: the subscription check should take place at the time a review is created (and the review should be accessible to everyone afterwards), and per-user pricing gets closest to striking a good balance between value proportionality and good incentives. The obvious inference is to charge for users who will be allowed to create reviews, but this doesn't play well with Reviewable's repo connection mechanics and is subject to gaming.

Instead, let's consider pricing based on PR authors. It turns out that in well-adjusted teams the number of distinct PR authors correlates fairly closely with the number of app users, and hence with value and cost. Any perverse incentive to manipulate the metric by reducing the number of PR authors is overridden by the imperative of submitting PRs to get work done, and there's no limitation on the number of reviews. With tiered, floating seats pricing (e.g., “$39 for any 10 PR authors per month”) the monthly spend is predictable and there's no need to maintain a separate user list for the app. We can further enhance predictability by letting subscribers limit who is able to create reviews (either explicitly or by connecting repos) and being flexible about temporary overages. The main downfall of this scheme is that it's atypical and may prove tricky to explain and understand, especially in situations where a subscribing team shares a repo with other developers.

I originally intended to use a per-repo pricing scheme for Reviewable, but at this point I'm strongly leaning towards the per-PR-author one as the least worst option. I'd love to know if you disagree with my analysis, though, or have an even better idea!