This was originally meant for the iFixit.com blog. iFixit never published it (because I never finished it), so I decided to clean it up a year later and publish it here. It should be noted that I am no longer employed by iFixit, and my views do not represent those of iFixit, nor my current employer, Google.
In November 2009, iFixit released Answers. When iFixit started on this project, we thought long and hard about how integral of a part iFixit wanted Answers to be in relation to the rest of the site. iFixit already had a forum system that had been coded based on the comment system throughout teardowns and guides, but it was growing very long in the tooth. So, after some debate, we decided to build a new tool that rewarded users for participating in our site and promoted quality content over quantity.
To reward users for quality content, we wanted to allow the community, not just the iFixit staff, to be able to judge what was useful, what was irrelevant and what was low quality. We decided on a basic reputation system, which allowed users to vote positively and negatively towards others posts, and for authors of questions to accept the answer they felt was best. That way there could be two "best" answers to every question, one the community liked and one the original poster liked. Often these line up, but this is not always the case, so we wanted to make sure everyone gained reputation when appropriate.
The reputation system was well received, but we started running into issues. Some of our users were gaining large amounts of reputation, while others were gaining very little. We had put limits in place on how fast and how often a user could do things, and these limits were being hit all far too often by some of our very active users. So we raised the limits to match our user's activity and scaled votes, so that the answers with tons of votes gave slightly less reputation.
When I was at CodeConf last year, in one of the presentations, Coda Hale talked about metrics and how you need to measure what's important to you and your code base. I found this funny, because we basically did exactly what the presentation said not to: we made assumptions about our users. We had no idea the users like Mayer would ever exist. We had trouble believing that a single individual would want to participate in our community so much he would answer a majority of the questions posted to the site, which meant we assumed he wasn't writing quality content. This assumption lead us to prevent him (and others) from creating content on our site. Going forward building iFixit features, we found that we could query the database about the use habits of our more active users, and use those numbers to build limits to help us prevent spammers but also let those who want to create real content do just that.
As we sorted out the issues limiting our users, we also came across the fact that people who wrote guides also wanted and deserved reputation for content that was referenced throughout the site. So we started keeping track of a few different ways guides were being used to help others in wiki pages and in answers to questions. This was a bit of a pain, because we had to make sure our users (or some of our employees...) could not game the system easily. This required harder limits on how much reputation a user could give other users and how much reputation one could receive per day. We started noticing that some users were getting strange ratios, where they were getting 95 percent of their reputation from one user, or one user was giving all of their reputation to another user. We found after some heavy investigating that in many cases the first scenario was suspect, but not always foul play. The second scenario though, was always a user trying to game the system, so we started moderating users that were acting suspiciously, which seemed to promote fair play on Answers and throughout the rest of the site.
The system itself had some design issues in retrospect. Originally I created a single class that managed all of the reputation. This, in theory made it simple for developers to add reputation to any action on the site. The problem was this didn't make much sense with the rest of the site. I tied reputation points to closely to objects, which was ok, but I did it in a weird non-object-oriented way. Probably one of the better ways to build the system would be to just pass in the object you wanted to reward a user for creating, instead of having the developer pass in an id and a table name. Also, instead of having a static method in a library, creating a reputation object would have been much easier for developers.
Anyways, these are just some insights into the first large project I ever launched. I should mention that Answers wouldn't exist without the fantastic efforts of Sterling Hirsh (he developed most of the frontend of Answers). Him and I constantly butted heads on how Answers should be built and I learned a lot from him. Also, a big shout out to all of the iFixit developers (Dave, Shawn, Kyle, Chris and all the rest), you guys are the best.
/Nat