Watch CBSN Live

AOL Finds TopCoder Tops

1912 Olympics track eventYou may have heard of TopCoder, which has been providing contract programming through a competition model since 2001. What you might not realize is that some high profile companies have used their services â€" repeatedly. I had a conversation with Mike Morris, executive vice-president of software development at TopCoder, and with Nic Perez, technical director at AOL, who has used the vendor for a number of major projects over the last three years.

BNET: Mike, can you explain the service and business model? Mike Morris: TopCoder really is a social network of software developers [and] provide development services to companies using this social network. We feel that a traditional model of staffing full-time employees or hourly contracts was an inefficient way of getting work done in the software field. What we did on top of this social network, which is now 165,000 people in over 200 countries, is build a methodology of getting software developed based on competition. If Nic [Perez] needed an application, it would be a series of competitions, everything from the front end design to the architecture on which we'd build it down to the software actually running it and even the testing of that system. We have about seven different competition types and they're all pieces of what you'd consider a traditional software lifecycle -- in a format that is close to a software factory, where companies like AOL can plug into TopCoder as a way to get software talent.

BNET: Nic, what has your experience been with TopCoder? Nic Perez: We started off with an initial let's-try-out project. TopCoder created a specification and architecture, created extensive documentation. They built the blocks we needed, ran an assembly competition, and came on site. We actually doubled the size of the project in the middle, combined a phase one and phase 2 together. It only increased the project by six weeks instead of six months. It was immediately able to scale, and it went up only by the new components added, which was about an 80 percent cost.

They also found a ton of bugs in a mature API we had. We were happy and surprised. We then started to do a couple of innovation projects; AOL could come up with a killer idea and TopCoder would build it, or we could run a competition for $100,000. [For example], AOL's Truveo video search index was a very formal API that was out there. We wanted [competing developers] to consume the Truveo API and play the video locally or play it back at Truveo. We challenged the TopCoder community to come up ideas and take the API and build an application and market it to themselves.

I have an internal dev team and for a front end and back end of an existing system, moving it to another platform, they quoted three man years. We asked TopCoder and they said 30 weeks. TopCoder was given that project. We team them up with an existing team and they drive the product team. They are very demanding on a time frame perspective. Once they've speced something, they push the internal team.

MM: Nic may come to TopCoder and say, "I need you to build this application." We'd do everything from the front end design. Then we would spec that out into a software system that is made up of multiple different pieces. Applications are built out of components. AOL as a subscriber has an unlimited access to our existing access to components. We have over 1,200 components. [If they need a custom component], AOL gets charged and they own it. The goal behind that is the minimum amount of software as possible in the integration piece. Over 40% of the lines of code of applications we built comes from our existing components.

BENT: How many people out of the social network actually compete and make money? MM: About 30,000 of that 165,000 is active, meaning they've competed within 90 days. They don't all get paid. A lot of that 30,000 are people who do TopCoder just for fun. They aren't attempting to make money. We have a lot of tings that are like a chess match, where you compete in solving an algorithm. There's about 5,000 of that 30,000 that is participating in contests that would result in work from clients. Out of those 5,000, a lot of them are making money. Some are making a few hundred dollars, others are making hundreds of thousands of dollars. There are probably 500 that are making more than $10,000. Our top earner will make over a half million dollars this year. To date he's made over $700,000. That is the top echelon. A lot of TopCoders focus on building of software, software components, what I would call the plumbing behind software. One community is singularly focused on the creative elements â€" anything from a logo to an HTML prototype to a widget. It could be a flash application, story boards to design web sites.

BNET: How do the competitions work? MM: The developers are paid when they win the competitions. They get paid a set prize amount for that design. We may increase or decrease the prize depending on who is competing. The designers will submit. They are all reviewed by three of our top designers in the community and will rate and rank the design based on about 30 different scores. The review boards are paid a set amount, based on the competition and how many submissions there were. For generic components, members will also receive royalties from a royalty pool of ten percent of the revenue we get from the component sales. You can get a share [in the royalties] either by being the winning designer, the winning developer, the design review board, and the test review board.

BNET: So how many people actually get paid in a given competition? MM: Only the top two get paid. It's fairly predictable. The [winners] are pretty much always among the top designers. If you're in the top 20 percent of the designers in our community, you're going to win a predictable amount of the times. The top designer wins 71 percent of the time [and has done] over 160 competitions.

NP: If you look at their stats, a typical developer does about 2 or 3 components a week and they look at the ones they're more likely to win. If we don't have the right mix of the higher rated guys, we'll typically increase the amount of prize money to attract the right guys.

BNET: Ultimately you're depending on many people trying but few really making a living at this. How sustainable is this as most people clearly aren't winning a significant amount? MM: We've been growing fairly rapidly as a company. Our community is growing even more rapidly. If you look at the leader boards, since I've been at TopCoder for 8 years, the same people aren't at the top of the list. There are new people constantly bettering the crowd. Our general feelings is that after you have done six competitions, we have a general feelings of statistically where you fall in.

BNET: How do you judge the competitions? MM: There are three times the amount of test code written for every amount of source code. Each reviewer gets assigned a role. They build tests based on the specs: how fast code performed on a stress test, say, or how accurate it is in 100 tests. There are definitely portions that are subjective. We do our best to make those anonymous. You don't know who you're reviewing and the submitters don't know who is reviewing them.

We did have an issue a couple of years ago of people getting started and competing when a lot weren't getting enough money to make it worth their while. We looked at what other industries had that problem and we came up with Nascar as a model. It's tough if you come in second or third place every week. About 50 percent of our prize payments are getting paid out in the digital run, or the equivalent of the Winston Cup. You get points based on the type of competition, where you come in, etc. If you're consistently competing and coming in third, you will get payment. If I look at last quarter, we paid out somewhere around $450,000. This was Q2, 2008. The top competitor won $65,000. It went down to number 50, and they're around $450. This is for software design and component design. Overall there are probably 500 who get paid in a digital run.

Nic: When I look at the forums and they're chatting about not winning or how is this compared to be a contractor, a lot of positive comments are they get to choose the type of work. They get to pick and choose weekly. I'm getting that a better caliber of talent through my more conventional consulting companies, because their people are churning and they bring me the new grad. [TopCoder] is giving me the quality of coder we needed. [For many] it didn't seem to worry them at all that they weren't getting paid for it. It was extending their knowledge. Notoriety is not to be belittled here.

1912 Olympics image via Library of Congress Flickr feed, public domain.

View CBS News In
CBS News App Open
Chrome Safari Continue