Open and Close Innovation Regimes: An Experiment on Institutional and Organizational Design

Kevin J. Boudreau & Karim R. Lakhani, How Disclosure Policies Impact Search in Open Innovation, Harv. Bus. Sch. Tech. & Operations Mgmt. Working Paper (2013).

One of the central tensions in the institutional design of innovation regimes is the trade-off between incentives and disclosure. Innovation systems, including intellectual property systems, are created to optimize creative output by balancing ex ante incentives for initial creators with ex post disclosure of the innovation to follow-on creators and the public. According to accepted theory, the more rigorous the disclosure—in terms of when and how it occurs—the weaker the incentives. But a fascinating new experiment by Kevin Boudreau and Karim Lakhani suggests that differences in disclosure regimes can affect not just the amount of innovation but also the kind of innovation that takes place.

Boudreau and Lakhani set up a tournament on the TopCoder programming platform that involved solving a complicated algorithmic task over the course of two weeks. All members of the community were invited to participate in the tournament, and contest winners would receive cash prizes (up to $500) and reputational enhancement within the TopCoder community. The coding problem was provided by Harvard Medical School, and solutions were scored according to accuracy and speed. Importantly, the top solutions in the tournament significantly outperformed those produced within the medical school, but that’s a different paper.

Boudreau and Lakhani randomly assigned participants into different conditions based on varying disclosure regimes and tracked their behavior. The three disclosure conditions were:

  • Intermediate Disclosure – Subjects could submit solutions to the contest, and, when they did, the solutions and their scores were immediately available for other subjects in the same condition to view and copy.
  • No Disclosure – Subjects’ solutions to the contest were not disclosed to other subjects until the end of the two-week contest.
  • Mixed – During the first week of the contest, submissions were concealed from other subjects, but, during the last week of the contest, they were open and free to copy.

For the Intermediate and Mixed conditions, subjects were asked to provide attribution to other subjects’ whose code they copied.

Cash prizes were given out at the end of the first and second weeks based on the top-scoring solutions. For the Intermediate condition, the prizes were split evenly between the subject who had the highest scoring solution and the subject who received the highest degree of attribution.

The subjects were about equally split between professional and student programmers, and they represented a broad range of skill levels. 733 subjects began the task. Of them, 124 submitted a total of 654 intermediate and final solutions. The solutions were determined to represent 56 unique combinations of programming techniques.

The authors predicted that mandatory disclosure in the Intermediate condition would reduce incentives to participate because other subjects could free-ride on the solutions of initial inventors. The data are consistent with this hypothesis: Fewer people submitted answers in the Intermediate condition than in the No Disclosure condition, and the average number of submissions and the number of self-reported hours worked were also lower by significant margins. The Mixed condition generally produced data that were between the other two conditions. Ultimately, scores in the Intermediate condition were better than those in the other conditions because subjects could borrow from high-performing solutions.

More importantly, the data also disclosed differences in how subjects solved the problem. Consistent with the authors’ hypotheses, subjects in the Intermediate condition tried fewer technical approaches and seemed to experiment less than did those in the No Disclosure condition. Once significant improvements were disclosed, other subjects in the Intermediate condition tended to borrow the successful code leading to a relatively smooth improvement curve. In the No Disclosure condition, by contrast, although new submissions were generally better than those the subjects had submitted before, they were more variable and less consistent in their improvement.

In summary, when subjects can view each others’ code, innovation tends to be more path-dependent and to happen more rapidly and successfully than when there is no disclosure. But when innovation systems are closed, people tend to participate more, and they tend to try a wider variety of solution strategies.

In previous research, these authors have explained how open-access innovation systems succeed in the face of diminished extrinsic incentives. This experiment provides valuable insight into the relative merits of open- and closed-access systems. Open-access systems will, all else equal, have advantages when creators have significant intrinsic incentives and when the innovation problem has one or few optimal solutions.

Closed-access systems, by contrast, will prove comparatively beneficial when the system must provide independent innovation incentives and when the problem involves a wide variety of successful solutions. The experiment’s contribution, then, is not to resolve the debate about open versus closed innovation but rather to help policymakers and organizations predict which kind of system will tend to be most beneficial.

The experiment also suggests helpful ways of thinking about the scope of Intellectual Property rights in terms of follow-on innovation. For example, strong derivative-works rights in copyright law create a relatively closed innovation system compared to patent law’s regime of blocking patents. If we think of the areas of copyright creativity as exhibiting a large variety of optimal solutions, then the closed-innovation system may help prevent path-dependence and encourage innovation (evidence from the movie industry not withstanding). Future research could test this hypothesis.

As with any experiment, many questions remain. Boudreau and Lakhani’s incentives manipulation is not as clean as could be hoped, both because payouts in the Intermediate condition are lower and because attribution in the No Disclosure condition is effectively unavailable. Accordingly, it is difficult to make causal arguments about the relationship between the disclosure regime and incentives. In addition, although the Intermediate condition produces lower participation incentives for subjects who expect to be high performing, it creates higher participation incentives for subjects who expect to be low performing because they can simply borrow from high-scoring submissions at the end of the game.

Interestingly, there seems to be surprisingly little borrowing, which could suggest a number of curious features about the experiment: Perhaps only high-skill subjects are capable of borrowing and/or there may be social norms against certain kinds of borrowing even though it is technically allowed. And, as always, there are questions about the representativeness of the sample. Subjects were likely disproportionately men, and they were also likely ones with significant open-source experience where they may have internalized the norms of that community. On the other hand, TopCoder bills itself as “A Place to Compete,” which may have primed competitive behaviors rather than sharing behaviors.

Ultimately, Boudreau and Lakhani have produced an exciting new contribution to intellectual property and innovation research.

 
 
Discussion

2 comments
  1. 1

    Nice writeup and great experiment. It’s nice to see empirical support for some of the theories I explain in my trade secret work.

     
  2. 2
    On May 4, 2014 at 9:42 pm, Ted Sichelman said:

    Agreed with Michael’s comments–a concise and lucid explanation of this fascinating experiment. I wonder if the barriers to entry were higher whether the results would differ. For instance, suppose that each coder were charged $25 to enter the contest. Presumably, that would provide a greater disincentive to participate for those given the intermediate disclosure condition, as they are arguably less certain (and also less likely from their self-interested, private perspective) to reap a large reward than those given the no disclosure condition. Given that there are often large barriers to entry in innovation races, this factor seems a very important one to throw into the mix.