What Are Crowds Good For?

The book “The Wisdom Of Crowds”, by James Surowiecki, evoked a surprising amount of interest upon its publication in 2004.  The book sets forth a thesis that seems to have struck a chord with many people.  Actually, there are two forms of the thesis, which it is helpful to distinguish:

Wisdom Of Crowds Thesis, Weak Form
: By properly selecting a large number of people with a good mix of skills and knowledge about some subject, and by adopting a carefully-designed procedure for distilling their views into some kind of weighted-average consensus view, it is possible to arrange matters so that this consensus view is almost always more accurate and reliable than any individual view held by any member of the group.

Wisdom Of Crowds Thesis, Strong Form: Given a sufficiently large group of people, if their average or consensus view on any subject is somehow ascertained, that view is always more reliable than the views of any individual.  Expertise on the subject in question is nice, but not required, since knowledge will emerge of its own accord once some “Large Number of Participants” threshold has been reached.

As a brief assessment of these two versions of the “Wisdom” Thesis, the Weak Thesis is demonstrably true, while the Strong Thesis is, not to put too fine a point on it, bunk.  On this, more below.

From reviews and excerpts of the book, I get the impression that the bulk of the argument in the book is dedicated to defending the Weak form of the thesis, although some rhetoric is also marshaled to support the Strong form, and the book’s promotional material is dominated by the Strong Thesis.

Whatever the balance of argument in the book, the Strong form of the thesis appears to have spoken to some deep belief held by many people, who appear to have taken it as a kind of validation of progressive principles of radically egalitarian democracy.  Here was proof that elites are superfluous where they are not harmful, that the People, if allowed to speak in its own voice, has wisdom transcending that of experts and technocrats.

You see, while the Weak thesis is a scientifically well-supported technical argument of applied epistemology, the Strong thesis is essentially a political statement, constructed by deforming the Weak thesis to suit ideological requirements.  As is invariably the case when technical-scientific results are appropriated for political-ideological purposes, the Strong Wisdom of Crowds thesis has been attended by an enormous outpouring of bullshit (as a physicist, I tend to notice this sort of thing).

To see where the Strong Thesis goes awry, it is helpful to consider examples of how the Weak Thesis succeeds.

(1) The Stock Market. The famous — and much-belabored — example is the stock market, in which tens of thousands of investors herding trillions of dollars around essentially manage to successfully forecast the performance of the economy a few months into the future, with some moderate degree of reliability (at least when a bubble isn’t in progress). Note the leadership role played here by analysts and portfolio managers, who steep themselves in highly relevant economic data, as well as the obvious incentive system that on average rewards thoughtful investment and penalizes amateurish gambling.

(2) USENET FAQs. A less widely discussed example comes from the early history of the Internet.  Usenet News Groups, now a neglected backwater of the net, were until the mid-90s the principal forum for discussion on the net.  Hundreds of thousands of people would congregate to post articles — and to respond to other posts — in groups organized hierarchically by topic with names like “sci.physics”, “comp.os.linux.networking”, “soc.culture.french” etc.  The resulting discussions were often cacophonous:  Together with polite and informative arguments, there were endless flame wars (the term in fact dates from the Usenet era), pointlessly repetitive arguments, bad information trafficked with authority, etc.

Miraculously, from this extremely low-signal-to-noise environment, there emerged a priceless treasure-trove of information:  The Frequently Asked Question Sheets.  People in the various newsgroups got tired of answering the same questions over and over again every time someone new joined the group.  In many groups, a few people took the initiative to start culling what were in their judgment the best answers given to these recurring questions. In effect, they appointed themselves editors.  In almost every case, the appointment received the approval of the majority of the group membership. They operated as information gatekeepers.  The best ones gave encouragement and advice to their contributors, and edited the contributions for clarity and accuracy.  They had final say on what went into the FAQs, and while there may have been some cases of revolts or schisms over FAQ maintenance, I never heard of such an occurrence.

The result is astonishing.  You may peruse some of those USENET FAQ sheets at http://www.faqs.org/faqs/.  There is a depth and breadth of distilled, high-quality information that is all the more shocking given the near-white-noise of the discussions from which that information frequently emerged.  I find that it is often a valuable resource, better and much more reliable (if now somewhat dated due to the decline of USENET) than Wikipedia.

(3) The Linux Kernel. One of the most visible and strikingly successful examples of a crowd coming together to produce a high-quality knowledge-based product is furnished by the Linux kernel.   This project consists of hundreds of (mostly volunteer) software developers, organized in a pyramid-shaped hierarchy of subsystem maintainers supervising coders and bug fixers, reviewing the resulting code patches, and when necessary passing them up the hierarchy to be signed off on by more senior maintainers.

I need say little about the quality, efficiency, broad portability, reliability, and performance of the resulting code, since Linux is now too well-known to belabor. I do wish to bring out a point that helps to explain how the success of this crowd enterprise: the kernel maintainers benefit from an objective “Truth” measure that is not available in many other knowledge-producing enterprises — it’s not a matter of opinion whether or not code works as it’s supposed to.  Either it does, or it doesn’t.  The kernel maintainers’ job is to make sure that it does.

A couple of necessary elements for the success of the Weak Wisdom of Crowds Thesis are apparent from the above examples.  In the first place, the knowledge that is to be distilled must be present to begin with — it does not mystically appear, fully-formed, in the output of an impersonal process that performs an equal-weight average of many uninformed opinions.  Garbage in inevitably results in garbage out — the GIGO law is not suspended or suppressed merely by virtue of the large number of participants in the process.

The second necessary element is that there should exist some kind of discipline-enforcing mechanism, to filter out the garbage from the knowledge.  In the stock market, the monetary incentive system performs this office.  On Usenet, the FAQ sheet maintainers acting in their editorial role supply the required regulation.  The kernel maintainers do the same for the Linux kernel.

In fact, we can now see that the success of the Weak thesis is hardly mysterious at all:  the thesis describes the operation of a filtering process, in which informative data concealed in a welter of uninformative noise is extracted by straining the data through a suitably-constructed, highly specialized filter.  The crowd membership bears the data, while the filter is either externally-supplied (maintainers, editors) or embedded in the organization and workflow of the crowd itself (the market).

The existence of informative data, and availability of the filter are crucial to the success of the Weak thesis.  And, tellingly, the caveats that distinguish the Weak thesis from the Strong thesis are precisely the ones responsible for guaranteeing that these preconditions for success should hold.  We can begin to see now why the Strong thesis is bunk — it fails when either informative data is not available, or when the filter is ineffective.  Two examples, embodying each of these failure modes, are the Policy Analysis Market and the Wikipedia project.

PAM: In 2003, news stories appeared reporting on Admiral Poindexter’s most recent project — the Policy Analysis Market (PAM).  PAM was supposed to be a sort of futures market in which investors would be invited to buy or sell derivatives whose value was based on whether (for example) a terrorist attack in (for example) Jordan, in (for example) three months time is estimated to be likely or unlikely.  The idea was that knowledge that might not be available to professional intelligence analysts might instead be extracted from a confusing background of data by the simple device of ascertaining the opinion of thousands of investors, as manifested in the daily closing price of the derivatives that they trade.

When the idea became public, it was rightfully derided as stupid (it was also denounced as immoral, but that is not relevant here).  “Stupid” is a blunt judgment, but what else can one say about the supposition that thousands of speculators who get their information from the Times, the WSJ, “Foreign Affairs”, and perhaps some more specialized news sources and newsletters, could somehow outguess a small corps of professional intelligence analysts with access to NSA ECHELON transcripts, Saudi Embassy gossip, agent reports, and back-channel information passed on from other governments?  PAM investors betting on terrorism have no access to the information required for the market process to synthesize a valid prediction (unless, of course they are terrorists).  The process would certainly produce random noise, uncorrelated with any actual terrorist events in the real world.

You see, however, how the fallacy of the Strong thesis has begun to gain intellectual traction.  The Federal Government was preparing to spend
tens of millions of dollars on this colossally dumb idea, spurred on by what is essentially a religious-mystical faith in the ability of a crowd to liberate “wisdom” by the mere fact of its large size.

WIKIPEDIA: The Wikipedia model of knowledge aggregation is unapologetically founded upon the Strong Wisdom of Crowds thesis.  Wikipedia explicitly rejects all forms of moderation.  The founding creed of the project is openly disdainful of the authority of experts.

The reason the project “succeeds” is that the metric for “success” that is used in the Wikipedia system of discussion and consensus-building has no pushback from reality.  There is no “Truth” that forces itself back on the discussion (the way code crashes force themselves on kernel developers, or massive losses on NYSE investors).  It’s all just belief.  And since Wikipedia is radically egalitarian by charter, the beliefs of the experts are weighted identically to those of the amateurs and blowhards.  There are many documented cases of the blowhards overwhelming the experts.  Revision history even shows frequent examples of initially excellent articles produced carefully by someone who knew what he was talking about, which were subsequently stupided down by the crowd of jackasses who descended on them, evidently feeling (unjustifiably) that their opinions were just as worth recording as those of the expert.

Here is the fatal flaw of Wikipedia, which the palliative measures adopted in the wake of recent embarrassments are powerless to correct.  It’s not a matter of locking down sensitive, high-visibility articles with a demonstrated tendency to fly off the rails.  Crap is everywhere on Wikipedia, mixed with gold, and not distinguishable from it.  The more opinions about the subject of an article, the more likely it is to contain bad information, and the less likely it is to be corrected.

The foundational principle of the project is wrong: Twenty jackasses do not make an expert.  The key to extracting signal from this kind of crowd noise is peer review, moderation, and editorship, all of which the Wikipedia project rejects as a matter of ideology.  Wikipedia consults crowds the way spiritualists consult ouija boards, not through a knowledge-refining filter but through a simple and essentially superstitious faith in oracles.

I think this is an interesting failure.  It is not acknowledged or understood by the project’s devotees, in my opinion because the Strong Wisdom of Crowds thesis has an unbreakable grip on their collective world view.  The reason this is interesting is that as the PAM caper demonstrates, they are not alone.  There is something seductive about the Strong thesis, with its irresistible promise of supernatural wisdom generated by prosaic processes not requiring any actual cogitation, and its reassuring apparent confirmation of cherished belief in egalitarian democracy.  Its attraction is spreading, and in years to come it will probably be found in other places that are currently less visible than Wikipedia and PAM.


Tags: ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: