Human Computation — Chapter-by-Chapter Outline

Author: Edith Law and Luis von Ahn First published: 2011 Edition covered: First edition (Morgan & Claypool Publishers, 2011). Part of the Synthesis Lectures on Artificial Intelligence and Machine Learning series, Lecture #13. ISBN 978-1-60845-516-4. The book is 105 pages and has not been substantially revised in a second edition; a Springer reprint (ISBN 978-3-031-00427-8) reproduces the same content.

---

Central thesis

Human computation is a research discipline organized around a single core idea: human intelligence can be harnessed as a computational resource — systematically, algorithmically, and at web scale — to solve problems that remain beyond the reach of the best automated artificial intelligence algorithms. The book argues that this is not a loose metaphor but a precise claim: if you define computation as the mapping of inputs to outputs via an explicit, finite set of instructions, then a system that enlists millions of humans to perform such mappings is literally doing computation, and the tools of computer science (algorithms, correctness criteria, complexity measures, programming frameworks) apply to it directly.

The organizing framework is three questions that any human computation system must answer. First, the "what" question: which computational problems should be solved by humans and how should those problems be algorithmically decomposed? Second, the "who" question: to which workers should each task be routed, and how should that matching be done? Third, the "how" question: how should tasks be designed and what incentive mechanisms should be put in place to ensure that workers compute accurately and truthfully? Part I of the book (Chapters 2–4) addresses the algorithmic side of these questions; Part II (Chapters 5–6) addresses the design side.

The book is also a field-defining survey. Written in 2011, when human computation was a young and loosely bounded research area, it deliberately draws connections to AI, machine learning, HCI, mechanism design, and psychology to show that many existing bodies of knowledge partially address human computation problems — and to argue for a unified framing that can drive future progress.

How do you design a system — algorithmic, social, and economic — that reliably enlists human intelligence to solve problems that machines cannot?

---

Chapter 1 — Introduction

Central question

What is human computation, how does it differ from crowdsourcing and collective intelligence, and what are the fundamental research questions the field must answer?

Main argument

Computation as a human activity first

Before any modern computer, the word "computer" referred to a person who performed calculation as a profession. The Hollerith Machine (used in the 1890 US census to tabulate data from 75 million individuals) was named to distinguish itself from these human computers, ending its acronym in "AC" for "Automatic Computer." Organized computation projects — from the calculation of Halley's Comet's trajectory in 1758 (three astronomers) to the Mathematical Table Project of 1938 (more than 450 workers) — already embodied the key concepts of human computation: decomposition of a complex problem into basic operations, explicit instructions (the "computing plan"), redundancy for accuracy, and dual criteria of correctness and efficiency. The story of computation is therefore not a story of humans being replaced by machines, but of the unit size of human involvement shrinking while the scale grows to millions of web users.

The definition of human computation

The authors adopt a precise definition: computation is the mapping of some input representation to some output representation using an explicit, finite set of instructions (an algorithm). Human computation is computation carried out by humans. A human computation system is an intelligent system that organizes humans to carry out the process of computation — whether performing basic operations, handling control flow, or synthesizing the algorithm itself. This definition deliberately excludes volunteer distributed computing (SETI@Home, Folding@Home) where humans merely donate CPU cycles, and it distinguishes human computation from broader concepts such as collective intelligence (emergent behavior, no explicit control) or social computing (any technology supporting social behavior).

Explicit control as the distinguishing feature

The concept that separates human computation from crowdsourcing, social computing, and collective intelligence is explicit control: the system explicitly decomposes tasks, assigns them, and designs mechanisms to ensure truthful outputs. Crowdsourcing is a method a human computation system can use (open call distribution), but is not synonymous with it. Collective intelligence assumes that much of the computational outcome is determined by the natural dynamics of a group, whereas human computation places explicit control at the greater end of the spectrum.

The three aspects: what, who, how

Every human computation system has three aspects where explicit control can be applied. The "what" aspect: deciding which operations to perform, in what order — covered by algorithms (Chapters 2–3). The "who" aspect: deciding to whom each operation is assigned — covered by task routing (Chapter 4). The "how" aspect: designing the environment that motivates accurate, truthful work — covered by markets and task design (Chapters 5–6).

The AI problem landscape

Three categories of problems are driving human computation. Perceptual tasks (object recognition, music classification, protein folding) remain far from human-level machine performance — e.g., the winning algorithm in the 2010 Large Scale Visual Recognition Challenge achieved only 71.8% accuracy on 200,000 photos. Language tasks (sentiment analysis, machine translation, NLP training data creation) require large hand-labeled corpora that are expensive to produce. Computationally intensive search problems (the Traveling Salesman Problem, graph coloring, protein folding) benefit from human heuristics and intuition — the FoldIt game demonstrated that amateur players could improve protein folding solutions substantially by exploiting a willingness to backtrack that computer algorithms lacked, and the Phylo game applied similar logic to genome alignment.

Key ideas

• The word "computer" originally meant a person; human computation is the return of that original meaning at internet scale.
• Human computation is precisely defined by explicit control, algorithmic decomposition, and measurable correctness/efficiency criteria — not just "using crowds."
• The field sits at the intersection of AI, machine learning, HCI, mechanism design, and psychology; its research questions span all five.
• Redundancy for accuracy, division of labor, and clear instructions were features of 18th-century organized computation projects and remain central to modern systems.
• Perceptual, linguistic, and combinatorially hard problems share the property of being easy for humans and hard for machines — making them the natural domain of human computation.

Key takeaway

Human computation reclaims humans as computational workers within a rigorous algorithmic framework, and the core research agenda is to understand how to design systems that decompose problems, route tasks, and incentivize truthfulness at scale.

---

Chapter 2 — Human Computation Algorithms

Central question

What is a human computation algorithm, how does it relate to traditional algorithms, and how can we program, evaluate, and compare algorithms that include humans in the loop?

Main argument

Grounding human computation in classical algorithm theory

The chapter argues for treating human computation algorithms using the same vocabulary as traditional algorithms — operations, control structures, correctness, efficiency — rather than inventing a new vocabulary. Following Knuth, an algorithm is a finite set of rules with five properties: input, output, finiteness, effectiveness, and definiteness. Human computation algorithms share all five but the last two raise new challenges. Effectiveness (that each operation can be done in finite time without ambiguous interpretation) is harder to guarantee: a human asked "find suspicious activities in a security video" cannot be given an exhaustive rule list. Definiteness (repeatability) is desirable but difficult; one response is to decompose ambiguous operations (e.g., "edit this sentence") into simpler, more precise ones (e.g., "find," "fix," "verify").

Building blocks: operations, controls, and program synthesis

There are two building blocks of any algorithm: operations (basic input-output tasks) and controls (instructions for how operations are ordered). Controls fall into four types: sequence (executing operations in order), parallel (executing operations simultaneously and aggregating outputs), selection (conditional branching), and repetition (looping until a condition is met). Human computation algorithms can be described as combinations of these. The ESP Game is a parallel algorithm: multiple pairs of players simultaneously label the same image until enough high-confidence tags are collected. A human version of QuickSort can sort objects by subjective attributes (e.g., attractiveness) that have no numerical scale. A distributed graph coloring algorithm assigns each worker a local view of the graph and asks them to change their node's color to resolve conflicts; Kearns et al. found that people reached optimal solutions across a variety of graph structures.

Beyond performing operations, humans can also control the flow — deciding when to halt, whether to decompose further, or which operation to execute next. TurKontrol learns a Partially Observable Markov Decision Process (POMDP) that automatically determines which operation to execute next based on the probability that an average worker can improve the current solution. The most flexible role is program synthesis: humans create the entire algorithm from scratch. Turkomatic coordinates workers to come up with an algorithm by running a meta-algorithm with three operations — solve, decompose, and combine — leaving design decisions (what operations to use, how granular each is, how many workers per step) entirely up to the humans.

Programming frameworks

TurKit and Boto provide programmatic access to Amazon Mechanical Turk in Java and Python, wrapping the MTurk REST API. TurKit introduces the once primitive, which stores the result of each completed HIT in a database so that if a program crashes, previously executed tasks do not need to be re-posted. The join/fork primitive supports parallelization. A pseudocode example shows a recipe-creation algorithm where the loop condition (is_done) and the step generation (get_next_step) are each posted as HITs; the machine handles control flow while humans provide the substance.

Evaluating human computation algorithms

Two criteria — correctness and efficiency — apply, with modifications.

For correctness: because human outputs are noisy, traditional pre/post-condition specifications do not apply cleanly. Correctness is instead evaluated in terms of robustness — can the algorithm produce the right answer in the presence of noise?

For efficiency, three sub-criteria apply:

• Time complexity (run-time complexity and clock time): an algorithm that scales exponentially in the number of operations is impractical when each operation costs money.
• Query complexity: the number of queries to human oracles. Active learning reduces query complexity by selecting only the most informative inputs to label, rather than labeling everything naively. Variations include uncertainty sampling, query-by-committee, and expected error reduction.
• Cost-effectiveness: total monetary cost, a dimension absent from classical algorithm analysis. Algorithms can be compared in terms of how well they simultaneously satisfy cost and quality requirements; some work on automated task design determines parameters such as granularity and price automatically.

Key ideas

• Human computation algorithms share structure with traditional algorithms (operations, controls) but require relaxed definitions of effectiveness and definiteness due to human variability.
• The four control structures (sequence, parallel, selection, repetition) are sufficient to describe existing human computation systems, including the ESP Game and distributed graph coloring.
• Program synthesis — humans designing the algorithm itself — is the most powerful and open-ended role humans can play.
• TurKit's once primitive solves the crash-recovery problem, a fundamental engineering issue unique to human computation programming.
• Query complexity and cost-effectiveness are two evaluation criteria that have no direct analogue in classical algorithm analysis.
• Active learning can dramatically reduce query complexity by selecting only informative inputs for human labeling.

Key takeaway

Human computation algorithms are genuine algorithms with operations, controls, and measurable efficiency properties; the key departure from classical analysis is that correctness is probabilistic, efficiency must account for monetary cost, and the richest role for humans is program synthesis itself.

---

Chapter 3 — Aggregating Outputs

Central question

When multiple human computers produce potentially noisy or conflicting outputs for the same task, how do we combine them to infer the correct answer — especially when no ground truth is available?

Main argument

Why redundancy is necessary

Human computation systems routinely assign the same task to multiple workers because any single worker's output can be inaccurate (lack of expertise, fatigue, bad intent), because truthful-response mechanisms require synchronization that most crowdsourcing platforms do not provide, and because even perfectly honest workers may disagree due to individual differences in perception. The challenge is to know which outputs to trust, which to discard, and how to merge them.

Objective versus cultural truth

A first distinction separates two types of outputs. Some functions have objective truth — an answer that is external to human judgment, such as whether a patient has cancer. Human computation is used here because the ground truth is expensive or impractical to obtain directly (a biopsy), and the best estimate is the aggregate of several expert opinions. Other functions have cultural truth — the answer is defined by shared beliefs among the sample, such as whether a piece of music evokes "serenity" or whether a website is pornographic. Mothers and college students differ systematically in the latter judgment; cultural background affects which facial features are considered beautiful. In both cases, the authors assume that a hidden consensus can be approximated from redundant responses of multiple workers — but if every worker lies or is systematically biased, aggregation cannot recover the truth.

Latent class models for classification

Most automated aggregation methods, including simple majority vote, are instances of latent class models — generative probabilistic models where the observed outputs are votes from M workers on N tasks, and the latent variables are the true answers plus hidden factors such as worker competence and task difficulty. Formally, given an M×N output matrix O (possibly sparse), the goal is to estimate the true answer Y_n for each task n.

The simplest model is majority vote, which assumes every worker has the same fixed error rate ε < 1/2:

Yn = argmaxj ∏{m=1}^{M} P(O{nm} = o{nm} | Yn = j)

A shortcoming of majority vote is that it treats all workers as equally reliable. The seminal Dawid-Skene model (1979) introduced worker-specific confusion matrices π^(k), where π^(k)_{jl} specifies how likely clinician k will declare state l when the true state is j. The EM algorithm iteratively (i) estimates true states by weighting worker votes according to current competence estimates, then (ii) re-estimates competence matrices given the current true-state beliefs. Cultural Consensus Theory from anthropology takes a similar approach for culturally correct answers. More sophisticated extensions model task difficulty and worker biases simultaneously (Welinder et al.'s bird classification model).

Learning from imperfect data

When redundant labeling is not feasible (each task may be done only once), Dekel et al. propose training a classifier on unfiltered data, then using the learned hypothesis to identify and prune workers whose labels systematically disagree with it. Two types of noise persist even after aggregation: inaccurate labels (handled by noise-tolerant learning methods), and redundant free-form labels — synonyms, spelling variants, and conversational text produced by games like TagATune. The TagATune dataset illustrates the tension between fun (open vocabulary) and clean data (closed labels).

Aggregating complex outputs

Beyond classification, many tasks produce structured outputs. For ranking: the Matchin game has players perform paired comparisons (A vs. B), and three aggregation methods — ELO, TrueSkills, and SVD — combine pairwise preferences into a global ranking. The rank aggregation problem has been studied extensively in social choice theory. For clustering: workers can specify must-link and cannot-link constraints between pairs, and clustering algorithms incorporate these constraints via consensus clustering. For text: information fusion techniques can merge multiple image descriptions into a single summary; machine translation research on selecting or merging multiple translated sentences is directly applicable. For beliefs: prediction markets aggregate probabilistic beliefs about future events from many participants, using financial incentives to elicit truthful reporting; peer prediction markets extend this to subjective assessments.

Key ideas

• Redundancy is structurally necessary in human computation due to noise, absence of truthful-response mechanisms, and individual differences.
• The distinction between objective and cultural truth matters for choosing which aggregation model to apply.
• Majority vote, while simple, ignores worker competence heterogeneity; Dawid-Skene latent class models address this directly.
• The EM algorithm is the standard inference technique for latent class models; it alternates between estimating true answers and estimating worker competence.
• Structured outputs (rankings, clusterings, sentences, beliefs) require different aggregation approaches that are underused in current human computation systems.
• Open-vocabulary task designs (fun but noisy) create post-processing challenges that closed-vocabulary designs avoid.

Key takeaway

Output aggregation is a probabilistic inference problem over hidden ground truth, with the Dawid-Skene family of latent class models as the foundational tool, extended to handle worker heterogeneity, task difficulty, and structured output types.

---

Chapter 4 — Task Routing

Central question

How should computational tasks be matched to workers — before computation occurs — in order to maximize output quality, minimize cost, and satisfy both system and worker objectives?

Main argument

Why routing matters

Although the wisdom of crowds can approximate expert performance by aggregating many non-expert outputs, worker competence does matter — and proactive matching of workers to tasks they are suited for constitutes a point of intervention upstream of computation. A doctor verifying the fact "Bordetella Pertussis is a bacterium" outperforms a non-expert at less cost in time and errors. Beyond domain expertise, other task characteristics matter: difficulty (a task should challenge without overwhelming), engagement (a game needs the right mix of known and novel clips), and geographic proximity (participatory sensing). Task routing is defined as the problem of deciding which worker performs which task.

Push versus pull

Routing methods divide into two classes. In the push approach, the system assigns tasks to workers actively; workers are passive receivers. This approach works well when worker characteristics are known or can be estimated, and when the system expects workers to be assigned tasks rather than search for them (as in games with a purpose). In the pull approach, workers actively search for tasks themselves; the system provides search interfaces, visualizations, and recommendations. Pull works well when worker turnover is high and modeling individual workers is impractical.

Push: allocation and matching when information is complete

When the system knows workers' qualifications and task requirements, task routing becomes a combinatorial optimization problem. Assigning tasks to qualified workers is an instance of the weighted exact set-cover problem (NP-hard); greedy algorithms find approximate solutions. If no qualified worker can do a task directly, the system can decompose it into subtasks — e.g., French-to-Italian translation by routing first to a French-English worker and then an English-Italian worker. This can be modeled as a multi-commodity flow problem and solved via linear programming.

The matching literature (stable matching, the marriage problem) offers relevant mechanisms: given two sets of agents (workers and tasks) each with preferences, find a stable matching in which no agent prefers to switch. The chapter notes that the connections between matching theory and task routing are not yet established but are promising.

Push: inference under uncertainty

When worker characteristics are unknown, the system faces an exploration-exploitation tradeoff: explore (probe a worker to learn their characteristics) or exploit (assign the task to the current best candidate). Proactive learning models this tradeoff explicitly. Each routing decision δ is characterized by a risk-utility tuple {R(δ), U(δ)}, where risk is the cost and utility depends on worker characteristics (reliability, willingness to answer). IEThres (Interval Estimation) keeps a confidence interval over each worker's success probability and selects the action with the highest upper bound — balancing uncertainty with estimated performance. The interval shrinks as more observations arrive. A Bayesian extension handles workers whose accuracy varies over time (improving with experience, degrading with fatigue).

Pull: search, recommendation, and peer routing

In current Mechanical Turk, workers refresh a simple search page looking for the most recently posted tasks — an inefficiency documented by Chilton et al. that causes tasks with few associated HITs to remain incomplete. Better search and visualization interfaces would reduce starved queues. Task recommendation systems apply collaborative filtering and content-based methods from e-commerce to suggest tasks that match a worker's expertise and interests. Peer routing is a more radical variant: instead of the system or the worker self-assigning, workers route tasks to other workers — analogous to academic paper review. A peer routing system gives workers the option to accept, reject, or nominate a more expert colleague.

Evaluation criteria

A task routing policy can be evaluated on four dimensions: accuracy (do routed workers produce better outputs?), discovery (do workers find tasks faster?), efficiency (are tasks completed more quickly?), and worker motivation (are workers satisfied?).

Key ideas

• Task routing is a proactive quality-control intervention: matching the right worker to the right task before computation is cheaper than filtering bad outputs afterward.
• Push vs. pull is not a binary choice but depends on worker expectations, scale, and how reliably worker characteristics can be inferred.
• When worker characteristics are known, routing is a combinatorial optimization problem; when unknown, it becomes a sequential decision problem with exploration-exploitation tradeoffs.
• The IEThres algorithm provides a principled way to balance discovery of worker quality against exploitation of known good workers.
• Peer routing — workers routing tasks to each other based on expertise — is an underexplored design space.
• Temporal and spatial routing (assigning tasks at the best time and location for a worker) is an open research direction.

Key takeaway

Task routing transforms worker-to-task assignment from a random or worker-chosen process into a principled optimization problem, with push algorithms using combinatorial and decision-theoretic methods and pull algorithms leveraging worker intelligence and recommendation systems.

---

Chapter 5 — Understanding Workers and Requesters

Central question

Who are the workers and requesters in human computation systems, what motivates them, and how should systems be designed to sustain their participation?

Main argument

Markets as the design unit

A human computation market is a pool of individuals whose motivations align with the system's computational goals. The chapter surveys six distinct market types, each with different worker profiles, motivations, and design implications. No single design fits all; an effective system provides multiple avenues for workers with different objectives to achieve them.

Paid crowdsourcing: Amazon Mechanical Turk

Amazon Mechanical Turk (released late 2005) names its tasks HITs — Human Intelligence Tasks. By 2010 it had an estimated 400,000 workers. Ninety percent of HITs pay less than 10 cents. Typical tasks are classification, transcription, and content creation. Surveys of Mechanical Turk workers reveal demographic asymmetries: US workers are predominantly female (>2:1 ratio), while Indian workers are predominantly male; US workers treat it as a secondary income source and source of entertainment, while Indian workers treat it as a primary income. Average earnings are approximately $2.00 per hour. The name echoes an 18th-century chess-playing machine that was secretly operated by a concealed human — a fitting metaphor.

Security and access: reCAPTCHA

CAPTCHAs prove humanness by presenting a visual recognition challenge that automated programs cannot easily solve. Over 200 million CAPTCHAs are typed daily, representing roughly 500,000 hours of human effort. reCAPTCHA repurposes this effort: each reCAPTCHA presents two words — one whose answer is known (the control) and one whose answer is unknown (a word that OCR failed to recognize from scanned books). If the user types the control word correctly, the system trusts their transcription of the unknown word. Over 750 million people — more than 10% of humanity — have helped digitize at least one word through reCAPTCHA. The security-usability tradeoff is real: audio CAPTCHAs have a 43–46% success rate for blind users, improved to 68.5% by interface optimization alone without changing the underlying CAPTCHA.

Gamers: games with a purpose

Over 200 million users play online games weekly. Casual games have low barrier to entry, simple controls, short play sessions (5–20 minutes), and gender-neutral inclusive design — making them suitable for the mass market. Luis von Ahn introduced the idea of using casual games for computation in 2002 with the ESP Game. GWAP.com hosts six such games: the ESP Game (image labels), TagATune (music tags), Verbosity (common sense facts), Squigl (object location via tracing), Matchin (image preference ranking), FlipIt (image similarity), and PopVideo (video tags). The fundamental design tension: making the game more fun (e.g., open vocabulary in TagATune) yields noisier data; constraining to clean data makes the game less attractive.

Citizen science

Citizen science projects enlist non-scientists in data collection and analysis. Historical examples include the National Audubon Society Christmas Bird Count (running since 1900, over 50,000 participants in 2005) and the American Association of Variable Star Observers (AAVSO). Galaxy Zoo attracted 200,000 participants from 113 countries making over 100 million galaxy classifications, leading to new discoveries. eBird (Cornell Lab of Ornithology) attracted 500,000 users and 21 million bird records over five years. The most successful citizen science projects convert an existing hobby (birding, stargazing) into science — not the reverse. Human computation research can help citizen science by bridging knowledge gaps with intelligent tutoring and routing algorithms, and by using bias-detection techniques to correct over-reporting of easily detected species.

Learners

When hard-for-computers problems also happen to be educational tasks, computation and learning can be combined. Duolingo is the prime example: students learning a foreign language by translating web content produce high-quality translations because their comprehension is verified by the task itself. Translating the Spanish Wikipedia from English would cost approximately $50,000,000 at 5 cents per word; Duolingo does it for free. The key design principle is that the system must interleave known items (to teach) with unknown items (to compute), weighting novice answers by past performance. The broader research questions are open: how to design curricula, when to trust novice outputs, and how to pair experts with learners.

Temporary markets

Natural disasters (earthquakes, oil spills) mobilize large numbers of motivated workers for short periods. Retaining or recovering these workers after the event ends is structurally difficult. Design implications for temporary markets remain largely unexplored.

Supporting workers and requesters

Workers need interfaces that enable them to search for tasks, visualize tradeoffs between competing tasks, monitor productivity, communicate grievances, understand competition via leaderboards, find community support, and rate requesters. Requesters need to automate task design, visualize the available worker pool, meet deadlines and budgets, and rate workers. Trends suggest requesters will increasingly access crowds through everyday applications — Soylent embeds crowdsourcing in a word processor; CrowdDB and Qurk expose Mechanical Turk as a queryable database where human-computed predicates are indistinguishable from machine-computed ones.

Key ideas

• Worker motivation is heterogeneous even within the same market; effective systems provide multiple parallel paths to worker satisfaction.
• reCAPTCHA demonstrates that computation can be productively embedded in tasks users perform for entirely different reasons (proving humanness).
• The fun-data quality tradeoff in games with a purpose is a structural design challenge with no universal solution.
• Citizen science succeeds most reliably when it converts an existing hobby into science, not when it tries to make science into a hobby.
• Duolingo shows that education and computation can be fused when the educational task is also the computational task.
• Future systems will treat requesters as end users, hiding the complexity of crowdsourcing behind application-level APIs.

Key takeaway

Human computation markets span paid workers, CAPTCHA solvers, gamers, citizen scientists, and learners — each with distinct motivations — and the system must be designed to satisfy heterogeneous worker objectives while delivering on the requester's computational goals.

---

Chapter 6 — The Art of Asking Questions

Central question

How should tasks be designed — and what mechanisms should govern worker interactions — to ensure that human computers generate accurate, truthful outputs?

Main argument

Why task design matters

A spammer who is not rewarded unless their answer agrees with the majority is less likely to cheat; a toddler who is asked to count oranges instead of abstract numbers can probably compute correctly; a computer scientist who is told the arithmetic interpretation of "+" will not give an answer of 1. The chapter argues that the way questions are asked is itself a point of quality control — placed at the time of computation, distinct from the pre-computation intervention (routing, Chapter 4) and the post-computation intervention (aggregation, Chapter 3).

Five task design dimensions

Tasks have three components: basic information (inputs, the question being asked, allowable outputs), conditions for success (when is the task considered done?), and incentives (reward/penalty structure). Five design decisions interact:

Information — Anchoring biases systematically distort human estimates. People asked "Was the percentage of African nations in the UN more or less than 10%?" give lower estimates than those asked "more or less than 65%?" Sequential context biases (assimilation and contrast) affect ratings in task sequences. Partial solutions provided by machines or prior workers can help (FoldIt players solved protein structures more effectively when given a partially solved conformation as a starting point) but can also harm (brainstorming tasks show reduced creativity when prior answers are shown, a "seeding" effect). CrowdPlan found that the first worker's answer style — short phrase vs. elaborate paragraph — propagates to subsequent workers.

Granularity — Ambiguous or multi-step tasks should be decomposed. The "find-fix-verify" pattern for crowdsourced text editing (a single ambiguous "edit" task decomposed into three: find errors, fix errors, verify fixes) controls effort variance among workers and improves output quality. Empirical testing of design parameters — Huang et al. ran 38 versions of an image labeling task varying number of images, number of tags, total HITs, and reward per HIT — can predict the optimal configuration for a given objective.

Independence — Most tasks are designed for independent workers, but collaborative platforms (Etherpad-based collaborative translation, simultaneous oil field exploration games) show that when workers coordinate, they voluntarily return even after being paid. Collaborative tasks turn task routing into a team-formation problem.

Incentives — Incentives shape whether workers participate at all, which tasks they choose, and how carefully they work. Intrinsic motivations (curiosity, status, idealism, social contact, competition, ownership) interact non-trivially with extrinsic ones — paying for tasks that workers intrinsically enjoy can demotivate them. "Target earner" psychology suggests that workers stop when they reach a self-set income target; pricing tasks at values divisible by five may help such workers visualize progress. Not paying at all can sometimes be better than paying too little.

Quality control — Dedicated quality control tasks include verification (check if an output is correct), voting (select the best output), filtering (select the worst), and merging (aggregate outputs into a single solution). Social forces — reputation, monitoring signals, community norms — complement explicit quality control tasks. Placing verifiable factual questions before subjective ones caused invalid comments to drop 43% and doubled worker effort in one study.

Eliciting truthful responses through mechanism design

Incentive-compatible mechanisms are formal rules under which participants maximize their payoff by telling the truth. The canonical example is the Vickrey-Clarke-Groves (VCG) mechanism — in a single-item auction, the highest bidder wins but pays the second-highest bid; no bidder benefits from misreporting their true valuation. In human computation, the analogue is designing rules for interactions such that workers generate accurate outputs as a natural consequence of self-interest.

Output-agreement games — The ESP Game implements output-agreement: two players are given the same image, independently enter tags, and are rewarded when their tags match. The shared image is the only common information; the only reliable strategy for matching is to enter accurate image tags. This mechanism has been applied to image preference (Matchin), music classification (HerdIt), ontology construction, and sentiment analysis.

Function computation (input-agreement) — When two players are given different inputs and must determine a relationship between them, they need to communicate truthfully to succeed. TagATune implements input-agreement: two players each hear a music clip, exchange tags, and guess whether the clips are the same or different. The system "eavesdrops" on the tag exchange to collect accurate music descriptions. The input-agreement mechanism is particularly useful for data with high description entropy — many semantically equivalent ways to describe the same item.

Asymmetric function computation (inversion problems) — Verbosity and Peekaboom use asymmetric mechanisms. In Verbosity, a describer gives clues about a secret word; the guesser must guess the word. In Peekaboom, a "boomer" progressively reveals parts of an image associated with a secret word; the "peeker" must guess the word. The auxiliary function in both cases is "what is the secret object your partner holds?" Both games face the data-quality tradeoff: allowing free communication (more fun) produces noisy data requiring post-processing; Verbosity players cheat by using clues about word length or mnemonics.

Explicitly preventing bad outputs — The ESP Game's output-agreement mechanism produces common, uninformative tags because matching on common words is the easiest strategy. Remedies include taboo words (blocking high-frequency tags), awarding higher scores for more specific labels, or introducing a third "blocker" player (KissKissBan) who tries to prevent the first two from matching, forcing them to use more specific tags. For fact verification, the complementary-agreement mechanism (Polarity game) assigns one player to select true tags and another to select false ones; players are rewarded for maximally different sets, incentivizing accuracy.

The chapter provides a comparative table of human computation games including the ESP Game, Peekaboom, Verbosity, TagATune, FoldIt, HerdIt, Categorilla, MoodSwings, Phrase Detective, and Phylo — each mapped to its mechanism type and target AI problem.

Key ideas

• Task design is a form of quality control that operates at the time of computation, not before or after.
• Five design dimensions interact: information, granularity, independence, incentives, and quality control mechanisms.
• Mechanism design provides the theoretical foundation for truthful-response mechanisms: participants maximize payoff by being honest.
• Output-agreement, input-agreement, and inversion-problem mechanisms form a taxonomy of human computation game designs.
• The fun-accuracy tradeoff is inherent in game design: freer communication produces more engagement but noisier data.
• Social forces (reputation, community norms) are underused quality control tools that complement explicit mechanisms.

Key takeaway

Designing a human computation task involves five interacting decisions about information, granularity, independence, incentives, and quality control, with game-theoretic mechanisms providing principled ways to make truthful computation the rational strategy for self-interested workers.

---

Chapter 7 — The Future of Human Computation

Central question

What are the most important open research directions that the preceding chapters have not yet addressed?

Main argument

The chapter is brief and forward-looking, identifying three directions that extend naturally from the book's framework but require dedicated research.

Interweaving human and machine intelligence

The boundary between human and automated computation is not fixed; the most powerful future systems will blur it. Machine intelligence can play three roles alongside humans:

• AI as optimizers: probabilistic output aggregation, active learning, and automated task design (covered in Chapters 2–3) are already using machines to improve human computation algorithms.
• AI as enablers: as tasks grow more complex and crowds grow larger, machine intelligence must coordinate workers and organize information displays — making humans compute better rather than replacing them.
• AI as workers: for some subtasks, machines outperform humans; hybrid systems should dynamically assign subtasks to whoever (human or machine) performs them best.

Examples of emerging hybrid systems: NELL (Never-Ending Language Learner) rapidly sifts web data for candidate facts that humans verify via a Polarity-style game. CrowdPlan uses workers to decompose high-level missions into search queries, leaving search itself to machines. Branson et al. built a computer vision algorithm for bird classification that queries Mechanical Turk workers ("is the belly red?") only when its own confidence is insufficient.

Fostering long-term relationships

Most human computation research assumes transient workers with sparse, intermittent interactions — making it hard to model individual characteristics over time. But stable long-term worker pools are possible and valuable. Ethical concerns arise: power asymmetry on Mechanical Turk allows requesters to reject work without justification, damaging workers' reputation scores without appeal. Tasks can be malicious (aimed at grabbing personal information) or impossible to complete in the allotted time. The authors call for manipulation-resistant reputation systems providing transparency and trust, which could encourage long-term participation and address worker welfare concerns.

Designing organizations and task markets

Malone et al.'s "Dawn of the E-lance Economy" framed business organizations as "mechanisms for coordination." Mechanical Turk is one organizational form, but not the only possible one. Future human computation systems might dynamically assign organizational roles — computer, checker, planner, task router, manager — creating temporary, problem-specific organizations on the fly. New market forms beyond spot markets (oDesk, Elance) and contest markets (99Designs, InnoCentive) might include auction-based allocation in which requesters and workers express preferences and tasks are matched through a market mechanism, with smart brokers acting as intermediaries.

Key ideas

• Hybrid human-machine systems — where AI optimizes, enables, and does work alongside humans — represent the frontier of human computation research.
• Long-term worker relationships are both possible and ethically necessary; reputation systems and power-balance mechanisms are prerequisites.
• Task markets will diversify beyond today's spot and contest models; auction mechanisms and smart brokers may emerge.
• Complementary computing, mixed-initiative systems, and interactive machine learning are adjacent research programs that address overlapping questions.
• Requesters may eventually be automated AI systems that request human feedback to self-monitor and improve.

Key takeaway

The future of human computation lies in three directions: tighter integration of human and machine intelligence, ethical systems that foster stable long-term worker relationships, and new organizational and market forms that go beyond current crowdsourcing platforms.

---

The book's overall argument

1. Chapter 1 (Introduction) — establishes that human computation is a precise research area defined by explicit control over algorithmic decomposition, task assignment, and output mechanisms; differentiates it from crowdsourcing, collective intelligence, and social computing; and frames the three-aspect (what/who/how) research agenda.

2. Chapter 2 (Human Computation Algorithms) — argues that the "what" aspect is best addressed by grounding human computation in classical algorithm theory, with operations, controls, and program synthesis as the design vocabulary, and correctness plus three types of efficiency as the evaluation framework.

3. Chapter 3 (Aggregating Outputs) — addresses the "what" aspect further: once computation is done, noisy outputs from multiple workers must be combined using probabilistic latent class models (Dawid-Skene and extensions) that jointly infer ground truth and worker competence, extended to structured outputs such as rankings, clusterings, and beliefs.

4. Chapter 4 (Task Routing) — turns to the "who" aspect: routing tasks to workers before computation is a principled optimization problem, ranging from combinatorial allocation under full knowledge to exploration-exploitation decision-theoretic algorithms under uncertainty, with pull methods leveraging human self-organization when worker modeling is impractical.

5. Chapter 5 (Understanding Workers and Requesters) — addresses the "how" aspect at the market level: six distinct market types (paid workers, CAPTCHA users, gamers, citizen scientists, learners, temporary workers) have different motivations, and the system must be designed to serve heterogeneous worker and requester objectives simultaneously.

6. Chapter 6 (The Art of Asking Questions) — addresses the "how" aspect at the task level: five design dimensions (information, granularity, independence, incentives, quality control) interact to determine output quality, and game-theoretic mechanisms (output-agreement, input-agreement, function computation, complementary-agreement) make truthful responses the dominant strategy for self-interested workers.

7. Chapter 7 (The Future of Human Computation) — synthesizes by projecting three open frontiers: hybrid human-machine systems, ethical long-term worker relationships, and new organizational and market forms — pointing toward a mature discipline that subsumes AI, economics, HCI, and mechanism design.

---

Common misunderstandings

Misunderstanding: Human computation is just a synonym for crowdsourcing.

The book defines human computation more narrowly. Crowdsourcing is one method a human computation system can use — distributing tasks via an open call — but it is not the defining feature. A human computation system can use a closed set of recruited workers. What defines human computation is explicit algorithmic control over decomposition, assignment, and output mechanisms, not the openness of the worker pool.

Misunderstanding: Any system that uses many humans online is a human computation system.

Volunteer distributed computing (BOINC, Folding@Home, SETI@Home) enlists millions of machines owned by humans but does not require humans to perform computation — only to donate idle CPU cycles. Participatory sensing is only human computation when participants consciously decide where and when to collect data; passive sensor carrying does not qualify. Wikipedia is not a human computation system in the strict sense because it lacks explicit control mechanisms that determine when computation halts or what outputs are final.

Misunderstanding: Human computation always involves large crowds.

The authors are explicit: human computation does not necessarily involve large crowds, and workers need not interact with each other directly or indirectly. A system assigning tasks to a small closed set of recruited experts is a human computation system. Scale is a pragmatic advantage enabled by the web, not a definitional requirement.

Misunderstanding: Majority vote is sufficient for aggregating worker outputs.

Majority vote assumes all workers have the same fixed error rate and ignores systematic biases, individual competence differences, and task difficulty. It fails particularly badly when the majority of workers are novices making correlated errors or spammers. Latent class models — even the simple Dawid-Skene model — substantially outperform majority vote in practice.

Misunderstanding: Games with a purpose primarily sacrifice data quality for player enjoyment.

The book frames games with a purpose as mechanism-design tools: the game structure is a mechanism that incentivizes truthful outputs, not merely an entertainment layer. Output-agreement, input-agreement, and inversion-problem mechanisms are formally designed so that the dominant strategy for a self-interested player is to be accurate. The fun-accuracy tradeoff is real, but the game is the mechanism, not a concession to it.

Misunderstanding: Task design is a secondary, practical concern rather than a research question.

The book places task design (Chapter 6) alongside algorithms and aggregation as a first-class research problem. The framing of questions, granularity of decomposition, incentive structures, and quality-control tasks all directly affect what gets computed and how accurately — they are the "how" aspect of the three-aspect framework and are as theoretically interesting as the "what" and "who" aspects.

---

Central paradox / key insight

The central paradox of human computation is this: computers were invented to replace human computers — to automate the calculation work that human workers performed laboriously and at great cost. Yet the most sophisticated AI systems today are still far from human-level performance on perceptual, linguistic, and combinatorially complex tasks. The solution the book proposes is to reverse the replacement logic: instead of using computers to eliminate human work, use computers to organize human work at a scale and precision that was previously impossible.

The key insight is that the Turing Machine — the theoretical foundation of all digital computing — was explicitly designed to mimic what human computers do. As Turing wrote, "digital computers may be explained by saying that these machines are intended to carry out any operations which could be done by a human computer." Human computation takes this correspondence seriously: if the Turing Machine was designed to replicate humans, then humans can stand in for Turing Machines wherever the automated version fails. The result is a productive loop: human computation systems collect labeled data that trains AI algorithms, which in turn select the most informative items for human labeling (active learning), which improves the AI further.

The deeper insight is that the distinction between "computing" and "human activity" dissolves. Typing a CAPTCHA is computation. Playing the ESP Game is computation. Classifying galaxies on Galaxy Zoo is computation. Translating a sentence to learn a language is computation. The book argues that if you define computation precisely enough, nearly any goal-directed human activity in an instrumented environment can be treated as computation — and that recognizing this opens an enormous design space for systems that deliver value to users while advancing AI.

---

Important concepts

Human computation system

A system that organizes humans to carry out the process of computation — performing basic operations, handling control flow, or synthesizing the program itself — under explicit algorithmic control. Distinguished from collective intelligence (emergent behavior, no explicit control) and voluntary computing (humans donate CPU, not cognitive effort).

Explicit control

The defining characteristic of human computation systems: the system explicitly decomposes a problem into tasks, assigns those tasks to workers, and designs mechanisms to govern outputs. The amount of explicit control is on a continuum; human computation systems are toward the high end relative to open crowdsourcing platforms like Wikipedia.

What / Who / How aspects

The three dimensions of explicit control in any human computation system. "What" governs algorithmic decomposition and output aggregation; "who" governs task routing and worker matching; "how" governs market design, task design, and incentive mechanisms.

Human computation algorithm

An algorithm (finite, effective, definite sequence of operations and controls) in which some or all operations are performed by human workers. Correctness is probabilistic rather than deterministic; efficiency includes monetary cost as a dimension absent from classical analysis.

Program synthesis

The role in which human workers create the algorithm itself — determining which operations to use, how granular each is, and how they are ordered — rather than merely executing pre-specified operations. Turkomatic and CrowdForge implement this paradigm.

Active learning (in human computation)

A machine learning framework that selects only the most informative inputs to query human workers about, minimizing query complexity. Reduces the cost of human computation by avoiding redundant labels for items the machine already classifies confidently. Variants include uncertainty sampling, query-by-committee, and expected error reduction.

Latent class model

A probabilistic generative model used for output aggregation in which the true answer Y_n and hidden worker characteristics (competence, bias) are latent variables, and the observed worker outputs are generated conditional on those latents. The EM algorithm is used for inference. The Dawid-Skene model is the canonical example.

Cultural Consensus Theory

An aggregation framework from anthropology (Romney et al.) in which culturally correct answers and individual cultural competence are jointly inferred from a matrix of responses, applicable to tasks where truth is defined by shared cultural belief rather than external fact.

Task routing

The problem of deciding to which workers each computational task should be assigned, balancing accuracy, discovery speed, efficiency, and worker motivation. Divided into push (system assigns) and pull (worker self-selects) approaches.

Push vs. pull routing

Push: the system takes full control; workers are passive receivers; appropriate when worker characteristics are known or learnable and workers expect to be assigned tasks. Pull: the system provides search and visualization interfaces; workers self-assign; appropriate when worker turnover is high and modeling is impractical.

Exploration-exploitation tradeoff

In task routing under uncertainty, the decision at each step whether to probe an unknown worker (explore) or assign the task to the current best-known worker (exploit). The IEThres algorithm manages this tradeoff using confidence intervals over success probabilities.

Games with a purpose (GWAP)

Multi-player online games in which players generate useful computational data as a by-product of play. The ESP Game, TagATune, Verbosity, Peekaboom, Matchin, FoldIt, and Phylo are examples. The game structure serves as a truth-elicitation mechanism, not merely an engagement layer.

Output-agreement mechanism

A game mechanic in which two players are given the same input and rewarded only when their independently generated outputs agree. The dominant rational strategy is to output accurate tags. Used in the ESP Game, Matchin, HerdIt, and others.

Input-agreement mechanism

A game mechanic in which two players are given different inputs (possibly the same or different objects), communicate with each other, and must determine whether their inputs are the same or different. Truthful communication maximizes reward. Used in TagATune.

Function computation mechanism

A generalization of output-agreement and input-agreement in which players compute an auxiliary function of each other's outputs (e.g., "is your input the same as mine?", "what is the secret word your partner holds?"). Verbosity, Peekaboom, and TagATune are instances.

Mechanism design

The field studying how to design rules of interaction such that self-interested agents, acting to maximize their own payoffs, collectively produce a socially optimal outcome. The VCG mechanism (highest bidder wins, pays the second-highest price) is the canonical example of an incentive-compatible auction. Applied in human computation to design games and tasks in which truthful responses are the dominant strategy.

reCAPTCHA

A security and human computation system that repurposes the effort of typing CAPTCHAs (already performed by 200 million users daily) to transcribe words from scanned books that OCR failed to recognize. Each challenge presents two words: one known (control) and one unknown; the unknown word's transcription is trusted when the control word is typed correctly.

Citizen science

Organized efforts that engage non-scientists in scientific data collection and analysis. Galaxy Zoo, eBird, and the Christmas Bird Count are examples. Human computation research can improve citizen science by providing intelligent task routing, knowledge-gap bridging algorithms, and bias-detection methods.

Duolingo

A language-learning system that doubles as a translation engine by assigning learners sentences to translate as the mechanism for teaching grammar and vocabulary. Illustrates the "learning by doing" market type where educational and computational objectives are fused.

---

References and Web Links

Primary book and edition information

• Law, Edith, and Luis von Ahn. Human Computation. Morgan & Claypool Publishers, 2011. Synthesis Lectures on Artificial Intelligence and Machine Learning, Vol. 5, No. 3. ISBN 978-1-60845-516-4.
  • Google Books listing
  • Springer/Morgan & Claypool edition
  • Amazon listing (original edition)
  • ACM Digital Library citation

Background and overview

• AAAI 2011 Tutorial: "Human Computation: Core Research Questions and State of the Art" (slides and materials)
• Author supplementary materials at humancomputation.com/book

Key foundational works cited in the book

• von Ahn, Luis, and Laura Dabbish. "Labeling Images with a Computer Game." CHI 2004. (The ESP Game)
• von Ahn, Luis. "Games with a Purpose." IEEE Computer, 2006.
• Dawid, A. P., and A. M. Skene. "Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm." Applied Statistics 28, no. 1 (1979): 20–28. (Foundational aggregation model)
• Cooper, Seth, et al. "Predicting Protein Structures with a Multiplayer Online Game." Nature 466 (2010): 756–760. (FoldIt)
• PDF of the book (hosted by NCSU)

The annual HCOMP conference (successor venue)

• AAAI Conference on Human Computation and Crowdsourcing
• Quo Vadis, HCOMP? A 12-year retrospective (arXiv 2025)

Related surveys

• A Survey of Incentives and Mechanism Design for Human Computation Systems (arXiv)
• Crowdsourcing and Human Computation, Introduction (ResearchGate)

Additional chapter summaries and study resources

These are secondary summaries and should be used alongside, rather than instead of, the original book.

• ResearchGate listing for Human Computation