Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration

Jonathan D. Cohen, Samuel McClure, Angela J. Yu

Research output: Contribution to journalArticle

438 Citations (Scopus)

Abstract

Many large and small decisions we make in our daily lives - which ice cream to choose, what research projects to pursue, which partner to marry - require an exploration of alternatives before committing to and exploiting the benefits of a particular choice. Furthermore, many decisions require re-evaluation, and further exploration of alternatives, in the face of changing needs or circumstances. That is, often our decisions depend on a higher level choice: whether to exploit well known but possibly suboptimal alternatives or to explore risky but potentially more profitable ones. How adaptive agents choose between exploitation and exploration remains an important and open question that has received relatively limited attention in the behavioural and brain sciences. The choice could depend on a number of factors, including the familiarity of the environment, how quickly the environment is likely to change and the relative value of exploiting known sources of reward versus the cost of reducing uncertainty through exploration. There is no known generally optimal solution to the exploration versus exploitation problem, and a solution to the general case may indeed not be possible. However, there have been formal analyses of the optimal policy under constrained circumstances. There have also been specific suggestions of how humans and animals may respond to this problem under particular experimental conditions as well as proposals about the brain mechanisms involved. Here, we provide a brief review of this work, discuss how exploration and exploitation may be mediated in the brain and highlight some promising future directions for research.

Original languageEnglish (US)
Pages (from-to)933-942
Number of pages10
JournalPhilosophical Transactions of the Royal Society B: Biological Sciences
Volume362
Issue number1481
DOIs
StatePublished - May 29 2007
Externally publishedYes

Fingerprint

trade-off
brain
Brain
Ice Cream
Behavioral Sciences
ice cream
Policy Making
Ice
research projects
Reward
Research
Uncertainty
Animals
uncertainty
Costs and Cost Analysis
familiarity
Costs
animals
ice
animal

Keywords

  • Decision making
  • Exploration
  • Learning
  • Neurotransmitters
  • Prefrontal cortex
  • Uncertainty

ASJC Scopus subject areas

  • Agricultural and Biological Sciences(all)
  • Agricultural and Biological Sciences (miscellaneous)

Cite this

Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. / Cohen, Jonathan D.; McClure, Samuel; Yu, Angela J.

In: Philosophical Transactions of the Royal Society B: Biological Sciences, Vol. 362, No. 1481, 29.05.2007, p. 933-942.

Research output: Contribution to journalArticle

@article{b5449690b0044e698f09d26a8ceb79cf,
title = "Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration",
abstract = "Many large and small decisions we make in our daily lives - which ice cream to choose, what research projects to pursue, which partner to marry - require an exploration of alternatives before committing to and exploiting the benefits of a particular choice. Furthermore, many decisions require re-evaluation, and further exploration of alternatives, in the face of changing needs or circumstances. That is, often our decisions depend on a higher level choice: whether to exploit well known but possibly suboptimal alternatives or to explore risky but potentially more profitable ones. How adaptive agents choose between exploitation and exploration remains an important and open question that has received relatively limited attention in the behavioural and brain sciences. The choice could depend on a number of factors, including the familiarity of the environment, how quickly the environment is likely to change and the relative value of exploiting known sources of reward versus the cost of reducing uncertainty through exploration. There is no known generally optimal solution to the exploration versus exploitation problem, and a solution to the general case may indeed not be possible. However, there have been formal analyses of the optimal policy under constrained circumstances. There have also been specific suggestions of how humans and animals may respond to this problem under particular experimental conditions as well as proposals about the brain mechanisms involved. Here, we provide a brief review of this work, discuss how exploration and exploitation may be mediated in the brain and highlight some promising future directions for research.",
keywords = "Decision making, Exploration, Learning, Neurotransmitters, Prefrontal cortex, Uncertainty",
author = "Cohen, {Jonathan D.} and Samuel McClure and Yu, {Angela J.}",
year = "2007",
month = "5",
day = "29",
doi = "10.1098/rstb.2007.2098",
language = "English (US)",
volume = "362",
pages = "933--942",
journal = "Philosophical Transactions of the Royal Society B: Biological Sciences",
issn = "0800-4622",
publisher = "Royal Society of London",
number = "1481",

}

TY - JOUR

T1 - Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration

AU - Cohen, Jonathan D.

AU - McClure, Samuel

AU - Yu, Angela J.

PY - 2007/5/29

Y1 - 2007/5/29

N2 - Many large and small decisions we make in our daily lives - which ice cream to choose, what research projects to pursue, which partner to marry - require an exploration of alternatives before committing to and exploiting the benefits of a particular choice. Furthermore, many decisions require re-evaluation, and further exploration of alternatives, in the face of changing needs or circumstances. That is, often our decisions depend on a higher level choice: whether to exploit well known but possibly suboptimal alternatives or to explore risky but potentially more profitable ones. How adaptive agents choose between exploitation and exploration remains an important and open question that has received relatively limited attention in the behavioural and brain sciences. The choice could depend on a number of factors, including the familiarity of the environment, how quickly the environment is likely to change and the relative value of exploiting known sources of reward versus the cost of reducing uncertainty through exploration. There is no known generally optimal solution to the exploration versus exploitation problem, and a solution to the general case may indeed not be possible. However, there have been formal analyses of the optimal policy under constrained circumstances. There have also been specific suggestions of how humans and animals may respond to this problem under particular experimental conditions as well as proposals about the brain mechanisms involved. Here, we provide a brief review of this work, discuss how exploration and exploitation may be mediated in the brain and highlight some promising future directions for research.

AB - Many large and small decisions we make in our daily lives - which ice cream to choose, what research projects to pursue, which partner to marry - require an exploration of alternatives before committing to and exploiting the benefits of a particular choice. Furthermore, many decisions require re-evaluation, and further exploration of alternatives, in the face of changing needs or circumstances. That is, often our decisions depend on a higher level choice: whether to exploit well known but possibly suboptimal alternatives or to explore risky but potentially more profitable ones. How adaptive agents choose between exploitation and exploration remains an important and open question that has received relatively limited attention in the behavioural and brain sciences. The choice could depend on a number of factors, including the familiarity of the environment, how quickly the environment is likely to change and the relative value of exploiting known sources of reward versus the cost of reducing uncertainty through exploration. There is no known generally optimal solution to the exploration versus exploitation problem, and a solution to the general case may indeed not be possible. However, there have been formal analyses of the optimal policy under constrained circumstances. There have also been specific suggestions of how humans and animals may respond to this problem under particular experimental conditions as well as proposals about the brain mechanisms involved. Here, we provide a brief review of this work, discuss how exploration and exploitation may be mediated in the brain and highlight some promising future directions for research.

KW - Decision making

KW - Exploration

KW - Learning

KW - Neurotransmitters

KW - Prefrontal cortex

KW - Uncertainty

UR - http://www.scopus.com/inward/record.url?scp=34250348767&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=34250348767&partnerID=8YFLogxK

U2 - 10.1098/rstb.2007.2098

DO - 10.1098/rstb.2007.2098

M3 - Article

VL - 362

SP - 933

EP - 942

JO - Philosophical Transactions of the Royal Society B: Biological Sciences

JF - Philosophical Transactions of the Royal Society B: Biological Sciences

SN - 0800-4622

IS - 1481

ER -