BAYESIAN STATISTICS THE FUN WAY

 the Myth

of the

Male and Female Brain

DAPHNA JOEL, PhD ASD LUBA VIKHANS

DANIEL

ΚΑΗΝΕΜΑ

WINNER OF THE NOBEL PRIZE IN ECONOMI

BAYESIAN STATISTICS THE FUN WAY

UNDERSTANDING STATISTICS AND PROBABILITY WITH LEGO, AND STAR WARS, RUBBER DUCKS

WILL KURT

no starch press

Myth

of the

Male and Female Brain

yesian Reasoning in the Twilight Zone When Dato Doesn't Convince You.

9. From Hypothesis Testing to Parameter Estimation

A: A Quick Introduction to R.....

B: Enough Calculus to Get By...

DAPHNA JOEL, PhD LUBA VIKHANSKI

CONTENTS IN DETAIL

ΚΑΗ

WINNER OF THE NO

"IA) masterpiece

This is one of the

XV

XVII

xviii

XIX

XIX

xix

XX

XXI

xxi

xxii xxii

ACKNOWLEDGMENTS

INTRODUCTION

Why Learn Statistics?

What is "Bayesian" Statistics?

What's in This Book

Part 1: Introduction to Probability Part II: Bayesian Probability and Prior Probabilities

Part III: Parameter Estimation

Part IV: Hypothesis Testing: The Heart of Statistics. Background for Reading the Book Now Off on Your Adventurel.

PART I: INTRODUCTION TO PROBABILITY

1 BAYESIAN THINKING AND EVERYDAY REASONING

Reasoning About Strange Experiences... Observing Data

Holding Prior Beliefs and Conditioning Probabilities Forming a Hypothesis

Spotting Hypotheses in Everyday Speech. Gathering More Evidence and Updating Your

Comparing Hypotheses...........

Beliefs.

Data Informs Belief; Belief Should Not Inform Data.

Wrapping Up

Exercises..

2 MEASURING UNCERTAINTY

What Is a Probability?.....

Calculating Probabilities by Counting Outcomes of Events. Calculating Probabilities as Ratios of Beliefs

Using Odds to Determine Probability. Solving for the Probabilities

Measuring Beliefs in a Coin Toss.

/rapping Up

cercises.

3

4

4

6

8

8

9

10

11

11

13

14

15

16 17

17

18

19

20

ation

AN HERBAL FIELD GUIDE TO PLANT FAMILIES OF NORTH AMERICA

CONTENTS IN DETAIL

XV

ACKNOWLEDGMENTS

INTRODUCTION

Why Learn Statistics?

What Is "Bayesian" Statistics?.

What's in This Book,

Part 1: Introduction to Probability.

Part II: Bayesian Probability and Prior Probabilities Part III: Parameter Estimation

Part IV: Hypothesis Testing: The Heart of Statistics

Background for Reading the Book Now Off on Your Adventure!.

PART I: INTRODUCTION TO PROBABILITY

BAYESIAN THINKING AND EVERYDAY REASONING Reasoning About Strange Experiences

Observing Data Holding Prior Beliefs and Conditioning Probabilities

Forming a Hypothesis

Spotting Hypotheses in Everyday Speech. Gathering More Evidence and Updating Your Beliefs.

Comparing Hypotheses.

Data Informs Belief; Belief Should Not Inform Data.

Wrapping Up

Exercises.

2 MEASURING UNCERTAINTY

What Is a Probability?

Calculating Probabilities by Counting Outcomes of Events.

Calculating Probabilities as Ratios of Beliefs

Using Odds to Determine Probability. Solving for the Probabilities

Measuring Beliefs in a Coin Toss.

XVII

xviii

xix

xix

xix . XX

xxi

xxi

xxii xxii

100,000+ copies sold!

344Sa

5

8

8

9

10

11

11

13

45677890 17

14

15 16

17

g Up 19 20

18THE LOGIC OF UNCERTAINTY Combining Probabilities with AND

Solving a Combination of Two Probabilities

Applying the Product Rule of Probability. Example: Calculating the Probability of Being Lote Combining Probabilities with OR Cakulating OR for Mutually Exclusive Events Using the Sum Rule for Non-Mutually Exclusive Events

Example: Calculating the Probability of Getting a Hely Fine Wrapping Up Exercises

CREATING A BINOMIAL PROBABILITY DISTRIBUTION Structure of a Binomial Distribution. Understanding and Abstracting Out the Details of Our Problem.

4

Counting Our Outcomes with the Binomial Coefficient Combinatorics: Advanced Counting with the Binomial Coefficient Calculating the Probability of the Desired Outcome.

Example: Gocha Games. Wrapping Up

Exercises..

5 THE BETA DISTRIBUTION

A Strange Scenario: Getting the Data... Distinguishing Probability, Statistics, and Inference

Collecting Data.

Calculating the Probability of Probabilities

The Beta Distribution..

Breaking Down the Probability Density Function Applying the Probability Density Function to Our Problem

Quantifying Continuous Distributions with Integration Reverse-Engineering the Gacha Game

Wrapping Up

Exercises.

PART II: BAYESIAN PROBABILITY AND PRIOR PROBABLE

AN HERBAL FIELG

PLANT FAMILI

Wpping Up

BAYES THEOREM WITH LEGO

Working Our Conditional Probabilities Visually Working Through the Moth

Wrapping Up

B THE PRIOR, LIKELIHOOD, AND POSTER

The Three Parts Investigating the Scene of a Crime

Solving for the likelihood

Calculating the Prior Normalizing the Data

Considering Alternative Hypotheses

The likelihood for Our Alternative Hypa

The Prior for Our Alternative Hypothesis The Posterior for Our Alternative Hypoth

Comparing Our Unnormalized Posteriors Wrapping Up

Exercises..

9 BAYESIAN PRIORS AND WORKING PROBABILITY DISTRIBUTIONS

C3PO's Asteroid Field Doubts.. Determining C-3PO's Beliefs

Accounting for Han's Badassery

Creating Suspense with a Posterior.

Wrapping Up

Exercises.

PART III: PARAMETER ESTIMA

10

INTRODUCTION TO AVERAGING A

Estimating Snowfall

Averaging Measurements to Minimi

Solving a Simplified Version of Our

Solving a More Extreme Case

Estimating the True Value with Wei

Defining Expectation, Mean, and A

CONDITIONAL PROBABILITY

Introducing Conditional Probability

Why Conditional Probabilities Are Important Dependence and the Revised Rules of Probability Conditional Probabilities in Reverse and Bayes' Theorem

X Contents in DetailAN HERBAL FIELD GUIDE TO PLANT FAMILIES OF NORTH AMERICA coples

THEOREM WITH LEGO

71

72

73

THE PRIOR, LIKELIHOOD, AND POSTERIOR OF BAYES THEOREM

75

75

76

Normalizing the Doo

The Likelihood for Our Alternative Hypothesis

The Prior for Our Alternative Hypothesis

The Posterior for Our Alternative Hypothesis Comparing Our Unnormalized Posteriors

Wrapping Up

BAYESIAN PRIORS AND WORKING WITH

PROBABILITY DISTRIBUTIONS

C3POs Asteroid Field Doubts

Determining C-3PO's Beliefs

Accounting for Han's Bodassery Creating Suspense with a Posterior

Wrapping Up

PART III: PARAMETER ESTIMATION

10

INTRODUCTION TO AVERAGING AND PARAMETER ESTIMATION

Estimating Snowfall

Averaging Measurements to Minimize Error Solving a Simplified Version of Our Problem

Solving a More Extreme Case Estimating the True Value with Weighted Probabilities.

Defining Expectation, Mean, and Averaging

78

79 80

81

81

100,000

sold!

vents

TION

83

84

85

*** 87

... 88

89

56

ABILITIES

93

94 ***********... 95

98

99

59

60

60

..61 62

Contenus in Detail

xite Deviation.

OUR DATA

AN HE

PLANT FAMILIES OF

THE NORMAL DISTRIBUTION

Measuring Ruses for Destordly Deeds The Noomol Distribution

Soking the Buse Problem.

Some Tricks and intuitions "N Sigmo" Bans

The Bure Distribution and the Normall Distribution

Wrapping Up

TOOLS OF PARAMETER ESTIMATION: THE POR CDF AND QUANTILE FUNCTION

Bimating the Conversion Rare for an Email Signup List The Probability Density Function. 123

Visualizing and Interpreting the PDF Working with the PDF in R..

Introducing the Cumulative Distribution Function.

Visualizing and Interpreting the CDF Finding the Median

Approximating Integrals Visually

Estimating Confidence intervals.

Using the CDF in R. The Quantile Function.

Visualizing and Understanding the Quantile Function Calculating Quantiles in R

Wrapping Up Exercises.

14

PARAMETER ESTIMATION WITH PRIOR PROBABILITIES

Predicting Email Conversion Rates

PART IV: HYPOTHESIS TESTING THE HEART OF STATISTICS

15

FROM PARAMETER ESTIMATION TO WYPOTHESIS TESTING

BUILDING A BATESIAN A/B TEST

Setting Up & Bayan A/B Ter Finding Cor Piter Play Collecting Destes

Monte Carlo Simulations

in Plow Many Worlds is the Better forint

Wrapping Up Exercises

16 INTRODUCTION TO THE BAYES FACTOR AND POSTERIOR GO

THE COMPETITION OF IDEAS

Ravisiting Bayes Thesnam

Building Hypothesis Test Using the Ratio of Posteriors

The Bayes Factor

Prior Odds, Posterior Oddis

Wrapping Up Exercises

17

BAYESIAN REASONING IN THE TWILIGHT ZONE

Bayesian Reasoning in the Twilight Zone Using the Bayes Factor to Understand the Mystic Sear Measuring the Bayes Factor

Accounting for Prior Beliefs Developing Our Own Psychic Powers.

Wrapping Upen Exercises.

18

WHEN DATA DOESN'T CONVINCE YOU A Psychic Friend Rolling Dice

Comparing Likelihoods

Incorporating Prior Odds. Considering Alternative Hypotheses

132

133

133 134

135 135

136

137

Taking in Wider Context with Priors.. Prior as a Means of Quantifying Experience. 143

138

139

175

********* 176

Arguing with Relatives and Conspiracy Theorists **** 178 ****** 179

Contents in DetoPLA

FROM HYPOTHESIS TESTING TO PARAMETER ESTIMATION Is the Carnival Game Really Fair? Considering Multiple Hypotheses Searching for More Hypotheses with R

19

Adding Priors to Our Likelihood Ratios..

Building a Probability Distribution

From the Bayes Wrapping Up

Exercises.

A A QUICK INTRODUCTION TO R

R and RStudio

Creating an R Script. Basic Concepts in R.

Data Types. Missing Values

Vectors

Functions..

Basic Functions.

Random Sampling

The runif() Function The rnorm() Function

The sample() Function Using set.seed() for Predictable Random Results Defining Your Own Functions

Creating Basic Plots..

Exercise: Simulating a Stock Price

Summary

B ENOUGH CALCULUS TO GET BY

Functions.

Determining How Far You've Run Measuring the Area Under the Curve: The Integral

Measuring the Rate of Change: The Derivative.

The Fundamental Theorem of Calculus

INDEX

Factor to Parameter Estimation

B

W

m

th

210

213

214

fa

te

re

a

fe

C

t

215

216

217

219

223

227

229INTRODUCTION

Virtually everything in life is, to some extent, uncertain. This may seem like a bit of an exaggeration, but to see the truth of it you can try a quick experiment. At the start of the day, write down something you think will hap pen in the next half-hour, hour, three hours, and six hours. Then see how many of these things happen exactly like you imagined. You'll quickly realize that your day is full of uncertainties. Even something as predictable as "I will brush my teeth" or "I'll have a cup of coffee" may not, for some reason or another, happen as you expect.AN HERBAL FIE PLANT FAMIL

For most of the uncertainties in life, we're able to get by planning our day. For example, even though traffic might make y ing commute longer than usual, you can make a pretty good estima what time you need to leave home in order to get to work on time have a super-important morning meeting, you might leave earlier wa for delays. We all have an innate sense of how to deal with uncenso tions and reason about uncertainty. When you think this way, y ing to think probabilistically.

Why Learn Statistics?

The subject of this book, Bayesian statistics, helps us get better at res ing about uncertainty, just as studying logic in school helps us to see the errors in everyday logical thinking. Given that virtually everyone deal w uncertainty in their daily life, as we just discussed, this makes the audies for this book pretty wide. Data scientists and researchers already using tistics will benefit from a deeper understanding and intuition for how they tools work. Engineers and programmers will learn a lot about how thes can better quantify decisions they have to make (I've even used Bayesian analysis to identify causes of software bugs!). Marketers and salespeople a apply the ideas in this book when running A/B tests, trying to understand their audience, and better assessing the value of opportunities. Anyone making high-level decisions should have at least a basic sense of probabile so they can make quick back-of-the-envelope estimates about the costs and benefits of uncertain decisions. I wanted this book to be something a CEO could study on a flight and develop a solid enough foundation by the time they land to better assess choices that involve probabilities and uncertainty

I honestly believe that everyone will benefit from thinking about prob lems in a Bayesian way. With Bayesian statistics, you can use mathematics to model that uncertainty so you can make better choices given limited infor mation. For example, suppose you need to be on time for work for a partic ularly important meeting and there are two different routes you could take The first route is usually faster, but has pretty regular traffic back-ups that can cause huge delays. The second route takes longer in general but is less prone to traffic. Which route should you take? What type of information would you need to decide this? And how certain can you be in your choice? Even just a small amount of added complexity requires some extra t and technique. thought

Typically when people think of statistics, they think of scientists work ing on a new drug, economists following trends in the market, analysts predicting the next election, baseball managers trying to build the best team with fancy math, and so on. While all of these are certainly fascinating uses of statistics, understanding the basics of Bayesian reasoning can help you in far more areas in everyday life. If you've ever questioned some new finding reported in the news, stayed up late browsing the web wondering if you have a rare disease, or argued with a relative over their irratibeliefs about the world, learning Bayesian statistics will help you

What is "Bayes

statistics

head of

Ba

ties re

coin

What's

avilider to get to w

FAMILIES OF NORTH AMERICA

What Is "Bayesian" Statistics?

You may be wondering what all this "Bayesian" stuff is. If you've ever taken a statistics class, it was likely based on frequentist statistics. Frequentist statistics is founded on the idea that probability represents the frequency with which something happens. If the probability of getting heads in a single coin toss is 0.5, that means after a single coin toss we can expect to get one-half of a head of a coin (with two tosses we can expect to get one head, which makes more sense).

Bayesian statistics, on the other hand, is concerned with how probabili ties represent how uncertain we are about a piece of information. In Bayesian terms, if the probability of getting heads in a coin toss is 0.5, that means we are equally unsure about whether we'll get heads or tails. For problems like coin tosses, both frequentist and Bayesian approaches seem reasonable, but when you're quantifying your belief that your favorite candidate will next election, the Bayesian interpretation makes much more sense. After the all, there's only one election, so speaking about how frequently your favorite candidate will win doesn't make much sense. When doing Bayesian statistics, we're just trying to accurately describe what we believe about the world given the information we have. One particularly nice thing about Bayesian statistics is that, because we

can view it simply as reasoning about uncertain things, all of the tools and

techniques of Bayesian statistics make intuitive sense. Bayesian statistics is about looking at a problem you face, figuring out how you want to describe it mathematically, and then using reason to solve it. There are no mysterious tests that give results that you aren't quite sure of, no distributions you have to memorize, and no traditional experiment designs you must perfectly replicate. Whether you want to figure out the probability that a new web page design will bring you more customers, if your favorite sports team will win the next game, or if we really are alone in the universe, Bayesian statistics will allow you to start reasoning about these things mathematically using just a few simple rules and a new way of look ing at problems.

What's in This Book

Here's a quick breakdown of what you'll find in this book.

Part 1: Introduction to Probability

Chapter 1: Bayesian Thinking and Everyday Reasoning This first chapter introduces you to Bayesian thinking and shows you how similar it is to everyday methods of thinking critically about a situation. We'll explore the probability that a bright light outside your window at night is a UFO based on what you already know and believe about the world.

Introduction

xix.

When you think tha

cs, helps

ic in school helps

chat virtually every

cussed, this makes the

researchers alread

ng and intuition for

carn a lot about how

e (I've even used

arketers and sale

ests, trying to d

opportunities. A

basic sense of p

ates about the

o be something at

undation by the

lities and uncen

inking about p

se mathematio

iven limited

work for a pa

es you could

c back-ups t

neral but is

information

n your choice

extra though

tists work

nalysts

best team

ating uses

elp you in

finding

you have

boutChapter 2: Measuring Uncertainty In this chapter you'll toss examples to assign actual values to your uncertainty HERBAL FIELD GUIDE TO NT FAMILIES OF NORT

of probabilities: a number from 0 to 1 that

are in your belief about something.

Uncertainty In logic we use AND OR operators to combine true or false facts. It turns out that pr Chapter 3: The Logic of ity has similar notions of these operators. We'll investigate how son about the best mode of transport to get to an appointment, chances of you getting a traffic ticket. represents ho

Chapter 4: Creating a Binomial Probability Distribution Using rules of probability as logic, in this chapter, you'll build your own ability distribution, the binomial distribution, which you can app many probability problems that share a similar structure. You' predict the probability of getting a specific famous statistician co able card in a Gacha card game.

Chapter 5: The Beta Distribution Here you'll learn about your fir continuous probability distribution and get an introduction to what makes statistics different from probability. The practice of statistics involves trying to figure out what unknown probabilities might be b more money than you lo on data. In this chapter's example, we'll investigate a mysterious con dispensing box and the chances of making

Part II: Bayesian Probability and Prior Probabilities

Chapter 6: Conditional Probability In this chapter, you'll condition probabilities based on your existing information. For example, know ing whether someone is male or female tells us how likely they are tob color blind. You'll also be introduced to Bayes' theorem, which allow us to reverse conditional probabilities.

Chapter 7: Bayes' Theorem with LEGO Here you'll gain a better intuition for Bayes' theorem by reasoning about LEGO bricks! This chapter will give you a spatial sense of what Bayes' theorem is doing mathematically.

Chapter 8: The Prior, Likelihood, and Posterior of Bayes' Theorem Bayes' theorem is typically broken into three parts, each of which per. forms its own function in Bayesian reasoning. In this chapter, you'll learn what they're called and how to use them by investigating whether an apparent break-in was really a crime or just a series of coincidences.

Chapter 9: Bayesian Priors and Working with Probability Distributions This chapter explores how we can use Bayes' theorem to better under stand the classic asteroid scene from Star Wars: The Empire Strikes Back, through which you'll gain a stronger understanding of prior probabili ties in Bayesian statistics. You'll also see how you can use entire distribu tions as your prior.

Part III: Parameter Estimation

Chapter 10: Introduction to Avera

Parameter estimation is the methe

for an uncertain value. The most

to simply average your observatic

works by analyzing snowfall level

Chapter 11: Measuring the Spr

is a useful first step in estimatin

to account for how spread out

introduced to mean absolute

deviation as ways to measure

Chapter 12: The Normal Dis

standard deviation, we get a

mates: the normal distribut

the normal distribution to

to know how certain you an

skills to time your escape c

Chapter 13: Tools of Para

Quantile Function Her

tile function to better un

making. You'll estimate

what insights each provi

Chapter 14: Parameter

The best way to impro

probability. In this cha

about the past success

estimate the true cor

Chapter 15: From P

Building a Bayesiar

values, we need a wi

a hypothesis. You'l

are in a new methe

Part IV: Hypothesis Te

Chapter 16: Intr

Competition of

ing if you might

another approa

worried you sh

Chapter 17: B

you believe in

mind-reading

The Twilight

Introduction