Statistical analysis - Revision history

Elembis: Formatting tweaks.

2006-07-17T06:11:02Z

Formatting tweaks.

Tycho44 at 07:28, 2 June 2006

2006-06-02T07:28:33Z

Lint: +Category:Statistics

2006-05-25T00:08:15Z

+Category:Statistics

Tycho44: testing navbox for search guide

2006-05-20T20:36:14Z

testing navbox for search guide

Tycho44: link search discussions together

2006-05-20T20:34:30Z

link search discussions together

Tycho44: /* Technical notes */ grammar

2006-05-20T19:47:56Z

Technical notes: grammar

Tycho44: /* How many failures to be sure? */

2006-05-16T22:59:51Z

How many failures to be sure?

Tycho44: /* Approximate find rate */

2006-05-16T22:45:05Z

Approximate find rate

Tycho44: /* Technical notes */

2006-05-16T22:44:02Z

Technical notes

Tycho44: explanatory stats

2006-05-16T22:43:22Z

explanatory stats

New page

Here are some introductory discussions to the statistical analysis used to determine find rates.

==Approximate find rate==

Suppose that we have performed N searches resulting in S successes (and F = N-S failures).

Our best guess for the find percentage p is given by:
p = S/N

Example: 371 searches on the beach yield 25 pieces of driftwood. N=371, S=25, p = '''6.7%'''. The chance of finding driftwood on the beach is around 6.7%. (May 16, 2006)

==General 95% confidence interval==

How accurate is our experimentally estimated find rate? Usually, the underlying true find rate will be speculated to be within a range, such as 6.7% +/-5.1% (or 1.6%-11.8%), with an associated confidence (95% certainty). There are several ways to compute the maximum error E.

A simple 95% confidence interval around our determined probability can be estimated by approximating with a normal distribution, using the central limit theorem. For N trials, the margin of error around our experimental mean is +/- 0.98/sqrt(N) with 95% confidence. Loosely speaking, we can be "95% certain" that the true probability is somewhere within +/- 0.98/sqrt(N) of our experimental probability.
E = 0.98/sqrt(N)

(See http://en.wikipedia.org/wiki/Margin_of_error and http://en.wikipedia.org/wiki/Checking_if_a_coin_is_fair true probability section.)

Example: 371 searches yield a maximum error of E = 0.051 = '''+/-5.1%'''.

==Narrower 95% confidence interval==
E = 1.9599*sqrt(p*(1-p)/N)

The previous formula uses the worst possible error bar, by replacing sqrt( p*(1-p)), where p is the probability of success, by 1/2. (This assumes that the p = 0.50, to obtain the maximum error bar.) This is great when testing a fair coin, but in our case the find rates are much lower, so generic error bars are too large. As long as the number of successes in our experimental data is large enough (most sources require at least 5 or 10 successes), then we can use our experimental p in the above formula for E.

Example: 371 searches and 25 finds is p = 6.7%, yielding a maximum error of E = 2.6%. This suggests a driftwood find range of '''4.1%-9.3%''' (with 95% confidence).

==Exact binomial tests==

When the number of successes S = n*p in our data is smaller than 5 (or 10 if you're feeling conservative), then our data is too skewed to use a normal distribution approximation. (Of course, there also need to be at least 5 (or 10) failures in our data, but in all of our examples successes are much rarer than failures.) In this case, it makes the most sense to compare to an exact binomial distribution for N trials of success probability p. In fact, it always makes the most sense to analyze our results using an exact binomial test, since that is exactly what is happening.

Our experimental data directly supports the hypothesis that the true find rate p = 6.7%. But how well does our data support the hypothesis that true p = 9.8%? Well, assuming a true find of 9.8%, we usually will have at least 36 successes. The chance of 25 or less successes out of 371 searches is the area under the lower tail of the binomial distribution. We can compute this probability in Excel using the BINOMDIST formula:
A<sub>-</sub> = BINOMDIST(25,371,0.098,TRUE) = 2.45%
When the true value of p is 9.8% (or more), our experimental data is extremely aberrant and unlikely – occurring in less than 1 in 40 collections.

Similarly, for true p = 4.4%, we usually have at most 17 successes. The chance of 25 or more successes is the area under the upper tail:
A<sub>+</sub> = 1 - BINOMDIST(24,371,0.044,TRUE) = 2.45%
With a true p of 4.4% (or less), our experimental data is extremely aberrant and unlikely – occurring in less than 1 in 40 collections.

This method yields a driftwood find range of '''4.4%-9.8%'''.

Statistical analysis can be somewhat contentious, but these values are in good agreement with the range determined previously.

==How many failures to be sure?==

The Excel formula BINOMDIST(0,N,p,TRUE) gives the exact probability of failing to find something in N searches, given a find rate of p. When this probability falls below 2.5%, we can be fairly certain that a find rate of p will return at least one success from N trials. (If this probability falls below 1%, we are 99% certain of at least one success in N trials.)

Hence 100 trials is good enough to strongly reject a 5% find rate (99% certainty), because BINOMDIST(0,100,0.05,TRUE) = 0.0059 < 0.01. Similarly, 200 trials will reasonably reject a 2% find rate (95% certainty) and strongly reject a 2.5% find rate (99%).

I believe that this analysis is duplicated on the Urban Dead wiki, so at some point I'll compare results…

{| border="1" cellpadding="2" style="text-align:center"
| Unsuccessful Searches || Unlikely Find Rate || Almost Certainly Not
|-
| N || p (95%) || p (99%)
|-
| 35 || 10% || 12.5%
|-
| 75 || 5% || 6%
|-
| 91 || 4% || 5%
|-
| 122 || 3% || 4%
|-
| 183 || 2% || 2.5%
|-
| 367 || 1% || 1.25%
|-
| 460 || 0.8% || 1%
|}

==Technical notes==

The following data used in above examples:

{| border="1" cellpadding="2" style="text-align:center"
| colspan="7"| Beaches (May 20, 2006)
|-
| Total Searches || Nothing || Piece of Driftwood || Gold Coin || Bottle of Beer || Crab || Bite from Crab
|-
| 371 || 322 || 25 || 16 || 2 || 4 || 2
|}

@@ Line 5: / Line 5: @@
 {| style="width:100%;margin-bottom:10px;padding:6px;border:1px solid #AAAAAA;background:#eee2e2;-moz-border-radius:1em;"
 | '''Guide to searching'''
-* If you'd like to look at or contribute to the experimental data, see [[search odds condensed]]
+* If you'd like to contribute to the experimental data, see [[search odds condensed]] or [[search odds]]
-* If you'd like to learn about or discuss the results, see [[search results]]
+* If you'd like to view the summarized results, see [[search odds results]]
 * If you'd like to learn about or discuss the underlying assumptions about game mechanics, see [[how searching works]]
 * If you'd like to learn about or discuss statistical techniques being used to analyze data, see [[statistical analysis]]

← Older revision		Revision as of 00:08, 25 May 2006
Line 105:		Line 105:
	\|}		\|}

−	[[Category:~~Guide to searching~~]]	+	[[Category:Statistics]]

← Older revision		Revision as of 20:36, 20 May 2006
Line 1:		Line 1:
		+	{\| cellpadding=5 align=center style="width:100%;"
		+	\| valign=top style="width:40%;" \|
		+	__TOC__
		+	\| valign=top style="width:60%;" \|
		+	{\| style="width:100%;margin-bottom:10px;padding:6px;border:1px solid #AAAAAA;background:#eee2e2;-moz-border-radius:1em;"
		+	\| '''Guide to searching'''
		+	* If you'd like to look at or contribute to the experimental data, see [[search odds condensed]]
		+	* If you'd like to learn about or discuss the results, see [[search results]]
		+	* If you'd like to learn about or discuss the underlying assumptions about game mechanics, see [[how searching works]]
		+	* If you'd like to learn about or discuss statistical techniques being used to analyze data, see [[statistical analysis]]
		+	\|}
		+	\|}
		+
	Here are some introductory discussions to the statistical analysis used to determine find rates.		Here are some introductory discussions to the statistical analysis used to determine find rates.
−

	==Approximate find rate==		==Approximate find rate==

← Older revision		Revision as of 20:34, 20 May 2006
Line 92:		Line 92:
	\| 371 \|\| 322 \|\| 25 \|\| 16 \|\| 2 \|\| 4 \|\| 2		\| 371 \|\| 322 \|\| 25 \|\| 16 \|\| 2 \|\| 4 \|\| 2
	\|}		\|}
		+
		+	[[Category:Guide to searching]]

@@ Line 17: / Line 17: @@
 ==Approximate find rate==
-Suppose that we have performed N searches resulting in S successes (and F = N-S failures).
+Suppose that we have performed ''n'' searches resulting in ''s'' successes and ''f'' failures (where ''s'' + ''f'' = ''n''). Our best guess for the find percentage ''p'' is given by:
-Our best guess for the find percentage p is given by:
+As an example, suppose 371 searches on the beach yield 25 pieces of driftwood, so ''n'' = 371, ''s'' = 25 and ''p'' = 25 / 371 &asymp; '''0.067'''. Given these numbers, the chance of finding driftwood on the beach is around 6.7%, or 1 find in 15 [[AP]].
 ==General 95% confidence interval==
-How accurate is our experimentally estimated find rate? Usually, the underlying true find rate will be speculated to be within a range, such as 6.7% +/-5.1% (or 1.6%-11.8%), with an associated confidence (95% certainty). There are several ways to compute the maximum error E.
+How accurate is our experimentally estimated find rate? Usually, the underlying true find rate will be speculated to be within a range, such as 6.7% &plusmn;5.1% (or 1.6%-11.8%), with an associated confidence (95% certainty). There are several ways to compute the maximum error ''E''.
 ==Narrower 95% confidence interval==
-The previous formula uses the worst possible error bar, by replacing sqrt( p*(1-p)), where p is the probability of success, by 1/2. (This assumes that the p = 0.50, to obtain the maximum error bar.) This is great when testing a fair coin, but in our case the find rates are much lower, so generic error bars are too large. As long as the number of successes in our experimental data is large enough (most sources require at least 5 or 10 successes), then we can use our experimental p in the above formula for E.
+:''E'' = 1.9599 &times; sqrt(''p'' &times; (1 &minus; ''p'') / ''n'')
-Example: 371 searches and 25 finds is p = 6.7%, yielding a maximum error of E = 2.6%. This suggests a driftwood find range of '''4.1%-9.3%''' (with 95% confidence).
+The previous formula uses the worst possible error bar, by replacing sqrt(''p'' &times; (1 &minus; ''p'')) with 0.5, where ''p'', as before, is the probability of success. (This assumes that ''p'' = 0.50, which gives the greatest margin of error.) This is great when testing a fair coin, but in our case the find rates are much lower, so generic error bars are too large. As long as the number of successes in our experimental data is large enough (most sources require at least 5 or 10 successes), then we can use our experimental ''p'' in the above formula for ''E''.
 ==Exact binomial tests==
-When the number of successes S = n*p in our data is smaller than 5 (or 10 if you're feeling conservative), then our data is too skewed to use a normal distribution approximation. (Of course, there also need to be at least 5 (or 10) failures in our data, but in all of our examples successes are much rarer than failures.) In this case, it makes the most sense to compare to an exact binomial distribution for N trials of success probability p. In fact, it always makes the most sense to analyze our results using an exact binomial test, since that is exactly what is happening.
+When the number of successes ''s'' = ''n'' &times; ''p'' in our data is smaller than 5 (or 10 if you're feeling conservative), then our data is too skewed to use a normal distribution approximation. (Of course, there also need to be at least 5 (or 10) failures in our data, but in all of our examples successes are much rarer than failures.) In this case, it makes the most sense to compare to an exact binomial distribution for ''n'' trials of success probability ''p''. In fact, it always makes the most sense to analyze our results using an exact binomial test, since that is exactly what is happening.
-Our experimental data directly supports the hypothesis that the true find rate p = 6.7%. But how well does our data support the hypothesis that true p = 9.8%? Well, assuming a true find of 9.8%, we usually will have at least 36 successes. The chance of 25 or less successes out of 371 searches is the area under the lower tail of the binomial distribution. We can compute this probability in Excel using the BINOMDIST formula:
+Our experimental data directly supports the hypothesis that the true find rate ''p'' = 6.7%. But how well does our data support the hypothesis that true ''p'' = 9.8%? Well, assuming a true find of 9.8%, we usually will have at least 36 successes. The chance of 25 or less successes out of 371 searches is the area under the lower tail of the binomial distribution. We can compute this probability in Excel using the BINOMDIST formula:
- A<sub>-</sub> = BINOMDIST(25,371,0.098,TRUE) = 2.45%
+:''A''<sub>-</sub> = BINOMDIST(25,371,0.098,TRUE) = 2.45%
-When the true value of p is 9.8% (or more), our experimental data is extremely aberrant and unlikely – occurring in less than 1 in 40 collections.
+When the true value of ''p'' is 9.8% (or more), our experimental data is extremely aberrant and unlikely, occurring in less than 1 in 40 collections. '
-Similarly, for true p = 4.4%, we usually have at most 17 successes. The chance of 25 or more successes is the area under the upper tail:
+Similarly, for true ''p'' = 4.4%, we usually have at most 17 successes. The chance of 25 or more successes is the area under the upper tail:
- A<sub>+</sub> = 1 - BINOMDIST(24,371,0.044,TRUE) = 2.45%
+:''A''<sub>+</sub> = 1 - BINOMDIST(24,371,0.044,TRUE) = 2.45%
-With a true p of 4.4% (or less), our experimental data is extremely aberrant and unlikely – occurring in less than 1 in 40 collections.
+With a true ''p'' of 4.4% (or less), our experimental data is extremely aberrant and unlikely – occurring in less than 1 in 40 collections.
-This method yields a driftwood find range of '''4.4%-9.8%'''.
+This method yields a driftwood find range of 4.4% to 9.8%.
 Statistical analysis can be somewhat contentious, but these values are in good agreement with the range determined previously.
@@ Line 66: / Line 60: @@
 For example, Water (blue), Swamps (brown), Caves (gray), Tunnels (gray), and Mountain Paths (gray) currently appear to be derelict. Are we reasonably certain of the emptiness of these locations, or should we perform additional (probably fruitless) searches?
-The Excel formula BINOMDIST(0,N,p,TRUE) gives the exact probability of failing to find something in N searches, given a find rate of p. When this probability falls below 2.5%, we can be fairly certain that a find rate of p will return at least one success from N trials. (If this probability falls below 1%, we are 99% certain of at least one success in N trials.)
+The Excel formula BINOMDIST(0,N,p,TRUE) gives the exact probability of failing to find something in ''n'' searches, given a find rate of ''p''. When this probability falls below 2.5%, we can be fairly certain that a find rate of ''p'' will return at least one success from ''n'' trials. (If this probability falls below 1%, we are 99% certain of at least one success in ''n'' trials.)
 Hence 100 trials is good enough to strongly reject a 5% find rate (99% certainty), because BINOMDIST(0,100,0.05,TRUE) = 0.0059 < 0.01. Similarly, 200 trials will reasonably reject a 2% find rate (95% certainty) and strongly reject a 2.5% find rate (99%).
@@ Line 74: / Line 68: @@
 I believe that this analysis is duplicated on the Urban Dead wiki, so at some point I'll compare results…
-{| border="1" cellpadding="2" style="text-align:center"
+{| border="1" cellpadding="5" cellspacing="1" style="background-color: #f9f9f9; border: solid 1px #ccc; border-collapse: collapse" |
-| Unsuccessful Searches || Unlikely Find Rate || Almost Certainly Not
+|-align="left" bgcolor="#ccc" valign="top"
-|-
+! Unsuccessful searches !! Unlikely find rate !! Extremely unlikely find rate
-| N || p (95%) || p (99%)
+|-bgcolor="#ddd"
 |-
 | 35 || 10% || 12.5%
@@ Line 93: / Line 88: @@
 | 460 || 0.8% || 1%
 |}
 ==Technical notes==
@@ Line 98: / Line 95: @@
 The above examples used the following data:
-{| border="1" cellpadding="2" style="text-align:center"
+{| border="1" cellpadding="5" cellspacing="1" style="background-color: #f9f9f9; border: solid 1px #ccc; border-collapse: collapse" |
-| colspan="7"| Beaches (May 16, 2006)
+|-bgcolor="#ccc" valign="top"
-|-
+!colspan="7"| Beaches (May 16, 2006)
-| Total Searches || Nothing || Piece of Driftwood || Gold Coin || Bottle of Beer || Crab || Bite from Crab
+|-align="left" bgcolor="#ddd" valign="top"
 |-
 | 371 || 322 || 25 || 16 || 2 || 4 || 2

@@ Line 9: / Line 9: @@
   p = S/N
-Example: 371 searches on the beach yield 25 pieces of driftwood. N=371, S=25, p = '''6.7%'''. The chance of finding driftwood on the beach is around 6.7%. (May 16, 2006)
+Example: 371 searches on the beach yield 25 pieces of driftwood. N=371, S=25, p = '''6.7%'''. The chance of finding driftwood on the beach is around 6.7%, or 1 find in 15 AP. (May 16, 2006)
 ==General 95% confidence interval==