pages tagged pvalue
http://meng6net.localhost/tag/pvalue/
<p><small>Copyright © 2005–2024 by <code>Meng Lu
<lumeng3@gmail.com></code></small></p>
Meng Lu's home page
ikiwiki
Tue, 16 May 2017 23:59:39 +0000

Relearning pvalue
http://meng6net.localhost/blog/relearning_pvalue/
http://meng6net.localhost/blog/relearning_pvalue/
academics
blog
fallacy
note
pvalue
statistics
Tue, 16 May 2017 23:59:39 +0000
20170516T23:59:39Z
<p>After reading <a href=
"http://www.tandfonline.com/doi/pdf/10.1080/00031305.2016.1154108">"The
ASA's statement on pvalues: context, process, and purpose"</a>,
and some other related references, here are some excerpts and notes
I took on pvalue and nullhypothesis significance testing.</p>
<ul>
<li>
<p>American Statistical Association (ASA) has stated the following
five principles about pvalues and null hypothesis significance
testing:</p>
<ol>
<li>"Pvalues can indicate how incompatible the data are with a
specified statistical model."</li>
<li>"Pvalues do not measure the probability that the studied
hypothesis is true, or the probability that the data were produced
by random chance alone."</li>
<li>" … It is a statement about data in relation to a specified
hypothetical explanation, and is not a statement about the
explanation itself."</li>
<li>"Scientific conclusions and business or policy decisions should
not be based only on whether a pvalue passes a specific
threshold."</li>
<li>"… Practices that reduce data analysis or scientific inference
to mechanical “brightline” rules (such as “p < 0.05”) for
justifying scientific claims or conclusions can lead to erroneous
beliefs and poor decisionmaking. …"</li>
<li>"Proper inference requires full reporting and
transparency."</li>
<li>"A pvalue, or statistical significance, does not measure the
size of an effect or the importance of a result."</li>
<li>"… Smaller pvalues do not necessarily imply the presence of
larger or more important effects, and larger pvalues do not imply
a lack of importance or even lack of effect. Any effect, no matter
how tiny, can produce a small pvalue if the sample size or
measurement precision is high enough, and large effects may produce
unimpressive pvalues if the sample size is small or measurements
are imprecise. …"</li>
</ol>
</li>
<li>
<p>Null hypothesis is usually a hypothesis that assumes that
observed data and its distribution is a result of random chances
rather than that of effects caused by some intrinsic mechanisms. It
is usually what is to disapprove or to reject in order to establish
evidence to or belief in that there is some real effect due to
underlying intrinsic mechanism. In turn, the details of the
statistical model used in this evaluation can be used to make
quantitative estimations on properties of the underlying
mechanism.</p>
</li>
<li>
<p>The pvalue is the probability that one has falsely rejected the
null hypothesis.</p>
<ul>
<li>The smaller is, the smaller the chance is that one has falsely
rejected the null hypothesis.</li>
<li>Being able to reject or not being able to reject the null
hypothesis may tells one if the observed data suggests that there
is an effect, however, it does not tell one how much an effect
there is and if the effect is true. See <a href=
"https://en.wikipedia.org/wiki/Effect_size">effect size</a>.</li>
<li>"a pvalue near 0.05 taken by itself offers only weak evidence
against the null hypothesis".</li>
<li>UK statistician and geneticist Sir Ronald Fisher introduced the
pvalue in the 1920s. "The pvalue was never meant to be used the
way it's used today."</li>
</ul>
</li>
<li>
<p>As ASA pvalue principle No. 3 states, the decision to reject
the null hypothesis should not be based solely on if pvalue passes
a "brightline" threshold. Rather, in order to reject the null
hypothesis, one must make a subjective judgment involving the
degree of risk acceptable for being wrong. The degree of risk of
being wrong may be specified in terms of confidence levels which
characterizes the sampling variability.</p>
</li>
<li>
<p>Alternative ways used for referring to data cherrypicking
include data dredging, significance chasing, significance questing,
selective inference, <a href=
"https://www.urbandictionary.com/define.php?term=phacking">phacking</a>,
snooping, fishing, and doubledipping.</p>
</li>
<li>
<p>"The difference between statistically significant and
statistically insignificant is not, itself, statistically
significant."</p>
</li>
<li>
<p>"According to one widely used calculation [<sup id=
"fnref:1"><a href="http://meng6net.localhost/tag/pvalue/#fn:1" rel="footnote">1</a></sup>], a pvalue of
0.01 corresponds to a falsealarm probability of at least 11%,
depending on the underlying probability that there is a true
effect; a pvalue of 0.05 raises that chance to at least 29%." See
the following figure:</p>
</li>
</ul>
<p><span class="createlink">pvalue and probable
cause.png</span></p>
<h2>Some related concepts</h2>
<ul>
<li>
<p>The <a href=
"https://en.wikipedia.org/wiki/Standard_score">standard score</a>,
or zscore is the deviation from the mean in units of standard
deviation. A small pvalue corresponds to a large positive
zscore.</p>
</li>
<li>
<p><a href=
"https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule">689599.7
rule</a></p>
</li>
<li>
<p><a href="https://en.wikipedia.org/wiki/MAGIC_criteria">MAGIC
criteria</a>.</p>
<ul>
<li>Magnitude  How big is the effect? Large effects are more
compelling than small ones.</li>
<li>Articulation  How specific is it? Precise statements are more
compelling than imprecise ones.</li>
<li>Generality  How generally does it apply?</li>
<li>Interestingness  interesting effects are those that "have the
potential, through empirical analysis, to change what people
believe about an important issue".</li>
<li>Credibility  Credible claims are more compelling than
incredible ones. The researcher must show that the claims made are
credible.</li>
</ul>
</li>
</ul>
<h2>References</h2>
<ul>
<li>
<p>"The problem with pvalues: how significant are they, really?",
phys.org Science News Wire, 2013, <a href=
"http://phys.org/wirenews/145707973/theproblemwithpvalueshowsignificantaretheyreally.html">
http://phys.org/wirenews/145707973/theproblemwithpvalueshowsignificantaretheyreally.html</a></p>
</li>
<li>
<p>Regina Nuzzo, "Scientific method: statistical errors," 2014,
<a href=
"http://folk.ntnu.no/slyderse/Nuzzo%20and%20Editorial%20%20pvalues.pdf">
http://folk.ntnu.no/slyderse/Nuzzo%20and%20Editorial%20%20pvalues.pdf</a></p>
</li>
<li>
<p>Tom Siegfried, "Odds Are, It's Wrong  Science fails to face the
shortcomings of statistics," 2010, <a href=
"https://www.sciencenews.org/article/oddsareitswrong">https://www.sciencenews.org/article/oddsareitswrong</a></p>
</li>
<li>
<p>Gelman, A., and Loken, E., "The Statistical Crisis in Science,"
American Scientist, 102., 2014, <a href=
"http://www.americanscientist.org/issues/feature/2014/6/thestatisticalcrisisinscience">
http://www.americanscientist.org/issues/feature/2014/6/thestatisticalcrisisinscience</a></p>
</li>
<li>
<p>"The vast majority of statistical analysis is not performed by
statisticians," simplystatistics.org, 2013, <a href=
"http://simplystatistics.org/2013/06/14/thevastmajorityofstatisticalanalysisisnotperformedbystatisticians/">
http://simplystatistics.org/2013/06/14/thevastmajorityofstatisticalanalysisisnotperformedbystatisticians/</a></p>
</li>
<li>
<p>"On the scalability of statistical procedures: why the pvalue
bashers just don't get it," simplystatistics.org, 2014, <a href=
"http://simplystatistics.org/2014/02/14/onthescalabilityofstatisticalprocedureswhythepvaluebashersjustdontgetit/">
http://simplystatistics.org/2014/02/14/onthescalabilityofstatisticalprocedureswhythepvaluebashersjustdontgetit/</a></p>
</li>
<li>
<p>Andrew Gelmana and Hal Sterna, The Difference Between
“Significant” and “Not Significant” is not Itself Statistically
Significant, The American Statistician, Volume 60, Issue 4, 2006,
<a href=
"http://www.tandfonline.com/doi/abs/10.1198/000313006X152649">http://www.tandfonline.com/doi/abs/10.1198/000313006X152649</a></p>
</li>
</ul>
<div class="footnotes">
<hr />
<ol>
<li id="fn:1">Goodman, "Of PValues and Bayes: A Modest Proposal,"
S. N. Epidemiology 12, 295–297 (2001), <a href=
"http://journals.lww.com/epidem/fulltext/2001/05000/of_p_values_and_bayes__a_modest_proposal.6.aspx">
http://journals.lww.com/epidem/fulltext/2001/05000/of_p_values_and_bayes__a_modest_proposal.6.aspx</a><a href="http://meng6net.localhost/tag/pvalue/#fnref:1"
rev="footnote">↩</a></li>
</ol>
</div>
/blog/relearning_pvalue/#comments