R/p-calibrations.R
pcal.RdCalibrate p-values under a robust perspective so that they can be directly interpreted as either lower bounds on the posterior probabilities of point null hypotheses or lower bounds on the probabilities of type I errors.
pcal(p, prior_prob = 0.5)
| p | A numeric vector with values in the [0,1] interval. |
|---|---|
| prior_prob | A numeric vector with values in the [0,1] interval.
If |
If length(p) > 1 then pcal returns a numeric vector
with the same length as p, otherwise it returns
a numeric vector with the same length as prior_prob.
Warning messages are thrown if there are NA or NaN values in p
or in prior_prob.
pcal is a vectorized implementation of the calibration of p-values
into lower bounds for the posterior probabilities of point null hypotheses
(or lower bounds for the probabilities of type I errors) developed by
Sellke et al. (2001)
. The calibration is:
$$
\alpha(p) = (1 + \lbrack -e \, p \, log(p) \rbrack^{-1})^{-1}
$$
where \(p\) is a p-value on a classical test statistic. This
calibration assumes that both the null and the alternative hypotheses have
0.5 prior probability. We generalized the aforementioned calibration for
prior probabilities other than 0.5:
$$
\alpha(p) = \left(
1 + \frac{1 - \pi_0}{\pi_0} \, \lbrack
-e \, p \, log(p)
\rbrack^{-1}
\right)^{-1}
$$
where \(\pi_0\) is the prior probability of the null hypothesis
and \(1 - \pi_0\) is the prior probability of the alternative
hypothesis.
For each element of p, pcal returns an approximation of the smallest
posterior probability of the null hypothesis that is found by changing the
prior distribution of the parameter of interest (under the alternative
hypothesis) over wide classes of distributions. Alternatively, the output
of pcal can also be interpreted as lower bounds on the probabilities of
type I errors, which means that this calibration has both Bayesian and
Frequentist interpretations. Sellke et al. (2001)
noted that a scenario in which they definitely recommend this calibration
is when investigating fit to the null model with no explicit alternative
in mind. Pericchi and Torres (2011)
warn that despite
the usefulness and appropriateness of this p-value calibration it does not
depend on sample size, and hence the lower bounds obtained with large
samples may be conservative.
The prior_prob argument is optional and is set to 0.5 by default,
implying prior equiprobability of hypotheses. prior_prob can only be
of length equal to the length of p, in which
case each prior probability in prior_prob is used in the calibration
of the corresponding element of p, or of length 1,
in which case it will be recycled (if length(p) > 1) and the same
prior_prob value is used in the calibration of all the elements of p.
Note that pcal(p, prior_prob) is equivalent to
bfactor_to_prob(bcal(p), prior_prob).
Pericchi L, Torres D (2011).
“Quick anomaly detection by the Newcomb—Benford law, with applications to electoral processes data from the USA, Puerto Rico and Venezuela.”
Statistical Science, 26(4), 502--516.
Sellke T, Bayarri MJ, Berger JO (2001).
“Calibration of p values for testing precise null hypotheses.”
The American Statistician, 55(1), 62--71.
bcal for a p-value calibration that returns lower
bounds on Bayes factors in favor of point null hypotheses.
# Calibration of a typical "threshold" p-value: # ---------------------------------------------------------------- pcal(.05)#> [1] 0.2893499# Calibration of typical "threshold" p-values: # ---------------------------------------------------------------- pcal(c(.1, .05, .01, .005, .001))#> [1] 0.38495887 0.28934989 0.11125450 0.06717427 0.01843114# Application: chi-squared goodness-of-fit test, # lower bound on the posterior probability of the null hypothesis: # ---------------------------------------------------------------- data <- matrix(c(18, 12, 10, 12, 10, 23), ncol = 2) pcal(chisq.test(data)[["p.value"]])#> [1] 0.273104