| Title: | Efficient Sequential Testing with Evidence Ratios |
| Version: | 0.2.0 |
| Date: | 2017-12-10 |
| Description: | An implementation of sequential testing that uses evidence ratios computed from the weights of a set of models. These weights correspond either to Akaike weights computed from the Akaike Information Criterion (AIC) or the Bayesian Information Criterion (BIC) and following Burnham & Anderson (2004, <doi:10.1177/0049124104268644>) recommendations, or to pseudo-BMA weights computed from the WAIC or the LOO-IC of models fitted with 'brms' and following Yao et al. (2017, <doi:10.48550/arXiv.1704.02030>). |
| License: | MIT + file LICENSE |
| LazyData: | yes |
| RoxygenNote: | 6.0.1 |
| Depends: | R (≥ 3.3.0) |
| Imports: | brms, lme4, dplyr, magrittr, tidyr, ggplot2, rlang, foreach, doParallel, cowplot |
| URL: | https://github.com/lnalborczyk/ESTER |
| BugReports: | https://github.com/lnalborczyk/ESTER/issues |
| Suggests: | knitr, rmarkdown |
| VignetteBuilder: | knitr |
| NeedsCompilation: | no |
| Packaged: | 2017-12-10 14:04:16 UTC; Ladislas |
| Author: | Ladislas Nalborczyk
 [aut, cre] |
| Maintainer: | Ladislas Nalborczyk <ladislas.nalborczyk@gmail.com> |
| Repository: | CRAN |
| Date/Publication: | 2017-12-10 14:21:14 UTC |
Efficient Sequential Testing with Evidence Ratios
Description
The ESTER package implements sequential testing based on evidence ratios computed from the Akaike weights of a set of models. These weights are being computed using either the Akaike Information Criterion (AIC) or the Bayesian Information Criterion (BIC).
Details
See vignette("ESTER") for a general introduction and overview.
Author(s)
Ladislas Nalborczyk
Maintainer: Ladislas Nalborczyk <ladislas.nalborczyk@gmail.com>
See Also
Computes the Akaike Information Criterion
Description
Computes the Akaike Information Criterion of a model. Except when the number of observations is much larger than the number of parameters (i.e., n / k > 40), we apply the second-order bias correction for small samples (AICc), as suggested by Burnham & Anderson (2002, 2004).
Usage
aic(mod)
Arguments
mod |
A fitted model of class |
Author(s)
Ladislas Nalborczyk <ladislas.nalborczyk@gmail.com>
References
Burnham, K. P., \& Anderson, D. R. (2002). Model Selection and Multimodel Inference: A Practical Information-Theoretical Approach. 2d ed. New York: Springer-Verlag.
Burnham, K. P., \& Anderson, D. R. (2004). Multimodel inference: Understanding AIC and BIC in model selection. Sociological Methods and Research, 33(2), 261-304.
See Also
Examples
data(mtcars)
mod1 <- lm(mpg ~ cyl, mtcars)
aic(mod1)
Analysing the results of simulations ran with simER
Description
Analysing the results of simulations ran with simER.
It computes the average sample number (ASN) at which the boundary is attained
(either the lower or the upper one), the percentage of hits of the lower
boundary as well as hits of the upper boundary, and the percentage of
trajectories that did not hit none of the boundaries.
Usage
analysER(sim)
Arguments
sim |
A |
Value
An object of class data.frame, which contains the average
sample number (ASN) at which the boundary is attained (either the lower or
the upper one), the percentage of hits of the lower boundary as well as hits
of the upper boundary, and the percentage of trajectories that did not hit
none of the boundaries (and thus end at nmax).
Author(s)
Ladislas Nalborczyk <ladislas.nalborczyk@gmail.com>
See Also
Examples
## Not run:
library(ESTER)
sim <- simER(cohensd = 0.8, nmin = 20, nmax = 100, boundary = 10, nsims = 100, ic = bic)
analysER(sim)
## End(Not run)
Computes the Bayesian Information Criterion
Description
Computes the Bayesian Information Criterion of a model (Schwarz, 1978).
Usage
bic(mod)
Arguments
mod |
A fitted model of class |
Author(s)
Ladislas Nalborczyk <ladislas.nalborczyk@gmail.com>
References
Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461-464.
See Also
Examples
data(mtcars)
mod1 <- lm(mpg ~ cyl, mtcars)
bic(mod1)
Simulating many sequential testing with evidence ratios and plotting their distribution
Description
Simulating many sequential evidence ratios using simER, keeps the last
of each simulation, and plotting their distribution.
Usage
distER(cohensd, nmin, nmax, nsims, ic = bic)
Arguments
cohensd |
Expected effect size |
nmin |
Minimum sample size from which start computing ERs |
nmax |
Maximum sample size at which stop computing ERs |
nsims |
Number of experiments to simulate. |
ic |
Indicates whether to use the aic or the bic |
Author(s)
Ladislas Nalborczyk <ladislas.nalborczyk@gmail.com>
See Also
Examples
## Not run: distER(cohensd = 0.6, nmin = 20, nmax = 100, nsims = 100, ic = bic)
Computes Akaike weights or pseudo-BMA weights for a set of models
Description
Returns a table with weights of a set of models, based on various
information criteria. Currently, ictab supports the computation of
Akaike weights from the aic or the bic computed on lm
or merMod models, as well as the computation of pseudo-BMA weights,
computed from the WAIC or LOOIC of brmsfit models.
Usage
ictab(mods, ic, ...)
Arguments
mods |
Should be a named list of models, of class |
ic |
Indicates which information criterion to use. Current supported
information criteria include |
... |
Additional parameters to be passed to |
Value
An object of class data.frame, which contains the value of the
information criterion (either AIC, BIC, WAIC or LOOIC), the number of parameters
(k for AIC and BIC or p for WAIC or LOOIC), the delta_IC (for AIC and BIC) or the
elpd for models compared with WAIC or LOOIC, and the weight of each
model (Akaike weights for AIC or BIC and pseudo-BMA weights for WAIC or LOOIC).
Author(s)
Ladislas Nalborczyk <ladislas.nalborczyk@gmail.com>
References
Burnham, K. P., \& Anderson, D. R. (2002). Model Selection and Multimodel Inference: A Practical Information-Theoretical Approach. 2d ed. New York: Springer-Verlag.
Burnham, K. P., \& Anderson, D. R. (2004). Multimodel inference: Understanding AIC and BIC in model selection. Sociological Methods and Research, 33(2), 261-304.
Yao, Y. P., Vehtari, A., Simpson, D., \& Gelman, A. (2017). Using stacking to average Bayesian predictive distributions.
See Also
Examples
library(ESTER)
data(mtcars)
mod1 <- lm(mpg ~ cyl, mtcars)
mod2 <- lm(mpg ~ cyl + vs, mtcars)
mod3 <- lm(mpg ~ cyl + vs + I(vs^2), mtcars)
mod4 <- lm(mpg ~ cyl * vs, mtcars)
mods <- list(mod1 = mod1, mod2 = mod2, mod3 = mod3, mod4 = mod4)
ictab(mods, aic)
ictab(mods, bic)
## Not run:
library(brms)
mod1 <- brm(mpg ~ cyl, mtcars)
mod2 <- brm(mpg ~ cyl + vs, mtcars)
mods <- list(m1 = mod1, m2 = mod2)
ictab(mods, LOO, reloo = TRUE, k_threshold = 0.6, cores = 2)
## End(Not run)
Plotting the results of simER
Description
Plotting the results of simER.
Usage
## S3 method for class 'simER'
plot(x, log = TRUE, hist = TRUE, ...)
Arguments
x |
A |
log |
Should the y-axis be log-transformed ? |
hist |
Should plot the histogram of simulations hitting either the lower, the upper boundary, or stopping at nmax ? |
... |
Further arguments passed to |
Author(s)
Ladislas Nalborczyk <ladislas.nalborczyk@gmail.com>
Computes sequential evidence ratios
Description
Computes sequential evidence ratios, either based on the AIC or the BIC.
Supported models currently include lm, merMod, or brmsfit models.
When data involve repeated measures (and so multiple lines per subject),
a column indicating the subject "id" should be provided to the id argument.
If nothing is passed to the id argument, seqER will suppose
that there is only one observation (i.e., one line) per subject.
Usage
seqER(ic = bic, mod1, mod2, nmin = 10, id = NULL, boundary = Inf,
blind = FALSE, nsims = NULL)
Arguments
ic |
Indicates whether to use the aic or the bic. |
mod1 |
A model of class |
mod2 |
A model of class |
nmin |
Minimum sample size from which start to compute sequential evidence ratios. |
id |
If applicable (i.e., repeated measures), name of the "id" column of your dataframe, in character string. |
boundary |
The Evidence Ratio (or its reciprocal) at which the run is stopped as well |
blind |
If true, the function only returns a "continue or stop" message |
nsims |
Number of permutation samples to evaluate (is ignored if blind = TRUE) |
Author(s)
Ladislas Nalborczyk <ladislas.nalborczyk@gmail.com>
See Also
Examples
## Not run:
data(mtcars)
mod1 <- lm(mpg ~ cyl, mtcars)
mod2 <- lm(mpg ~ cyl + disp, mtcars)
seqER(ic = bic, mod1, mod2, nmin = 10)
# Example with ten permutation samples
data(mtcars)
mod1 <- lm(mpg ~ cyl, mtcars)
mod2 <- lm(mpg ~ cyl + disp, mtcars)
seqER(ic = bic, mod1, mod2, nmin = 10, nsims = 10)
# Example with blinding
data(mtcars)
mod1 <- lm(mpg ~ cyl, mtcars)
mod2 <- lm(mpg ~ cyl + disp, mtcars)
seqER(ic = bic, mod1, mod2, nmin = 10, boundary = 10, blind = TRUE)
# Example with repeated measures
library(lme4)
data(sleepstudy)
mod1 <- lmer(Reaction ~ Days + (1|Subject), sleepstudy)
mod2 <- lmer(Reaction ~ Days + I(Days^2) + (1|Subject), sleepstudy)
seqER(ic = bic, mod1, mod2, nmin = 10, id = "Subject", nsims = 10)
# Example with brmsfit models
library(brms)
mod1 <- brm(Reaction ~ Days + (1|Subject), sleepstudy)
mod2 <- brm(Reaction ~ Days + I(Days^2) + (1|Subject), sleepstudy)
seqER(ic = WAIC, mod1, mod2, nmin = 10, id = "Subject", nsims = 10)
## End(Not run)
Computes sequential evidence ratios for a given data set and permutation samples
Description
Computes sequential evidence ratios for a given data set as well as
for order_nb random permutations of this dataset. When data involve repeated
measures (and so multiple lines per subject), a column indicating the
subject "id" should be provided to the id argument. If nothing
is passed to the id argument, seqERboot will suppose
that there is only one observation (i.e., one line) per subject.
Usage
seqERboot(ic, mod1, mod2, nmin, id = NULL, order_nb)
Arguments
ic |
Indicates whether to use the aic or the bic. |
mod1 |
A model of class |
mod2 |
A model of class |
nmin |
Minimum sample size from which start to compute sequential evidence ratios. |
id |
If applicable (i.e., repeated measures), name of the "id" column of your dataframe, in character string. |
order_nb |
Number of permutation samples to evaluate. |
Author(s)
Ladislas Nalborczyk <ladislas.nalborczyk@gmail.com>
See Also
Examples
## Not run:
data(mtcars)
mod1 <- lm(mpg ~ cyl, mtcars)
mod2 <- lm(mpg ~ cyl + disp, mtcars)
seqERboot(ic = bic, mod1, mod2, nmin = 10, order_nb = 20)
## End(Not run)
Simulates sequential testing with evidence ratios
Description
Simulates one or many sequential testing with evidence ratios from independent two-groups comparisons, as a function of sample size and standardized mean difference. Evidence ratios are computed from the so-called Akaike weights from either the Akaike Information Criterion or the Bayesian Information Criterion.
Usage
simER(cohensd = 0, nmin = 20, nmax = 100, boundary = 10, nsims = 20,
ic = bic, cores = 2, verbose = FALSE)
Arguments
cohensd |
Expected effect size |
nmin |
Minimum sample size from which start computing ERs |
nmax |
Maximum sample size at which stop computing ERs |
boundary |
The Evidence Ratio (or its reciprocal) at which the run is stopped as well |
nsims |
Number of simulated samples (should be dividable by cores) |
ic |
Indicates whether to use the aic or the bic |
cores |
Number of parallel processes. If cores is set to 1, no parallel framework is used (default is two cores). |
verbose |
Show output about progress |
Value
An object of class data.frame, which contains...
Author(s)
Ladislas Nalborczyk <ladislas.nalborczyk@gmail.com>
See Also
Examples
## Not run:
sim <- simER(cohensd = 0.8, nmin = 20, nmax = 100, boundary = 10,
nsims = 100, ic = bic, cores = 2, verbose = TRUE)
plot(sim, log = TRUE, hist = TRUE)
## End(Not run)