Version:	0.42.2
Date:	2023-06-28
Author:	Samuel Pawel [aut, cre]
Maintainer:	Samuel Pawel <samuel.pawel@uzh.ch>
Title:	Bayesian Analysis of Replication Studies
Description:	Provides tools for the analysis of replication studies using Bayes factors (Pawel and Held, 2022) <doi:10.1111/rssb.12491>.
License:	GPL-3
Encoding:	UTF-8
Imports:	lamW, hypergeo
Suggests:	roxygen2, tinytest
NeedsCompilation:	no
RoxygenNote:	7.2.3
URL:	https://github.com/SamCH93/BayesRep
BugReports:	https://github.com/SamCH93/BayesRep/issues
Packaged:	2023-06-28 08:59:36 UTC; sam
Repository:	CRAN
Date/Publication:	2023-06-29 15:50:16 UTC

Equality of effect size Bayes factor

Description

Computes the equality of effect size Bayes factor

Usage

BFe(to, so, tr, sr, tau, log = FALSE)

Arguments

Details

The equality of effect size Bayes factor is the Bayes factor contrasting the hypothesis of equal original and replication effect sizes H_0: \theta_o = \theta_r to the hypothesis of unequal effect sizes H_1: \theta_o \neq \theta_r. Under the hypothesis of unequal effect sizes H_1 the study specific effect sizes are assumed to be normally distributed around an overall effect size with heterogeneity standard deviation tau.

Value

The equality of effect size Bayes factor \mathrm{BF}_{01}. \mathrm{BF}_{01} > 1 indicates that the data favour the hypothesis of equal effect sizes H_0 (replication success), whereas \mathrm{BF}_{01} < 1 indicates that the data favour the hypothesis of unequal effect sizes H_1 (replication failure).

Author(s)

Samuel Pawel

References

Bayarri, M. and Mayorall, A. (2002). Bayesian Design of "Successful" Replications. The American Statistician, 56(3): 207-214. doi:10.1198/000313002155

Verhagen, J. and Wagenmakers, E. J. (2014). Bayesian tests to quantify the result of a replication attempt. Journal of Experimental Psychology: General, 145:1457-1475. doi:10.1037/a0036731

Examples

## strong evidence for unequal effect sizes
BFe(to = 1, tr = 0.5, so = sqrt(1/100), sr = sqrt(1/100), tau = 0.3)

## some evidence for equal effect sizes
BFe(to = 1, tr = 1, so = sqrt(1/200), sr = sqrt(1/200), tau = 0.3)

 

Generalized replication Bayes factor

Description

Computes the generalized replication Bayes factor

Usage

BFr(
  to,
  so,
  tr,
  sr,
  ss = 0,
  truncate = FALSE,
  log = FALSE,
  zo = NULL,
  zr = NULL,
  c = NULL,
  g = 0
)

Arguments

Details

The generalized replication Bayes factor is the Bayes factor contrasting the sceptic's hypothesis that the effect size is about zero

H_{\mathrm{S}}: \theta \sim \mathrm{N}(0, \code{ss}^2)

to the advocate's hypothesis that the effect size is compatible with its posterior distribution based on the original study and a uniform prior

H_{\mathrm{A}}: \theta \sim f(\theta \, | \, \mathrm{original~study}).

The standard replication Bayes factor from Verhagen and Wagenmakers (2014) is obtained by specifying a point-null hypothesis ss = 0 (the default).

The function can be used with two input parametrizations, either on the absolute effect scale (to, so, tr, sr, ss) or alternatively on the relative z-scale (zo, zr, c, g). If an argument on the effect scale is missing, the z-scale is automatically used and the other non-missing arguments on the effect scale ignored.

Value

The generalized replication Bayes factor \mathrm{BF}_{\mathrm{SA}}. \mathrm{BF}_{\mathrm{SA}} < 1 indicates that the data favour the advocate's hypothesis H_{\mathrm{A}} (replication success), whereas \mathrm{BF}_{\mathrm{SA}} > 1 indicates that the data favour the sceptic's hypothesis H_{\mathrm{S}} (replication failure).

Author(s)

Samuel Pawel

References

Verhagen, J. and Wagenmakers, E. J. (2014). Bayesian tests to quantify the result of a replication attempt. Journal of Experimental Psychology: General, 145:1457-1475. doi:10.1037/a0036731

Ly, A., Etz, A., Marsman, M., Wagenmakers, E. J. (2019). Replication Bayes factors from evidence updating. Behavior Research Methods, 51(6):2498-2508. doi:10.3758/s13428-018-1092-x

Pawel, S. and Held, L. (2022). The sceptical Bayes factor for the assessment of replication success. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3): 879-911. doi:10.1111/rssb.12491

See Also

BFrSMD, BFrlogOR

Examples

to <- 2
tr <- 2.5
so <- 1
sr <- 1
BFr(to = to, so = so, tr = tr, sr = sr)
BFr(zo = to/so, zr = tr/sr, c = so^2/sr^2)

 

Generalized replication Bayes factor for SMD effect sizes

Description

Computes the generalized replication Bayes factor for standardized mean difference (SMD) effect sizes

Usage

BFrSMD(
  to,
  no,
  n1o = no,
  n2o = no,
  tr,
  nr,
  n1r = nr,
  n2r = nr,
  ss,
  type = c("two.sample", "one.sample", "paired")
)

Arguments

Details

This function computes the generalized replication Bayes factor for standardized mean difference (SMD) effect sizes using an exact t-likelihood for the data instead of the normal approximation used in BFr (for details, see Section 4 in Pawel and Held, 2022). Data from both studies are summarized by t-statistics and sample sizes. The following types of t-tests are accepted:

• Two-sample t-test where the SMD represents the standardized mean difference between two group means (assuming equal variances in both groups).

• One-sample t-test where the SMD represents the standardized mean difference to the null value.

• Paired t-test where the SMD represents the standardized mean difference score.

Value

The generalized replication Bayes factor \mathrm{BF}_{\mathrm{SA}}. \mathrm{BF}_{\mathrm{SA}} < 1 indicates that the data favour the advocate's hypothesis H_{\mathrm{A}} (replication success), whereas \mathrm{BF}_{\mathrm{SA}} > 1 indicates that the data favour the sceptic's hypothesis H_{\mathrm{S}} (replication failure).

Author(s)

Samuel Pawel

References

Verhagen, J. and Wagenmakers, E. J. (2014). Bayesian tests to quantify the result of a replication attempt. Journal of Experimental Psychology: General, 145:1457-1475. doi:10.1037/a0036731

Pawel, S. and Held, L. (2022). The sceptical Bayes factor for the assessment of replication success. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3): 879-911. doi:10.1111/rssb.12491

See Also

BFr, BFrlogOR

Examples

data("SSRPexact")
morewedge2010 <- subset(SSRPexact, study == "Morewedge et al. (2010), Science")
with(morewedge2010,
     BFrSMD(to = to, n1o = n1o, n2o = n2o, tr = tr, n1r = n1r, n2r = n2r, ss = 0))

Generalized replication Bayes factor for logOR effect sizes

Description

Computes the generalized replication Bayes factor for log odds ratio (logOR) effect sizes

Usage

BFrlogOR(
  ao,
  bo,
  nTo = ao + bo,
  co,
  do,
  nCo = co + do,
  ar,
  br,
  nTr = ar + br,
  cr,
  dr,
  nCr = cr + dr,
  ss,
  method = c("integration", "hypergeo")
)

Arguments

Details

This function computes the generalized replication Bayes factor for log odds ratio (logOR) effect sizes using an exact binomial likelihood for the data instead of the normal approximation used in BFr (for details, see Section 4 in Pawel and Held, 2022).

Value

The generalized replication Bayes factor \mathrm{BF}_{\mathrm{SA}}. \mathrm{BF}_{\mathrm{SA}} < 1 indicates that the data favour the advocate's hypothesis H_{\mathrm{A}} (replication success), whereas \mathrm{BF}_{\mathrm{SA}} > 1 indicates that the data favour the sceptic's hypothesis H_{\mathrm{S}} (replication failure).

Author(s)

Samuel Pawel

References

Verhagen, J. and Wagenmakers, E. J. (2014). Bayesian tests to quantify the result of a replication attempt. Journal of Experimental Psychology: General, 145:1457-1475. doi:10.1037/a0036731

Pawel, S. and Held, L. (2022). The sceptical Bayes factor for the assessment of replication success. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3): 879-911. doi:10.1111/rssb.12491

Examples

data("SSRPexact")
balafoutas2012 <- subset(SSRPexact, study == "Balafoutas and Sutter (2012), Science")
with(balafoutas2012,
     BFrlogOR(ao = ao, bo = bo, co = co, do = do, ar = ar, br = br, cr = cr, dr = dr,
              ss = 0))

Sceptical Bayes factor

Description

Computes the sceptical Bayes factor

Usage

BFs(to, so, tr, sr, truncate = FALSE, zo = NULL, zr = NULL, c = NULL)

Arguments

Details

The sceptical Bayes factor is a summary measure of the following two-step reverse-Bayes procedure for assessing replication success:

1. Use the data from the original study to determine the standard deviation \tau_{\gamma} of a sceptical normal prior \theta \sim \mathrm{N}(0, \tau_{\gamma}^2) such that the Bayes factor contrasting the null hypothesis H_0: \theta = 0 to the sceptic's hypothesis H_{\mathrm{S}}: \theta \sim \mathrm{N}(0, \tau_{\gamma}^2) equals a specified level \gamma \in (0, 1]. This prior represents a sceptic who remains unconvinced about the presence of an effect at level \gamma.

2. Use the data from the replication study to compare the sceptic's hypothesis H_{\mathrm{S}}: \theta \sim \mathrm{N}(0, \tau_{\gamma}^2) to the advocate's hypothesis H_{\mathrm{A}}: \theta \sim f(\theta \, | \, \mathrm{original~study}). The prior of the effect size under H_{\mathrm{A}} is its posterior based on the original study and a uniform prior, thereby representing the position of an advocate of the original study. Replication success at level \gamma is achieved if the Bayes factor contrasting H_{\mathrm{S}} to H_{\mathrm{A}} is smaller than \gamma, which means that the replication data favour the advocate over the sceptic at a higher level than the sceptic's initial objection. The sceptical Bayes factor \mathrm{BF}_{\mathrm{S}} is the smallest level \gamma at which replication success can be established.

The function can be used with two input parametrizations, either on the absolute effect scale (to, so, tr, sr) or alternatively on the relative z-scale (zo, zr, c). If an argument on the effect scale is missing, the z-scale is automatically used and the other non-missing arguments on the effect scale ignored.

Value

The sceptical Bayes factor \mathrm{BF}_{\mathrm{S}}. \mathrm{BF}_{\mathrm{S}} < 1 indicates replication success, the smaller the value of \mathrm{BF}_{\mathrm{S}} the higher the degree of replication success. It is possible that the result of the replication is so inconclusive that replication success cannot be established at any level. In this case, the sceptical Bayes factor does not exist and the function returns NaN.

Author(s)

Samuel Pawel

References

Pawel, S. and Held, L. (2022). The sceptical Bayes factor for the assessment of replication success. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3): 879-911. doi:10.1111/rssb.12491

See Also

BFsSMD, BFslogOR

Examples

to <- 2
tr <- 2.5
so <- 1
sr <- 1
BFs(to = to, so = so, tr = tr, sr = sr)
BFs(zo = to/so, zr = tr/sr, c = so^2/sr^2)

Sceptical Bayes factor for SMD effect sizes

Description

Computes the sceptical Bayes factor for standardized mean difference (SMD) effect sizes

Usage

BFsSMD(
  to,
  no,
  n1o = no,
  n2o = no,
  tr,
  nr,
  n1r = nr,
  n2r = nr,
  type = c("two.sample", "one.sample", "paired")
)

Arguments

Details

This function computes the sceptical Bayes factor for standardized mean difference (SMD) effect sizes using an exact t-likelihood for the data instead of the normal approximation used in BFs (for details, see Section 4 in Pawel and Held, 2022). Data from both studies are summarized by t-statistics and sample sizes. The following types of t-tests are accepted:

• Two-sample t-test where the SMD represents the standardized mean difference between two group means (assuming equal variances in both groups).

• One-sample t-test where the SMD represents the standardized mean difference to the null value.

• Paired t-test where the SMD represents the standardized mean difference score.

Value

The sceptical Bayes factor \mathrm{BF}_{\mathrm{S}}. \mathrm{BF}_{\mathrm{S}} < 1 indicates replication success, the smaller the value of \mathrm{BF}_{\mathrm{S}} the higher the degree of replication success. It is possible that the result of the replication is so inconclusive that replication success cannot be established at any level. In this case, the sceptical Bayes factor does not exist and the function returns NaN.

Author(s)

Samuel Pawel

References

Pawel, S. and Held, L. (2022). The sceptical Bayes factor for the assessment of replication success. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3): 879-911. doi:10.1111/rssb.12491

See Also

BFs, BFslogOR

Examples

data("SSRPexact")
morewedge2010 <- subset(SSRPexact, study == "Morewedge et al. (2010), Science")
with(morewedge2010,
     BFsSMD(to = to, n1o = n1o, n2o = n2o, tr = tr, n1r = n1r, n2r = n2r))

Sceptical Bayes factor for logOR effect sizes

Description

Computes the sceptical Bayes factor for logOR effect sizes

Usage

BFslogOR(
  ao,
  bo,
  nTo = ao + bo,
  co,
  do,
  nCo = co + do,
  ar,
  br,
  nTr = ar + br,
  cr,
  dr,
  nCr = cr + dr,
  method = c("integration", "hypergeo")
)

Arguments

Details

This function computes the sceptical Bayes factor for log odds ratio (logOR) effect sizes using an exact binomial likelihood for the data instead of the normal approximation used in BFs (for details, see Section 4 in Pawel and Held, 2022).

Value

The sceptical Bayes factor \mathrm{BF}_{\mathrm{S}}. \mathrm{BF}_{\mathrm{S}} < 1 indicates replication success, the smaller the value of \mathrm{BF}_{\mathrm{S}} the higher the degree of replication success. It is possible that the result of the replication is so inconclusive that replication success cannot be established at any level. In this case, the sceptical Bayes factor does not exist and the function returns NaN.

Author(s)

Samuel Pawel

References

Pawel, S. and Held, L. (2022). The sceptical Bayes factor for the assessment of replication success. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3): 879-911. doi:10.1111/rssb.12491

See Also

BFs, BFslogOR

Examples

data("SSRPexact")
balafoutas2012 <- subset(SSRPexact, study == "Balafoutas and Sutter (2012), Science")
with(balafoutas2012,
     BFslogOR(ao = ao, bo = bo, co = co, do = do, ar = ar, br = br, cr = cr, dr = dr))

BayesRep package

Description

The BayesRep package provides various tools for Bayesian analysis of replication studies.

repPosterior visualizes the posterior distribution of the effect size based on both studies. BFs computes the sceptical Bayes factor (Pawel and Held, 2022), BFr computes the replication Bayes factor (Verhagen and Wagenmakers, 2014), and BFe computes the equality of effect size Bayes factor (Bayarri and Mayorall, 2002).

These functions take effect estimates and their standard errors from original and replication study as inputs. Throughout, original effect estimate and standard error are denoted by to and so and replication effect estimate and standard error are denoted tr and sr. It is assumed that each effect estimate is normally distributed around its true underlying effect size with variance equal to its squared standard error

\code{to} \, | \, \theta_o \sim \mathrm{N}(\theta_o, \code{so}^2) ~ \mathrm{and} ~ \code{tr} \, | \, \theta_r \sim \mathrm{N}(\theta_r, \code{sr}^2).

These assumptions may be inadequate for studies with small sample size (there are special functions for data with continuous outcomes and standardized mean difference effect size, BFsSMD and BFrSMD, and binary outcomes with log odds ratio effects, BFslogOR and BFrlogOR, which are based on the exact distribution of the data). If not specified otherwise, it is assumed that the true effect sizes from both studies are the same (\theta_o = \theta_r).

References

Bayarri, M. and Mayorall, A. (2002). Bayesian Design of "Successful" Replications. The American Statistician, 56(3): 207-214. doi:10.1198/000313002155

Verhagen, J. and Wagenmakers, E. J. (2014). Bayesian tests to quantify the result of a replication attempt. Journal of Experimental Psychology: General, 145:1457-1475. doi:10.1037/a0036731

Pawel, S. and Held, L. (2022). The sceptical Bayes factor for the assessment of replication success. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3): 879-911. doi:10.1111/rssb.12491

Data from the Social Sciences Replication Project

Description

Data from the Social Sciences Replication Project. The variables are as follows:

study

Authors, year, and journal of the original study

type

Type of effect size. Either "logOR" for log oddds ratio effect size, "SMD1" for standardized mean difference from one-sample or paired t-test, or "SMD2" for standardized mean difference from two-sample t-test

to

t-statistic from the original study (only available for "SMD1" and "SMD2")

n1o

Sample size in group 1 of the original study (only available for "SMD1" and "SMD2")

n2o

Sample size in group 2 of the original study (only available for "SMD2")

tr

t-statistic from the replication study (only available for "SMD1" and "SMD2")

n1r

Sample size in group 1 of the replication study (only available for "SMD1" and "SMD2")

n2r

Sample size in group 2 of the replication study (only available for "SMD2")

ao

Number of cases in original study treatment group (only available for "logOR")

bo

Number of non-cases in original study treatment group (only available for "logOR")

co

Number of cases in original study control group (only available for "logOR")

do

Number of non-cases in original study control group (only available for "logOR")

ar

Number of cases in replication study treatment group (only available for "logOR")

br

Number of cases in replication study control group (only available for "logOR")

cr

Number of cases in replication study control group (only available for "logOR")

dr

Number of non-cases in replication study control group (only available for "logOR")

Usage

data(SSRPexact)

Format

A data frame with 21 rows and 16 variables

Author(s)

Samuel Pawel

Source

The data were manually extracted from the Bayesian supplement of the SSRP (https://osf.io/nsxgj/). The data are licensed under CC0 1.0 Universal.

References

Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., ... Wu, H. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2, 637-644. doi:10.1038/s41562-018-0399-z

Formatting of Bayes factors

Description

Formats Bayes factors such that Bayes factors smaller than 1 are represented as ratios 1/x, where x is rounded to the specified number of digits, while Bayes factors larger than 1 are only rounded to the specified number of digits.

Usage

formatBF(BF, digits = "default")

Arguments

Details

The default formatting, which is recommended in Held and Ott (2018), is as follows: For very small Bayes factors BF < 1/1000, "< 1/1000" is returned. Bayes factors BF with 1/1000 \leq BF \leq 1/10 are formatted as 1/x where x is an integer and Bayes factors BF with 1/10 < BF < 1 as 1/x, where x is rounded to one decimal place. Accordingly, Bayes factors \leq BF < 10 are rounded to one decimal place, Bayes factors 10 \leq BF \leq 1000 are rounded to the next integer and for larger Bayes factors, "> 1000" is returned.

If digits is specified, the Bayes factor (if it is \geq 1) or its inverse (if the Bayes factor is < 1) is rounded to the number of decimal places specified and returned as a ratio if the Bayes factor is < 1.

Value

A character vector of ratios (for inputs < 1) or rounded numeric values (for inputs \geq 1) ).

Author(s)

Manuela Ott (creator of package pCalibrate), Leonhard Held (contributor of package pCalibrate), Samuel Pawel (made small changes to pCalibrate::formatBF)

References

Held, L. and Ott, M. (2018). On p-values and Bayes factors. Annual Review of Statistics and Its Application, 5, 393-419. doi:10.1146/annurev-statistics-031017-100307

Examples

(bf <- BFr(to = 2, so = 0.5, tr = 2.5, sr = 0.9))
formatBF(BF = bf)

Effect size posterior distribution

Description

Computes the posterior distribution of the effect size based on the original and replication effect estimates and their standard errors, assuming a common underlying effect size and an initial flat prior.

Usage

repPosterior(
  to,
  so,
  tr,
  sr,
  lower = min(c(to, tr)) - 4/sqrt(1/so^2 + 1/sr^2),
  upper = max(c(to, tr)) + 4/sqrt(1/so^2 + 1/sr^2),
  nGrid = 1000,
  plot = TRUE,
  CI = TRUE,
  ...
)

Arguments

Value

Plots posterior distribution of the effect size, invisibly returns a list with the data for the plot

Author(s)

Samuel Pawel

Examples

## Example from Reproducibility Project Cancer Biology
## Aird: Data from https://elifesciences.org/articles/21253 Fig4B
hro <- 25.93
lhro <- log(hro)
hroCI <- c(5.48, 122.58)
se_lhro <- diff(log(hroCI))/(2*qnorm(0.975))
hrr <- 3.75
lhrr <- log(hrr)
hrrCI <- c(1.19, 11.81)
se_lhrr <- diff(log(hrrCI))/(2*qnorm(0.975))
repPosterior(to = lhro, so = se_lhro, tr = lhrr, sr = se_lhrr)