R function to calculate model performance measure adjusted for predictor distributions.

Description

This package provides the function to estimate model performance measures (L_1, L_2, C-statistics). The difference in the distribution of predictors between two datasets (training and validation) is adjusted by a density ratio estimate.

Details

Index of help topics:

APPEstimation-package   R function to calculate model performance
                        measure adjusted for predictor distributions.
appe.glm                C-statistics adjusted for predictor
                        distributions
appe.lm                 L_1 and L_2 errors adjusted for predictor
                        distributions
cvalest.bin             Estimation of C-statistics
densratio.appe          A wrapper function

Author(s)

Eisuke Inoue, Hajime Uno

Maintainer: Eisuke Inoue <eisuke.inoue@marianna-u.ac.jp>

References

Sugiyama, M., Suzuki, T. & Kanamori, T. Density Ratio Estimation in Machine Learning. Cambridge University Press 2012. ISBN:9781139035613.

Examples

set.seed(100)

# generating learning data
n0  = 100
Z   = cbind(rbeta(n0, 5, 5), rbeta(n0, 5, 5))
Y   = apply(Z, 1, function (xx) {
        rbinom(1, 1, (1/(1+exp(-(sum(c(-2,2,2) * c(1,xx)))))))})
dat = data.frame(Y=Y, Za=Z[,1], Zb=Z[,2])

# the model to be evaluated
mdl = glm(Y~., binomial, data=dat)

# validation dataset, with different centers on predictors
n1   = 100
Z1   = cbind(rbeta(n1, 6, 4), rbeta(n1, 6, 4))
Y1   = apply(Z1, 1, function (xx) {
         rbinom(1, 1, (1/(1+exp(-(sum(c(-2,2,2) * c(1,xx)))))))})
dat1 = data.frame(Y=Y1, Za=Z1[,1], Zb=Z1[,2])

# calculation of L1 and L2 for this model
appe.glm(mdl, dat, dat1, reps=0)

C-statistics adjusted for predictor distributions

Description

Calculates adjusted C statistics by predictor distributions for a generalized linear model with binary outcome.

Usage

appe.glm(mdl, dat.train, dat.test, method = "uLSIF", sigma = NULL,
         lambda = NULL, kernel_num = NULL, fold = 5, stabilize = TRUE, 
         qstb = 0.025, reps = 2000, conf.level = 0.95)

Arguments

Value

Adjusted and non-adjusted estimates of C-statistics are provided as matrix form. "Cstat" indicates non-adjusted version, "C adjusted by score" indicates adjusted version by linear predictors distribution, and "C adjusted by predictors" indicates adjusted version by predictor distributions (multi-dimensionally). For confidence intervals, "Percentile" indicates a confidence interval by percentile method and "Approx" indicates approximated versions by Normal distribution.

Examples

set.seed(100)

# generating learning data
n0  = 100
Z   = cbind(rbeta(n0, 5, 5), rbeta(n0, 5, 5))
Y   = apply(Z, 1, function (xx) {
        rbinom(1, 1, (1/(1+exp(-(sum(c(-2,2,2) * c(1,xx)))))))})
dat = data.frame(Y=Y, Za=Z[,1], Zb=Z[,2])

# the model to be evaluated
mdl = glm(Y~., binomial, data=dat)

# validation dataset, with different centers on predictors
n1   = 100
Z1   = cbind(rbeta(n1, 6, 4), rbeta(n1, 6, 4))
Y1   = apply(Z1, 1, function (xx) {
         rbinom(1, 1, (1/(1+exp(-(sum(c(-2,2,2) * c(1,xx)))))))})
dat1 = data.frame(Y=Y1, Za=Z1[,1], Zb=Z1[,2])

# calculation of L1 and L2 for this model
appe.glm(mdl, dat, dat1, reps=0)

L_1 and L_2 errors adjusted for predictor distributions

Description

Calculates adjusted L_1 and L_2 errors by predictor distributions for a linear model.

Usage

appe.lm(mdl, dat.train, dat.test, method = "uLSIF", sigma = NULL,
        lambda = NULL, kernel_num = NULL, fold = 5, stabilize = TRUE,
        qstb = 0.025, reps = 2000, conf.level = 0.95)

Arguments

Value

Adjusted and non-adjusted estimates of L_1 and L_2 errors are provided as matrix form. "L1" and "L2" indicate non-adjusted versions, "L1 adjusted by score" and "L2 adjusted by score" indicate adjusted versions by linear predictors distribution, "L1 adjusted by predictors" and "L2 adjusted by predictors" indicate adjusted versions by predictor distributions (multi-dimensionally). For confidence intervals, "Percentile" indicates a confidence interval by percentile method and "Approx" indicates approximated versions by Normal distribution.

Examples

set.seed(100)

# generating development data
n0  = 100
Z   = cbind(rbeta(n0, 3, 3), rbeta(n0, 3, 3))
Y   = apply(Z, 1, function(xx) { rlnorm(1, sum(c(1, 1) * xx), 0.3) })
dat = data.frame(Za=Z[,1], Zb=Z[,2], Y=Y)

# the model to be evaluated
mdl = lm(Y~ Za + Zb, data=dat)

# generating validation dataset 
n1   = 100
Z1   = cbind(rbeta(n0, 3.5, 2.5), rbeta(n0, 3.5, 2.5))
Y1   = apply(Z1, 1, function(xx) { rlnorm(1, sum(c(1, 1) * xx), 0.3) })
dat1 = data.frame(Za=Z1[,1], Zb=Z1[,2], Y=Y1)

# calculation of L1 and L2 for this model
appe.lm(mdl, dat, dat1, reps=0)

Estimation of C-statistics

Description

Calculates C-statistics. Individual case weight can be incorporated.

Usage

  cvalest.bin(Y, scr, wgt = NULL)

Arguments

Value

C-statistics is provided.

A wrapper function

Description

A wrapper function to use "densratio" function from the densratio package.

Usage

densratio.appe(xtrain, xtest, method = "uLSIF", sigma = NULL,
               lambda = NULL, kernel_num = NULL, fold = 5,
               stabilize = TRUE,  qstb = 0.025)

Arguments