boot.test.Rd
A statistical test function to assess three-class ROC data. It can be used for assessment of a single classifier or comparison of two independent / correlated classifiers, using the Bootstrap test.
boot.test(
x1,
y1,
z1,
x2 = 0,
y2 = 0,
z2 = 0,
dat = NULL,
paired = FALSE,
n.boot = 1000,
conf.level = 0.95,
alternative = c("two.sided", "less", "greater")
)
Non-empty numeric vectors of data from the healthy, intermediate and diseased class from Classifier 1.
Numeric vectors of data from the healthy, intermediate and diseased class from Classifier 2, only needed in a comparison of two classifiers.
A data frame of the following structure: The first column represents a factor with three levels, containing the true class membership of each measurement. The levels are ordered according to the convention of higher values for more severe disease status. The second column contains all measurements obtained from Classifier 1 (in the case of single marker assessment). In the case of comparison of two markers, column three contains the measurementss from the Classifier.
A logical indicating whether data arose from a paired setting.
If TRUE
, each class must have equal sample size for both
classifiers.
An integer incicating the number of bootstrap replicates sampled to obtain the variance of the VUS. Default is 1000.
confidence level of the interval. A numeric value between (0,1) yielding the significance level \(\alpha=1-\code{conf.level}\).
character string specifying the alternative hypothesis,
must be one of "two.sided"
(default), "greater"
or "less"
. You can specify
just the initial letter. For two sided test, notice \(H0: Z = (VUS_1-VUS_2) /
(Var(VUS_1)+Var(VUS_2)-2Cov(VUS_1,VUS_2))^{0.5}\).
A list of class "htest"
containing the following components:
the value of the Z-statistic.
the p-value for the test.
a data frame containing the estimated parameters from Classifier 1 and Classifier 2 (if specified).
a character expressing the null hypothesis.
a character string describing the alternative hypothesis.
a character string indicating what type of extended Metz--Kronman test was performed.
a character string giving the names of the data.
A data frame representing the number of NA's as well as the means and the standard deviations per class.
The covariance matrix of the VUS.
Based on the reference standard, the Bootstrap test assesses the
discriminatory power of classifiers by comparing the volumes under the ROC
surfaces (VUS). It distinguishes between single classifier assessment,
where a classifier is compared to the chance plane with VUS=1/6, and
comparison between two classifiers. The latter case tests the equality
between VUS_1 and VUS_2. The data can arise in a unpaired or paired
setting. If paired
is TRUE
, a correlation is introduced which
has to be taken into account. Therefore the sets of the two classifiers
have to have classwise equal size. The data can be input as the data
frame dat
or as single vectors x1, y1, z1, ...
. The
implemented methods to evaluate the VUS
and var(VUS),
cov(vus.1,vus.2)
are based on the empirical model assumptions and
resampling techniques. This means, there are no underlying distributions
assumed in any of the classes.
Nakas, C. T. and C. T. Yiannoutsos (2004). Ordered multiple-class ROC analysis with continuous measurements. Statistics in Medicine, 23(22), 3437–3449.
data(cancer)
data(krebs)
# investigate a single marker:
boot.test(dat = krebs[,c(1,2)], n.boot=500)
#>
#> Bootstrap test for single classifier assessment
#>
#> data: healthy intermediate diseased of Fac1
#> Z-stat = 3.8303, p-value = 0.000128
#> alternative hypothesis: true Difference in VUS is not equal to 0
#> sample estimates:
#> VUS of Fac1
#> 0.5647059
#>
# result is equal to:
x1 <- with(krebs, krebs[trueClass=="healthy", 2])
y1 <- with(krebs, krebs[trueClass=="intermediate", 2])
z1 <- with(krebs, krebs[trueClass=="diseased", 2])
boot.test(x1, y1, z1, n.boot=500)
#>
#> Bootstrap test for single classifier assessment
#>
#> data: x1 y1 and z1
#> Z-stat = 3.8303, p-value = 0.000128
#> alternative hypothesis: true Difference in VUS is not equal to 0
#> sample estimates:
#> VUS of Classifier 1
#> 0.5647059
#>
# comparison of marker 2 and 6:
boot.test(dat = krebs[,c(1,2,5)], paired = TRUE)
#>
#> Bootstrap test for comparison of two independent classifiers
#>
#> data: healthy intermediate diseased of Fac1 and healthy intermediate diseased of Fac4
#> Z-stat = 2.2198, p-value = 0.02643
#> alternative hypothesis: true Difference in VUS is not equal to 0
#> sample estimates:
#> VUS of Fac1 VUS of Fac4
#> 0.5647059 0.2830317
#>
# result is equal to:
x2 <- with(krebs, krebs[trueClass=="healthy", 5])
y2 <- with(krebs, krebs[trueClass=="intermediate", 5])
z2 <- with(krebs, krebs[trueClass=="diseased", 5])
boot.test(x1, y1, z1, x2, y2, z2, paired = TRUE)
#>
#> Bootstrap test for comparison of two independent classifiers
#>
#> data: x1 y1 z1 and x2 y2 z2
#> Z-stat = 2.2198, p-value = 0.02643
#> alternative hypothesis: true Difference in VUS is not equal to 0
#> sample estimates:
#> VUS of Classifier 1 VUS of Classifier 2
#> 0.5647059 0.2830317
#>