After a long break due to a vacation and other taking care of some other more urgent issues, I have started resuming work on fbroc. I hope to have version 0.3.0 out in a few weeks. The main new feature (handling paired ROC curves) is already completed, but I need to polish the code and add documentation and more examples.
I also finally fixed a sort-of-bug when trying to plot or calculate confidence intervals for a ROC curve with a large (> number of negative samples. By default, the number of grid points into which the interval was subdivided to calculate confidence intervals for the TPR was set to the maximum of 100 and the number of negative samples.
This in turn led to an integer overflow when trying to allocate enough memory to store the TPR at each of these points for every bootstrap iteration. Changing the default to a more reasonable 250 also nicely reduces the time needed to plot the curve.
Another change will be that I will now use the S3 generic function perf to get performance estimates, instead of perf.roc. Paired ROC curves will be a different object and this will help make the interface intuitive. Some example code:
# Generate some example data
y <- rep(c(TRUE, FALSE), each = 500)
x1 <- rnorm(1000) + y
x2 <- 0.5*x1 + 0.5*rnorm(1000) + y
# Single ROC curve
result.boot <- boot.roc(x1, y)
perf(result.boot, "tpr", fpr = 0.2)
Bootstrapped ROC performance metric
Metric: TPR at a fixed FPR of 0.2
Bootstrap replicates: 1000
Observed: 0.586
Std. Error: 0.037
95% confidence interval:
0.52 0.66
# Paired ROC curve
Bootstrapped ROC performance metric
Metric: TPR at a fixed FPR of 0.2
Bootstrap replicates: 1000
Classifier 1:
Observed:0.586
Std. Error: 0.037
95% confidence interval:
0.52 0.666
Classifier 2:
Observed:0.878
Std. Error: 0.019
95% confidence interval:
0.844 0.918
Delta:
Observed:-0.292
Std. Error: 0.033
95% confidence interval:
-0.356 -0.226
Correlation: 0.47