After a long break due to a vacation and other taking care of some other more urgent issues, I have started resuming work on fbroc. I hope to have version 0.3.0 out in a few weeks. The main new feature (handling paired ROC curves) is already completed, but I need to polish the code and add documentation and more examples.
I also finally fixed a sort-of-bug when trying to plot or calculate confidence intervals for a ROC curve with a large (> number of negative samples. By default, the number of grid points into which the interval was subdivided to calculate confidence intervals for the TPR was set to the maximum of 100 and the number of negative samples.
This in turn led to an integer overflow when trying to allocate enough memory to store the TPR at each of these points for every bootstrap iteration. Changing the default to a more reasonable 250 also nicely reduces the time needed to plot the curve.
Another change will be that I will now use the S3 generic function perf to get performance estimates, instead of perf.roc. Paired ROC curves will be a different object and this will help make the interface intuitive. Some example code:
# Generate some example data y <- rep(c(TRUE, FALSE), each = 500) x1 <- rnorm(1000) + y x2 <- 0.5*x1 + 0.5*rnorm(1000) + y # Single ROC curve result.boot <- boot.roc(x1, y) perf(result.boot, "tpr", fpr = 0.2) Bootstrapped ROC performance metric Metric: TPR at a fixed FPR of 0.2 Bootstrap replicates: 1000 Observed: 0.586 Std. Error: 0.037 95% confidence interval: 0.52 0.66 # Paired ROC curve Bootstrapped ROC performance metric Metric: TPR at a fixed FPR of 0.2 Bootstrap replicates: 1000 Classifier 1: Observed:0.586 Std. Error: 0.037 95% confidence interval: 0.52 0.666 Classifier 2: Observed:0.878 Std. Error: 0.019 95% confidence interval: 0.844 0.918 Delta: Observed:-0.292 Std. Error: 0.033 95% confidence interval: -0.356 -0.226 Correlation: 0.47