Due to vacation and other work related stuff, I will not be active for the next week either on this site or developing fbroc. When I come back the next thing to do is to upgrade the Shiny app; in addition to supporting the new features in fbroc 0.3 I want to check out the shiny dashboards package.
Yesterday, I submitted fbroc 0.3.0 to CRAN and it was accepted shortly afterwards. By now it should have been built for most platforms. As usual, I will spent the next time to upgrade the Shiny application as well. If you haven’t read it before, the main new feature in 0.3.0 is the analysis of paired ROC curves.
Afterwards I will have to decide whether I want to polish what is there, add some documentation and a vignette to release a v1.0. The alternative would be add one more major feature. Currently what I want to add most are some functionalities related to cutoff optimization.
After a long break due to a vacation and other taking care of some other more urgent issues, I have started resuming work on fbroc. I hope to have version 0.3.0 out in a few weeks. The main new feature (handling paired ROC curves) is already completed, but I need to polish the code and add documentation and more examples.
I also finally fixed a sort-of-bug when trying to plot or calculate confidence intervals for a ROC curve with a large (> number of negative samples. By default, the number of grid points into which the interval was subdivided to calculate confidence intervals for the TPR was set to the maximum of 100 and the number of negative samples.
This in turn led to an integer overflow when trying to allocate enough memory to store the TPR at each of these points for every bootstrap iteration. Changing the default to a more reasonable 250 also nicely reduces the time needed to plot the curve.
Another change will be that I will now use the S3 generic function perf to get performance estimates, instead of perf.roc. Paired ROC curves will be a different object and this will help make the interface intuitive. Some example code:
# Generate some example data y <- rep(c(TRUE, FALSE), each = 500) x1 <- rnorm(1000) + y x2 <- 0.5*x1 + 0.5*rnorm(1000) + y # Single ROC curve result.boot <- boot.roc(x1, y) perf(result.boot, "tpr", fpr = 0.2) Bootstrapped ROC performance metric Metric: TPR at a fixed FPR of 0.2 Bootstrap replicates: 1000 Observed: 0.586 Std. Error: 0.037 95% confidence interval: 0.52 0.66 # Paired ROC curve Bootstrapped ROC performance metric Metric: TPR at a fixed FPR of 0.2 Bootstrap replicates: 1000 Classifier 1: Observed:0.586 Std. Error: 0.037 95% confidence interval: 0.52 0.666 Classifier 2: Observed:0.878 Std. Error: 0.019 95% confidence interval: 0.844 0.918 Delta: Observed:-0.292 Std. Error: 0.033 95% confidence interval: -0.356 -0.226 Correlation: 0.47