intermittent issue w/ fcase in 1.16 #6448
Comments
|
if you can get some code that reliably produces the error (even if it's only intermittent), that will help a lot. There was some change to memory management in fcase to support vectorized default=, so it's most likely that's the issue. but without example code, it's hard to isolate the issue & fix it. |
|
this consistently eventually produces the error for me. library(xts)
data <- matrix(runif(4000, 0, 100), ncol = 4)
data[5,4] <- NA
data[10,1] <- NA
data[15,c(1,4)] <- NA
data <- xts(data, order.by = rep(Sys.Date(), 1000))
repeat {
fcase(is.na(data[,1]) & is.na(data[,4]), data[,1] + data[,4],
is.na(data[,1]), data[,4],
is.na(data[,4]), data[,1],
data[,1] > data[,4], data[,1] - data[,4],
data[,1] < data[,4], data[,4] - data[,1]
)
}
#Error in fcase(is.na(data[, 1]) & is.na(data[, 4]), data[, 1] + data[, :
# Argument #8 has different class than argument #2, Please make sure all output values have the same class.
> sessionInfo()
R version 4.4.1 (2024-06-14 ucrt)
Platform: x86_64-w64-mingw32/x64
Running under: Windows 11 x64 (build 22631)
Matrix products: default
locale:
[1] LC_COLLATE=English_United States.utf8 LC_CTYPE=English_United States.utf8 LC_MONETARY=English_United States.utf8 LC_NUMERIC=C LC_TIME=English_United States.utf8
time zone: America/Denver
tzcode source: internal
attached base packages:
[1] parallel stats graphics grDevices utils datasets methods base
other attached packages:
[1] jsonlite_1.8.8 readxl_1.4.3 xml2_1.3.6 curl_5.2.2 Rcpp_1.0.13 matrixStats_1.3.0 data.table_1.16.0 quantmod_0.4.26.1 TTR_0.24.4 doFuture_1.0.1 future_1.34.0 doParallel_1.0.17 iterators_1.0.14 foreach_1.5.2
[15] xts_0.14.1 zoo_1.8-12 plotrix_3.8-4
loaded via a namespace (and not attached):
[1] future.apply_1.11.2 compiler_4.4.1 stringr_1.5.1 systemfonts_1.1.0 globals_0.16.3 scales_1.3.0 fastmap_1.2.0 lattice_0.22-6 R6_2.5.1 knitr_1.48 kableExtra_1.4.0 munsell_0.5.1 svglite_2.1.3
[14] rlang_1.1.4 stringi_1.8.4 xfun_0.47 viridisLite_0.4.2 cli_3.6.3 magrittr_2.0.3 digest_0.6.37 grid_4.4.1 rstudioapi_0.16.0 lifecycle_1.0.4 evaluate_0.24.0 glue_1.7.0 cellranger_1.1.0
[27] listenv_0.9.1 codetools_0.2-20 colorspace_2.1-1 parallelly_1.38.0 rmarkdown_2.28 tools_4.4.1 htmltools_0.5.8.1 |
|
Odd. I just ran that code on my machine (with 256GB of ram) and got this: I restarted R, loaded And then (without restarting R) ran the same code again and got the segfault again. Sounds like there's a memory allocation problem. (The system is not under any load.) But then, since it is a fair (purpose-wise) comparison, I modified it to use data <- matrix(runif(4000, 0, 100), ncol = 4)
data[5,4] <- NA
data[10,1] <- NA
data[15,c(1,4)] <- NA
data <- xts::xts(data, order.by = rep(Sys.Date(), 1000))
repeat {
dplyr::case_when(
is.na(data[,1]) & is.na(data[,4]) ~ data[,1] + data[,4],
is.na(data[,1]) ~ data[,4],
is.na(data[,4]) ~ data[,1],
data[,1] > data[,4] ~ data[,1] - data[,4],
data[,1] < data[,4] ~ data[,4] - data[,1]
)
}
Error in c.xts(NA_real_, 91.7918950784951, 68.4573939070106, c(73.4160375548527, :
zero-length vectors with non-zero-length index are not allowed(I do not know if the R-4.3.2, |
|
Thinking it might be an issue relating specifically to |
|
yes. xts is a matrix w/ an index attribute. subsetting the matrix requites subsetting the index, which is prolly a bit much for fcase to get involved with. in this scenario, fcase works, since i make sure all the inputs are already aligned and dont need to be aligned via the index. so its simple enough to just have fcase deal with the raw data vectors and then convert back to xts after. this is a fairly common practice... i didnt think of it as it used to work under 1.15.4. I just put in a wrapper to handle this and it works fine. fcase <- function(..., default = NA) {
d <- lapply(list(...), FUN = as.vector)
d["default"] <- as.vector(default)
do.call(data.table::fcase, d)
} |
|
I wasn't suggesting that you should find a way to not use |
|
valgrind does not report any issues |
|
there are lots of scenarios where one needs to operate on the underlying data, then convert back to the original class. this applies not just to xts, but to underlying zoo as well. its so common, that lets decompose this into underlying issues:
FWIW, calling |
|
This example crashes for me (R-devel, At this point, If says Should |
@tdhock, I appreciate your insight and taking the steps for that test. I would not assume that a problem others reproduce using |
|
I confirm the issue on our codespace, and also that the issue is not present in 1.15.4. |
|
Great sleuthing @aitap indeed, I was able to reproduce the segfault on master even with this version that strips attributes: data <- matrix(runif(4000, 0, 100), ncol = 4)
data[5,4] <- NA
data[10,1] <- NA
data[15,c(1,4)] <- NA
data <- xts(data, order.by = rep(Sys.Date(), 1000))
i <- 0L
repeat {
message(i <- i+1L, " ", appendLF=i %% 100 == 1L)
fcase(as.logical(is.na(data[,1]) & is.na(data[,4])),
as.double(data[,1] + data[,4]),
as.logical(is.na(data[,1])),
as.double(data[,4]),
as.logical(is.na(data[,4])),
as.double(data[,1]),
as.logical(data[,1] > data[,4]),
as.double(data[,1] - data[,4]),
as.logical(data[,1] < data[,4]),
as.double(data[,4] - data[,1])
)
}I reckon it's just that having a @tdhock I think |
im getting a frequent, but intermittent issue with
fcasewith 1.16The offending code has not changed in weeks
The issue only started after upgrading to 1.16
The issues does not surface after reverting to 1.15.4
The issue is intermittent. it cannot reliably be reproduced with the same input
The problem code does not use
data.table, but usesfcaseandfcoalesceIm am trying to isolate the issue to produce an MRE, but due to the intermittent nature, i cant even reliably trap it in the debugger start investigating
The text was updated successfully, but these errors were encountered: