Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make tfb_fpc work with (small amounts of) missingness #81

Merged
merged 7 commits into from
Mar 13, 2024
Merged

Conversation

fabian-s
Copy link
Contributor

@fabian-s fabian-s commented Mar 6, 2024

pve <- .95
y <- tf_rgp(50, arg = 51L)
y_pc <- tfb_fpc(y, pve = pve)
y_mis <- y |> tf_sparsify(dropout = .05)
y_pc_sparse <- tfb_fpc(y_mis, pve = pve)
## Using softImpute SVD on 5.1% missing data
## Warning message:
## High <pve> with many missings likely to yield bad FPC estimates. 
y_pc_sparse_impute <- y_mis |>
  tf_interpolate(arg = tf_arg(y), evaluator = tf_approx_fill_extend) |>
  tfb_fpc(pve = pve)
y_pc_rebase <- tf_rebase(y_mis, y_pc)
Warning messages:
## 1: In tf_rebase.tfd.tfb_fpc(y_mis, y_pc) : 
## 2: 6 evaluations were NA, returning irregular tfd. 


layout(t(1:4))
plot(y[1:10], main = "full data")
lines(y_pc[1:10], col = 2, lty = 2)
plot(y[1:10], main = "5% missing")
lines(y_pc_sparse[1:10], col = 2, lty = 2)
plot(y[1:10], main = "from interpolated\n missings")
lines(y_pc_sparse_impute[1:10], col = 2, lty = 2)
plot(y[1:10], main = "scores from \nsparse data")
lines(y_pc_rebase[1:10], col = 2, lty = 2)

image

see docs for fpc_wsvd for details.
seems useful for applications, probably only works well in cases where simply replacing missings with interpolated values before FPCA would work about as well.
but at least this means one can now simply do cca_pc <- tfb_fpc(tidyfun::dti_df$cca), e.g., without worrying about that .5% of missing data...

@jeff-goldsmith @m-muecke @sebffischer
if you have time, could you read the docs / try this out on some data and LMK what's missing/broken?

somewhat related to #10 as well - can now do tfb_fpc for irregular data, in principle

@fabian-s fabian-s merged commit 188b9a9 into dev Mar 13, 2024
2 checks passed
@fabian-s fabian-s deleted the wsvd_na branch March 13, 2024 16:59
@fabian-s fabian-s restored the wsvd_na branch May 22, 2024 10:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant