Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NaNs During Training #9

Open
ctralie opened this issue Oct 17, 2024 · 0 comments
Open

NaNs During Training #9

ctralie opened this issue Oct 17, 2024 · 0 comments

Comments

@ctralie
Copy link

ctralie commented Oct 17, 2024

Hello,
Thank you for making this very cool work publicly accessible!
I'm running into an issue where I get NaNs in training. I noticed you created a catch for that. I thought it might be coming from taking logs of very small numbers in the MSS loss, but that doesn't seem to be the issue. Since you put a catch there, I'm assuming you've dealt with this before, so do you have any idea what's causing it and how to avoid it?
Thanks,
Chris Tralie

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant