[Bug] CKAN crash on arch linux when updating (even with ulimit changed)

[edg-l]

Background

• Operating System: Arch Linux x86_64
• CKAN Version: v1.30.0
• KSP Version: 1.11.2.3077

Have you made any manual changes to your GameData folder (i.e., not via CKAN)? No

Problem

CKAN crashes while updating mods, even after setting the recommended ulimit values from the wiki.

Note: I tried updating mods with ckan console ui and it worked.

exception inside UnhandledException handler: (null) assembly:/usr/lib/mono/gac/Mono.Posix/4.0.0.0__0738eb9f132ed756/Mono.Posix.dll type:UnixIOException member:(null)

exception inside UnhandledException handler: (null) assembly:/usr/lib/mono/gac/Mono.Posix/4.0.0.0__0738eb9f132ed756/Mono.Posix.dll type:UnixIOException member:(null)

exception inside UnhandledException handler: (null) assembly:/usr/lib/mono/gac/Mono.Posix/4.0.0.0__0738eb9f132ed756/Mono.Posix.dll type:UnixIOException member:(null)

exception inside UnhandledException handler: (null) assembly:/usr/lib/mono/gac/Mono.Posix/4.0.0.0__0738eb9f132ed756/Mono.Posix.dll type:UnixIOException member:(null)

exception inside UnhandledException handler: (null) assembly:/usr/lib/mono/gac/Mono.Posix/4.0.0.0__0738eb9f132ed756/Mono.Posix.dll type:UnixIOException member:(null)

exception inside UnhandledException handler: (null) assembly:/usr/lib/mono/gac/Mono.Posix/4.0.0.0__0738eb9f132ed756/Mono.Posix.dll type:UnixIOException member:(null)

exception inside UnhandledException handler: (null) assembly:/usr/lib/mono/gac/Mono.Posix/4.0.0.0__0738eb9f132ed756/Mono.Posix.dll type:UnixIOException member:(null)

exception inside UnhandledException handler: (null) assembly:/usr/lib/mono/gac/Mono.Posix/4.0.0.0__0738eb9f132ed756/Mono.Posix.dll type:UnixIOException member:(null)

exception inside UnhandledException handler: (null) assembly:/usr/lib/mono/gac/Mono.Posix/4.0.0.0__0738eb9f132ed756/Mono.Posix.dll type:UnixIOException member:(null)

exception inside UnhandledException handler: (null) assembly:/usr/lib/mono/gac/Mono.Posix/4.0.0.0__0738eb9f132ed756/Mono.Posix.dll type:UnixIOException member:(null)

exception inside UnhandledException handler: (null) assembly:/usr/lib/mono/gac/Mono.Posix/4.0.0.0__0738eb9f132ed756/Mono.Posix.dll type:UnixIOException member:(null)

exception inside UnhandledException handler: (null) assembly:/usr/lib/mono/gac/Mono.Posix/4.0.0.0__0738eb9f132ed756/Mono.Posix.dll type:UnixIOException member:(null)

exception inside UnhandledException handler: (null) assembly:/usr/lib/mono/gac/Mono.Posix/4.0.0.0__0738eb9f132ed756/Mono.Posix.dll type:UnixIOException member:(null)

exception inside UnhandledException handler: (null) assembly:/usr/lib/mono/gac/Mono.Posix/4.0.0.0__0738eb9f132ed756/Mono.Posix.dll type:UnixIOException member:(null)

exception inside UnhandledException handler: (null) assembly:/usr/lib/mono/gac/Mono.Posix/4.0.0.0__0738eb9f132ed756/Mono.Posix.dll type:UnixIOException member:(null)

[ERROR] FATAL UNHANDLED EXCEPTION: Mono.Unix.UnixIOException: Resource temporarily unavailable [EWOULDBLOCK].
  at Mono.Unix.UnixMarshal.ThrowExceptionForLastError () [0x00005] in <7bcd6d815ebc477ab0521e0361c44f6c>:0
  at Mono.Unix.UnixStream.Write (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x00055] in <7bcd6d815ebc477ab0521e0361c44f6c>:0
  at System.Windows.Forms.XplatUIX11.WakeupMain () [0x00000] in <4b6c441381804088ab0fff508a3fbabf>:0
  at System.Windows.Forms.XplatUIX11.SendAsyncMethod (System.Windows.Forms.AsyncMethodData method) [0x00080] in <4b6c441381804088ab0fff508a3fbabf>:0
  at System.Windows.Forms.XplatUI.SendAsyncMethod (System.Windows.Forms.AsyncMethodData data) [0x00000] in <4b6c441381804088ab0fff508a3fbabf>:0
  at System.Windows.Forms.Control.BeginInvokeInternal (System.Delegate method, System.Object[] args, System.Windows.Forms.Control control) [0x0003f] in <4b6c441381804088ab0fff508a3fbabf>:0
  at System.Windows.Forms.Control.Invoke (System.Delegate method, System.Object[] args) [0x00017] in <4b6c441381804088ab0fff508a3fbabf>:0
  at (wrapper remoting-invoke-with-check) System.Windows.Forms.Control.Invoke(System.Delegate,object[])
  at System.Windows.Forms.Control.Invoke (System.Delegate method) [0x0001d] in <4b6c441381804088ab0fff508a3fbabf>:0
  at (wrapper remoting-invoke-with-check) System.Windows.Forms.Control.Invoke(System.Delegate)
  at CKAN.Util.Invoke[T] (T obj, System.Action action) [0x0000d] in <152b683e73a7426a9ab8fa7c1d7980ff>:0
  at CKAN.Wait.set_ProgressValue (System.Int32 value) [0x00014] in <152b683e73a7426a9ab8fa7c1d7980ff>:0
  at (wrapper remoting-invoke-with-check) CKAN.Wait.set_ProgressValue(int)
  at CKAN.Main.SetProgress (System.Int32 progress) [0x00014] in <152b683e73a7426a9ab8fa7c1d7980ff>:0
  at (wrapper remoting-invoke-with-check) CKAN.Main.SetProgress(int)
  at CKAN.GUIUser.RaiseProgress (System.String message, System.Int32 percent) [0x0001c] in <152b683e73a7426a9ab8fa7c1d7980ff>:0
  at CKAN.NetAsyncDownloader.FileProgressReport (System.Int32 index, System.Int32 percent, System.Int64 bytesDownloaded, System.Int64 bytesToDownload) [0x0019e] in <152b683e73a7426a9ab8fa7c1d7980ff>:0
  at CKAN.NetAsyncDownloader+<>c__DisplayClass16_0.<DownloadModule>b__0 (System.Object sender, System.Net.DownloadProgressChangedEventArgs args) [0x0001e] in <152b683e73a7426a9ab8fa7c1d7980ff>:0
  at CKAN.NetAsyncDownloader+NetAsyncDownloaderDownloadPart.<ResetAgent>b__20_0 (System.Object sender, System.Net.DownloadProgressChangedEventArgs args) [0x00008] in <152b683e73a7426a9ab8fa7c1d7980ff>:0
  at System.Net.WebClient.OnDownloadProgressChanged (System.Net.DownloadProgressChangedEventArgs e) [0x0000a] in <1af2426e91d1474d92e5d471c4ca8f95>:0
  at System.Net.WebClient.<StartAsyncOperation>b__78_9 (System.Object arg) [0x00000] in <1af2426e91d1474d92e5d471c4ca8f95>:0
  at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context (System.Object state) [0x00007] in <efe941bb62534dc3a62ceb1a818964a0>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00071] in <efe941bb62534dc3a62ceb1a818964a0>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <efe941bb62534dc3a62ceb1a818964a0>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00021] in <efe941bb62534dc3a62ceb1a818964a0>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <efe941bb62534dc3a62ceb1a818964a0>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <efe941bb62534dc3a62ceb1a818964a0>:0
[ERROR] FATAL UNHANDLED EXCEPTION: Mono.Unix.UnixIOException: Resource temporarily unavailable [EWOULDBLOCK].
  at Mono.Unix.UnixMarshal.ThrowExceptionForLastError () [0x00005] in <7bcd6d815ebc477ab0521e0361c44f6c>:0
  at Mono.Unix.UnixStream.Write (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x00055] in <7bcd6d815ebc477ab0521e0361c44f6c>:0
  at System.Windows.Forms.XplatUIX11.WakeupMain () [0x00000] in <4b6c441381804088ab0fff508a3fbabf>:0
  at System.Windows.Forms.XplatUIX11.SendAsyncMethod (System.Windows.Forms.AsyncMethodData method) [0x00080] in <4b6c441381804088ab0fff508a3fbabf>:0
  at System.Windows.Forms.XplatUI.SendAsyncMethod (System.Windows.Forms.AsyncMethodData data) [0x00000] in <4b6c441381804088ab0fff508a3fbabf>:0
  at System.Windows.Forms.Control.BeginInvokeInternal (System.Delegate method, System.Object[] args, System.Windows.Forms.Control control) [0x0003f] in <4b6c441381804088ab0fff508a3fbabf>:0
  at System.Windows.Forms.Control.Invoke (System.Delegate method, System.Object[] args) [0x00017] in <4b6c441381804088ab0fff508a3fbabf>:0
  at (wrapper remoting-invoke-with-check) System.Windows.Forms.Control.Invoke(System.Delegate,object[])
  at System.Windows.Forms.Control.Invoke (System.Delegate method) [0x0001d] in <4b6c441381804088ab0fff508a3fbabf>:0
  at (wrapper remoting-invoke-with-check) System.Windows.Forms.Control.Invoke(System.Delegate)
  at CKAN.Util.Invoke[T] (T obj, System.Action action) [0x0000d] in <152b683e73a7426a9ab8fa7c1d7980ff>:0
  at CKAN.Wait.SetDescription (System.String message) [0x00014] in <152b683e73a7426a9ab8fa7c1d7980ff>:0
  at (wrapper remoting-invoke-with-check) CKAN.Wait.SetDescription(string)
  at CKAN.GUIUser.RaiseProgress (System.String message, System.Int32 percent) [0x00017] in <152b683e73a7426a9ab8fa7c1d7980ff>:0
  at CKAN.NetAsyncDownloader.FileProgressReport (System.Int32 index, System.Int32 percent, System.Int64 bytesDownloaded, System.Int64 bytesToDownload) [0x0019e] in <152b683e73a7426a9ab8fa7c1d7980ff>:0
  at CKAN.NetAsyncDownloader+<>c__DisplayClass16_0.<DownloadModule>b__0 (System.Object sender, System.Net.DownloadProgressChangedEventArgs args) [0x0001e] in <152b683e73a7426a9ab8fa7c1d7980ff>:0
  at CKAN.NetAsyncDownloader+NetAsyncDownloaderDownloadPart.<ResetAgent>b__20_0 (System.Object sender, System.Net.DownloadProgressChangedEventArgs args) [0x00008] in <152b683e73a7426a9ab8fa7c1d7980ff>:0
  at System.Net.WebClient.OnDownloadProgressChanged (System.Net.DownloadProgressChangedEventArgs e) [0x0000a] in <1af2426e91d1474d92e5d471c4ca8f95>:0
  at System.Net.WebClient.<StartAsyncOperation>b__78_9 (System.Object arg) [0x00000] in <1af2426e91d1474d92e5d471c4ca8f95>:0
  at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context (System.Object state) [0x00007] in <efe941bb62534dc3a62ceb1a818964a0>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00071] in <efe941bb62534dc3a62ceb1a818964a0>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <efe941bb62534dc3a62ceb1a818964a0>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00021] in <efe941bb62534dc3a62ceb1a818964a0>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <efe941bb62534dc3a62ceb1a818964a0>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <efe941bb62534dc3a62ceb1a818964a0>:0
[ERROR] FATAL UNHANDLED EXCEPTION: Mono.Unix.UnixIOException: Resource temporarily unavailable [EWOULDBLOCK].
  at Mono.Unix.UnixMarshal.ThrowExceptionForLastError () [0x00005] in <7bcd6d815ebc477ab0521e0361c44f6c>:0
  at Mono.Unix.UnixStream.Write (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x00055] in <7bcd6d815ebc477ab0521e0361c44f6c>:0
  at System.Windows.Forms.XplatUIX11.WakeupMain () [0x00000] in <4b6c441381804088ab0fff508a3fbabf>:0
  at System.Windows.Forms.XplatUIX11.SendAsyncMethod (System.Windows.Forms.AsyncMethodData method) [0x00080] in <4b6c441381804088ab0fff508a3fbabf>:0
  at System.Windows.Forms.XplatUI.SendAsyncMethod (System.Windows.Forms.AsyncMethodData data) [0x00000] in <4b6c441381804088ab0fff508a3fbabf>:0
  at System.Windows.Forms.Control.BeginInvokeInternal (System.Delegate method, System.Object[] args, System.Windows.Forms.Control control) [0x0003f] in <4b6c441381804088ab0fff508a3fbabf>:0
  at System.Windows.Forms.Control.Invoke (System.Delegate method, System.Object[] args) [0x00017] in <4b6c441381804088ab0fff508a3fbabf>:0
  at (wrapper remoting-invoke-with-check) System.Windows.Forms.Control.Invoke(System.Delegate,object[])
  at System.Windows.Forms.Control.Invoke (System.Delegate method) [0x0001d] in <4b6c441381804088ab0fff508a3fbabf>:0
  at (wrapper remoting-invoke-with-check) System.Windows.Forms.Control.Invoke(System.Delegate)
  at CKAN.Util.Invoke[T] (T obj, System.Action action) [0x0000d] in <152b683e73a7426a9ab8fa7c1d7980ff>:0
  at CKAN.Wait.SetDescription (System.String message) [0x00014] in <152b683e73a7426a9ab8fa7c1d7980ff>:0
  at (wrapper remoting-invoke-with-check) CKAN.Wait.SetDescription(string)
  at CKAN.GUIUser.RaiseProgress (System.String message, System.Int32 percent) [0x00017] in <152b683e73a7426a9ab8fa7c1d7980ff>:0
  at CKAN.NetAsyncDownloader.FileProgressReport (System.Int32 index, System.Int32 percent, System.Int64 bytesDownloaded, System.Int64 bytesToDownload) [0x0019e] in <152b683e73a7426a9ab8fa7c1d7980ff>:0
  at CKAN.NetAsyncDownlo

[DasSkelett]

even after setting the recommended ulimit values from the wiki.

Did you start CKAN from terminal in the same shell session after setting the ulimit values?
Copying from #3343:

but it only works in current bash session (you need to launch ckan from terminal after adjusting limits).

[edg-l]

yes

[DasSkelett]

Unfortunately we can't really help you there. This seems to be an OS (or maybe Mono) bug, not a CKAN bug.
It's basically the exact same stack trace from #3343. Maybe the ulimit needs raising even further?

Maybe @BlazingPL has some more ideas on how to fight this.

[edg-l]

I think this is most likely a mono bug, or i would even say a app level bug, since this is the first time i had to modify my ulimit settings, this is most likely a CKAN bug where it opens lot of file descriptors for some reason...

[DasSkelett]

If you look at the stack trace, this is happening far into the Mono code, way outside of CKAN's control:
Everything starting with CKAN. is code written by us, lines beginning with System. or Mono. are mono code.
This also does not happen while opening a file, but while switching active thread using Invoke(), which isn't an unusual thing to do (it's required to update GUI controls from a background thread).

If CKAN would open an unreasonable amount of files, we should see a similar exception on other OSes as well. But so far, this has only ever happened to Arch users.
So this is caused by some weird interaction between Mono and Arch, on which side the bug is needs to be answered by someone with a better understanding of this.

You could try opening a issue in the Mono repository, they know the innerworkings better than us: https://github.com/mono/mono/issues
Although their support on WinForms issues is a bit lacking, unfortunately :/

[Blazingi]

To be honest I've given up on playing modded ksp on Linux, as with modlist of ~100 mods there were enormous memory leaks, and some mods complained about things, whereas there was no problem on windows.
So I'm unable to help right now, and my arch needs a long due overhaul before I get back to gaming on it.

[DasSkelett]

I'm closing this now since I really can't see how this could be caused or solved from our side.
If someone is able to track this down further, or is even able to find a fix, you are welcome to leave a comment here and/or extend the Arch installation wiki page.

[jasnyj]

This is an upstream bug presents in mono version 6.12.0.86 and upward. It's neither CKAN's nor Arch Linux' fault. I made a pull request to mono that fix this bug but it's taking time to be reviewed. Setting ulimit to weird values is unfortunately useless, it's not going to fix the bug in any way. One needs to use an older mono version or patch the current version with the pull request's patch.

See my comment here and the mono's pull request for more details.

I'll probably make a pull request to the wiki to explain everything and what to do in the mean time.

[RonAndDumbleTheCats]

It works with the consoleui flag though. mabye it has something to do with the graphics api?

[HebaruSan]

It works with the consoleui flag though. mabye it has something to do with the graphics api?

Indeed, @RonAndDumbleTheCats, I think at some point we said that it might be due to having too many graphical updates happening at once while downloading. We made some changes to reduce that recently in #3659, so you can try the latest dev build to see if it helps you:

https://ksp-ckan.s3-us-west-2.amazonaws.com/ckan.exe