You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
There are a number of places in the code where data can be side-band written to a distributed filesystem, e.g.: file scan input dumps, GPU core dumps, profiling data, etc. A lot of this code has to solve the problem of how to get the proper Hadoop configuration and credentials on the executor to access the filesystem to create the new file, and that code is often replicated across subsystems.
Describe the solution you'd like
There should be a common API in the executor plugin that subsystems on the executor can use to create a file on the distributed filesystem. The API should allow specifying a specific filename or generating a temporary name based on a filename template.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
There are a number of places in the code where data can be side-band written to a distributed filesystem, e.g.: file scan input dumps, GPU core dumps, profiling data, etc. A lot of this code has to solve the problem of how to get the proper Hadoop configuration and credentials on the executor to access the filesystem to create the new file, and that code is often replicated across subsystems.
Describe the solution you'd like
There should be a common API in the executor plugin that subsystems on the executor can use to create a file on the distributed filesystem. The API should allow specifying a specific filename or generating a temporary name based on a filename template.
The text was updated successfully, but these errors were encountered: