Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

implement parallel copy from NFS (or IFS) to HDFS #1509

Closed
akiezun opened this issue Feb 19, 2016 · 3 comments
Closed

implement parallel copy from NFS (or IFS) to HDFS #1509

akiezun opened this issue Feb 19, 2016 · 3 comments
Labels
performance Spark

Comments

@akiezun
Copy link
Contributor

akiezun commented Feb 19, 2016

A lot of data we have lives on NFS (or underlying IFS - Isilon FS). Copying files in and out is a bottleneck and a pain. This ticket for an implementation of a parallel copy of a BAM/CRAM file to HDFS (sharded or unsharded)
@pfxuan
Copy link

pfxuan commented Feb 25, 2016

Is NetApp-Hadoop-NFS-Connector useful for your problem? Also, the datasets pumped from NFS could be held at in-memory storage layer:
https://github.com/NetApp/NetApp-Hadoop-NFS-Connector
http://www.alluxio.org/

@akiezun akiezun self-assigned this Mar 17, 2016
@akiezun akiezun added this to the alpha-2 milestone Mar 17, 2016
@akiezun
Copy link
Contributor Author

akiezun commented Mar 22, 2016

unclear if this is needed given the cloud.

@akiezun akiezun removed this from the alpha-2 milestone Mar 22, 2016
@akiezun akiezun removed their assignment Mar 22, 2016
@droazen
Copy link
Contributor

droazen commented Mar 22, 2017

This was resolved in #2014

@droazen droazen closed this as completed Mar 22, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
performance Spark
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@droazen @akiezun @pfxuan