impl Read for S3 Object

[avjewe]

I'd like to be able to read an S3 object like any other file. That is, I want to implement S3Reader like this :

struct S3Reader {
   ...GetObject...
}
impl Read for S3Reader {
   fn read(&mut self, buf: &mut [u8]) -> Result<usize> {
      ...
   }
}

Any hints? Everything I try hits a wall, but it seems like the first thing anyone would try.

[rcoh]

Since objects being downloaded from S3 are asynchronous, you would need to implement AsyncRead or similar and not Read.

Do you have any other information about your use case? If you want to download an entire file, you can use collect on the body field of get object. Full example.

If you wanted to stream the file incrementally, you can use the impl Stream<Result<Bytes, Err>> implementation on ByteStream: https://docs.rs/aws-sdk-s3/0.4.0/aws_sdk_s3/struct.ByteStream.html#getting-data-out-of-a-bytestream

[avjewe]

Thanks.
I don't want AsyncRead, I want a blocking Read, preceded by a blocking open of some sort.
I need to read incrementally, so I'll look into Stream.
I'd also like a hint about what, exactly, goes in the struct S3Reader.

[rcoh]

This is a little beyond the level of support we can provide right now. You'd probably need to use something like https://docs.rs/tokio/latest/tokio/runtime/struct.Runtime.html#method.block_on to convert the asynchronous response into a synchronous one but you need to be careful to avoid deadlock.

[evbo]

Here is the synchronous approach to reading the entirety of the bucket object into memory and printing it's contents to stdout:

aws-sdk-rust/examples/s3/src/bin/s3-helloworld.rs

Lines 88 to 90 in 75e7cf4

	let resp = client.get_object().bucket(bucket).key(key).send().await?;
	let data = resp.body.collect().await;
	println!("data: {:?}", data.unwrap().into_bytes());

Note: This is from an older commit that's no longer maintained that I'm aware of at the time of this comment

[jgmartin]

Second this... many compression libraries, for instance, expect to wrap an object that is Read or AsyncRead. ByteStream can be very unwieldy in many cases like this.

[sdlarsen]

I ended up using something like this, for decompressing and reading files line by line:

use async_compression::tokio::bufread::GzipDecoder;
use aws_sdk_s3::types::ByteStream;
use futures::TryStreamExt;
use tokio::io::{AsyncBufReadExt, BufReader};
use tokio_stream::StreamExt;
use tokio_util::io::StreamReader;

// Convert aws-smithy-http::bytestream::Error to std::io:Error
fn convert_smithy_error(err: aws_smithy_http::byte_stream::Error) -> std::io::Error {
    std::io::Error::new(std::io::ErrorKind::Other, err)
}

// Read line by line
let data = s3.download_object_stream(&bucket, Some(&pf), &key).await?;  // Function returning they ByteStream from get_object
let mut lines = BufReader::new(GzipDecoder::new(StreamReader::new(
    data.map_err(convert_smithy_error),
)))
.lines();

while let Some(line) = next_line().await? {
    println!("Line: {line}");
}
...

Your use-case may be different, but I hope this is helpful if anyone gets stuck on this

[jgmartin]

+1 for this solution, it's exactly where I ended up as well.

[Fuuzetsu]

For Rusoto, we made https://github.com/Fuuzetsu/zstd-seekable-s3 for use internally (i.e. buyer-be-ware and I suspect there are bugs as we sometimes see problems using this). The large part of the crate is providing Read and Sync interface over async rusoto API for S3.

While you can't just use it as-is, perhaps you could try to implement something similar for aws-sdk-rust to suit your use-case.

In practice/hindsight, I'd just recommend using async in your code instead of trying to sync-ify an async API.