How to stream response to file

[higoka]

I need to download large files to my disk. How could i stream the response directly to a file?

[metcoder95]

Hey! Assuming you're using undici.request in any way (Client or directly from undici) it will always give you back a ResponseData object. This object has the body property which is an instance of Redeable stream, and that can be used to pipe your response directly into a file in disk.

e.g.

const { pipeline } = require('node:stream')
const { createWriteStream } = require('node:fs')
const { request } = require('undici')
;(async () => {
  const file = createWriteStream('response.json')
  const response = await request('https://httpbin.org/get')

  pipeline(response.body, file, err => {
    if (err) console.log('pipe errored')
    else console.log('done')
  })
})()

If you need something more customizable due to the high size of files that enables you to do backpressure or to handle chunk by chunk of data, you can use Client#dispatch API.

Reference

• https://undici.nodejs.org/#/docs/api/Dispatcher?id=dispatcherrequestoptions-callback

[higoka]

Thx for the example i will try it. What is the difference between the undici.request and undici.stream? I tought i needed to use stream function.

Also how would i limit the amount of parallel request to not overload the server or get rate-limited? I thought of using a custom http agent and setting the maxSockets to something like 10. Would this work?

I have like 10k+ files i need to download. Im currently using the Promise.allSettled to start all downloads in parallel. Is there any better way of doing this?

[mcollina]

I have like 10k+ files i need to download. Im currently using the Promise.allSettled to start all downloads in parallel. Is there any better way of doing this?

You should likely pick a limit to concurrency to achieve the most performing output. You can do that either using p-limit, p-map, or fastq (with different range of flexibility.

You can also create a custom Pool for this operation with a limited number of connections. This will ensure only a few files are downloaded in parallel.

You'd need to do a few experiment to pick the right limit.

[higoka]

I have it working now. I just don't know how i can setup a Pool to limit the connections. Does someone have an example? This is my code so far:

import { request } from 'undici'
import { createWriteStream } from 'node:fs'
import { pipeline } from 'node:stream/promises'

async function download (file) {
  const response = await request(file.url)
  await pipeline(response.body, createWriteStream(file.destination))
}

// about 10k+ files
const files = [
  { url: 'https://example.com/file.txt', destination: 'files/file.txt' },
  { url: 'https://example.com/image.png', destination: 'files/img/image.png' },
  // ... many more files
]

await Promise.allSettled(
  files.map((file) => download(file))
)

console.log('done')

[metcoder95]

It should be slightly similar, you can try:

import { Pool } from 'undici'
import { createWriteStream } from 'node:fs'
import { pipeline } from 'node:stream/promises'

const pool = new Pool('https://example.com');

async function download (file) {
  const response = await pool.request({ path: file.path, method: 'GET' })
  await pipeline(response.body, createWriteStream(file.destination))
}

// about 10k+ files
const files = [
  { path: '/file.txt', destination: 'files/file.txt' },
  { path: '/image.png', destination: 'files/img/image.png' },
  // ... many more files
]

await Promise.allSettled(
  files.map((file) => download(file))
)

console.log('done')

Depending on your setup and your needs, it might be better to use one of the concurrency libraries mentioned by @mcollina.