feat: new way of creating temporary files backed up by fs

[jimmywarting]

One thing I have been thinking of in node-fetch when we are parsing large multipart payloads with response.formData() is for the possibility to write dose temporary large file to the filesystem in order to not blow up the memory.

But writing them to a place on the disk would also require cleaning it up.
That's why I have now also introduced the new createTemporaryBlob and createTemporaryFile that writes whatever fsPromise.writeFile(data) supports

The purpose of this PR is:

Creating new temporary files to offload some data to the filesystem and cleaning when it's garbage collected

This is what had to change:

I exported two new features createTemporaryBlob and createTemporaryFile in ./from.js

This is what like reviewers to know:

• Removing / Cleaning up those temporary Blob also depends on the newish FinalizationRegistry. When a blob is GC and you no longer have any references to that blob anymore, then the file will automatically be removed.
• I have just learned that fsPromise.writeFile(stream) is more superior to stream.pipe(fs.createWritableStream(path)) as fsPromises. writeFile supports more different type of data. It works with string, ArrayBufferView and (async)Iterator so it supports both whatwg streams and node streams (from v15.14.0, v14.18.0)
  • so i'm going to stop using fs.createWritableStream in all my projects... + it can tell when it's finish with a promise.

---

• I updated ./CHANGELOG.md with a link to this PR or Issue ( TODO )
• I updated the README.md ( TODO )
• I Added unit test(s)

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (main@f077471). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff            @@
##             main      #137   +/-   ##
========================================
  Coverage        ?   100.00%           
========================================
  Files           ?         4           
  Lines           ?       511           
  Branches        ?        83           
========================================
  Hits            ?       511           
  Misses          ?         0           
  Partials        ?         0           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f077471...a644eab. Read the comment docs.

[jimmywarting]

import { createTemporaryBlob } from 'fetch-blob/from.js'


const res = await fetch(url)

// thinking `response.blob()` should basically do this
const iterable = res.body
let blob = await createTemporaryBlob(iterable, 'text/html') // will be written to the OS temporary directory

blob = undefined // will GC the blob and eventually remove the file from the temporary location on the disk

import { createTemporaryFile } from 'fetch-blob/from.js'

const fd = new FormData()

for await (const part of parseMultipart(payload)) {
  const file = await createTemporaryFile(part.iterable, part.fileName, { type: part.type })
  fd.append(part.fieldName, file)
}

return fd

[jimmywarting]

Oh btw, it's also possible to stop the writing if you pass a AbortSignal as an option

either via: createTemporaryFile(data, name, { signal, lastModified}) or createTemporaryBlob(data, type, signal)

[jimmywarting]

cc @KhafraDev @ronag @tunnckoCore who may all be interested in this.
with regards to this issues:

• the v3 plan? node-formidable/formidable#635
• Implement multipart parsing nodejs/undici#974

[tunnckoCore]

thanks for the ping.

I've spent last few hours digging around File, FormData, and the different parsers. Making the parser just an (async) iterator function seems awesome.

[gr2m]

elegant 👍🏼

[jimmywarting]

was wondering if createTemporaryBlob should match the signature of Blob's constructor. so you do: createTemporaryBlob(data, { type, signal })

Everything else in ./from.js had blobFrom|blobFromSync|fileFrom|fileFromSync(path, [mimetype])
So i initially try to match ☝️ this style and having type as 2nd argument

createTemporaryFile pretty much match the same as File's constructor

[tunnckoCore]

createTemporaryBlob(data, { type, signal }) makes sense and looks good, yep.

[tunnckoCore]

By the way, this createTemporaryBlob(data, { type, signal }) signature will also allow passing more options. Which will be passed from createTemporaryFile.

Thought it a bit deeper for my use-case, I think that allowing options.filename or something like that being a function would be good. Or it can be just a string. Thing is, I want to be able to pass generated filenames or the original filename. Well, not yet sure completely, but will be good addition.

[jimmywarting]

If you want to have a filename, then u should use createTemporaryFile instead that returns a File class rather than a Blob so i don't think you should be able to pass in a filename as an option

Also you should be able to create multiple files with the same name

[tunnckoCore]

Well, yea, my thinking was wrong. They are temporary files and it doesn't matter what the filename is. I actually implemented it without any changes to the branch.

The thing that I'll make formidable to do by default is to always create temp files, and later give that standard File to the user. If they want to save on disk there will be option that will just write permanent file to non temp user-chosen dir. Simple. Haha.

And i think we can actually introduce it in the v3 and v2, with a bit backward compat file props. Turns out we switch out off of file.type and file.name in v3.. and v2 been more File compat, haha.

All these recent things really lead to a more clear vision of what v4 can be.