Stream write is slow and laggy

[mariusrak]

Support plan

• Which support plan is this issue covered by? Community
• Currently blocking your project/work? no
• Affecting a production system? no

Context

• Node.js version: v16.17.0
• Release Line of Formidable: Current
• Formidable exact version: 3.2.4
• Environment (node, browser, native, OS): node, browser
• Used with (popular names of modules): Koa, koa-body

What are you trying to achieve or the steps to reproduce?

I'm trying to upload files directly to oracle cloud object storage. I use their API with npm package and fileWriteStreamHandler feature of formidable. When I upload a file, the upload from broweser perspective is terribly slow and not responsive to changes.

On browser side I'm checking upload progress by event

xhr.upload.onprogress = e => {
        this._uploadedSize = parseFloat(e.loaded);
        this._changed();
};

But this event is not often called and oftentimes the upload seems like it's stuck. It gets stuck on some percentage, then after like a minute it jumps to different percentage and gets stuck again. Direct upload to oracle cloud from my computer is fast.

On server side I have a class which is basically adapter and this is the important code:

class OracleStorageDriver {
        // ...
        get formidable() {
                if (this.localTemp) {
                        return { uploadDir: this.tempDir };
                }
                const fileWriteStreamHandler = file => {
                        const object = tempDir + file.newFilename;
                        return this.createWriteStream(object);
                };
                return { fileWriteStreamHandler };
        }
        createWriteStream(path) {
                let uploadId;
                let partNum = 0;
                const partsToCommit = [];
                const { client } = this;

                const construct = done => {
                        const createMultipartUploadDetails = { object: path };
                        const requestCreate = this.cleanRequest({ createMultipartUploadDetails });
                        client.createMultipartUpload(requestCreate).then(response => {
                                uploadId = response.multipartUpload.uploadId;
                                done();
                        });
                };
                const write = (chunk, encoding, done) => {
                        // In debugger, breakpoint here is hitted oftentimes and quick
                        const uploadPartRequest = this.request(path, {
                                uploadId,
                                uploadPartNum: ++partNum,
                                contentLength: chunk.length,
                                uploadPartBody: chunk,
                        });
                        client.uploadPart(uploadPartRequest).then(response => {
                                partsToCommit.push({ partNum, etag: response.eTag });
                                done(); // This place is also often and quickly hit with breakpoint
                        });
                };
                const final = done => {
                        const commitMultipartUploadDetails = { partsToCommit };
                        const commitMultipartUploadRequest = this.request(path, {
                                uploadId,
                                commitMultipartUploadDetails,
                        });
                        client.commitMultipartUpload(commitMultipartUploadRequest).then(() => done());
                };
                const destroy = (err, done) => {
                        const abortMultipartUploadRequest = this.request(path, { uploadId });
                        client.abortMultipartUpload(abortMultipartUploadRequest).then(() => done());
                        throw err;
                };
                const stream = new Writable({ construct, write, destroy, final });
                return stream;
        }
        // ...
}

this.localTemp is variable from configuration, which switches between local storage of temporary files and uploading file directly from form to cloud storage. When I use local sotrage, everything works good and fast. With stream write, it's slow. But write and done() in write are fired often and fast.

What was the result you got?

Slow and laggy upload with fileWriteStreamHandler

What result did you expect?

Approximately same speed/responsivenes as uploading when using local temp storage or as uploading directly to oracle cloud.

[GrosSacASac]

The problem is in your write method: Are you making a http request on every write call ?

[mariusrak]

Yes, it is according to oracle documentation for multipart uploads.

[GrosSacASac]

Well, making a HTTP request on every chunk is extremely inefficient, and no wonder it is slow. Try to find a way to pipe the stream directly.

[mariusrak]

As I said, those http requests are fast. And it is according to oracle documentation. So that is not the problem. Also oracle API is AWS S3 compliant. So how do you imagine utilizing fileWriteStreamHandler if not the way I'm using it?

[GrosSacASac]

Also oracle API is AWS S3 compliant

Do it like the s3 example with a passtrough stream https://github.com/node-formidable/formidable/blob/master/examples/store-files-on-s3.js