Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File upload got stuck while uploading large text files #1235

Open
SamantaChinmoy opened this issue Oct 8, 2023 · 8 comments
Open

File upload got stuck while uploading large text files #1235

SamantaChinmoy opened this issue Oct 8, 2023 · 8 comments

Comments

@SamantaChinmoy
Copy link

Hi,

I am using multer to upload a zip file which contains large text files. Zip file size is 960 MB. There are total 10 text files is there inside zipped file, size of each text file is in the range of 500-2 gb. While I am uploading this zipped file using multer , The file upload got stuck and it is not processed further after 919 mb. But I have tested with other zipped file with size of 1.5 gb which contains many small size text file in the range of 1-2 mb, that time it is got successfully uploaded.

N.B: I am using disk based storage.

@almigiusz
Copy link

@SamantaChinmoy have you solved it? I have the same problem.

@aroldoGoulart-estuda
Copy link

Same problem here

@joeyguerra
Copy link

have ya'll tried the latest release candidate? npm install [email protected]? It adds a stream property on the req.file so you can pipe directly to disk. I would try that and see what you get.

@johnny77221
Copy link

I have same issue uploading a video file
It works well when I run the code on Mac
but the file does not receive completely on deployed Ubuntu

tried 2.0.0-rc.2 and this happens nodejs/node#38058
for 2.0.0 rc.3 and rc.4, I cannot even import mutler:
Uncaught Error Error [ERR_REQUIRE_ESM]: require() of ES Module /(project_path)/node_modules/multer/index.js from /(project_path)/index.js not supported.

but I am using node v21.1.0, is it necessary to downgrade to v15.3 as mentioned above?

@joeyguerra
Copy link

can you provide any code? test script? Technically, the ERR_REQUIRE_ESM error is because the consuming code is trying to require an ESM module.

@johnny77221
Copy link

johnny77221 commented Apr 30, 2024

I am building with nodejs v21
package.json:

"dependencies": {
    "body-parser": "^1.20.2",
    "cookie-parser": "^1.4.6",
    "dotenv": "^16.4.5",
    "express": "^4.19.1",
    "express-fileupload": "^1.5.0",
    "multer-utf8": "^1.4.5-lts.11",
    "socket.io": "^4.7.5",
    "uuid": "^9.0.1"
  }

code:

const express = require('express')
const multer = require('multer-utf8')
const storage = multer.diskStorage({
  destination: function (request, file, callback) {
      callback(null, './temp/')
  },
  filename: function (request, file, callback) {
      callback(null, file.originalname)
  }
})
const app = express()

...

app.post('/api/upload-video',multer({ storage: storage }).single('videoFile'), async (req, res) => {
 if (!req.file ) {
      return res.status(400).send('no file uploaded')
  }
  // process logics
  return res.status(201).send('{"message": "created"}')
})

@joeyguerra
Copy link

Have you looked into nodes request timeout as noted on https://github.com/mscdex/busboy

@johnny77221
Copy link

johnny77221 commented May 1, 2024

after lots of try and error
I found this way working for 8GB file

nginx sites-available conf:

server {
    server_name (my DN);
    location / {
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://localhost:3000;
        client_max_body_size 20480M;

        proxy_request_buffering off;
        client_body_timeout 1200s;   // 3200s for 15GB file
        proxy_send_timeout 3600;  // 9600 for 15GB file
        proxy_read_timeout 3600;  // 9600 for 15GB file
    }
}

node js app code:

const multer = require('multer-utf8')
const storage = multer.diskStorage({
  destination: function (request, file, callback) {
      callback(null, './temp/')
  },
  filename: function (request, file, callback) {
      callback(null, file.originalname)
  }
})
const app = express()
const http = require('http').createServer(app)
app.post('/api/upload-video',multer({ storage: storage, limits: { fileSize: 21474836480 /* 20GB */ } }).single('videoFile'), async (req, res) => {
  if (!req.file ) {
      return res.status(400).send('no file uploaded')
  }
  req.setTimeout(86400000)
  // process logics
  return res.status(201).send('{"message": "created"}')
})

so basically my issue is mainly related by nginx forwarding request,
the timeout and file size limit for the mutler also affects

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants