Node and Error: EMFILE, too many open files

Using the graceful-fs module by Isaac Schlueter (node.js maintainer) is probably the most appropriate solution. It does incremental back-off if EMFILE is encountered. It can be used as a drop-in replacement for the built-in fs module.

I am not sure whether this will help anyone, I started working on a big project with lot of dependencies which threw me the same error. My colleague suggested me to install watchman using brew and that fixed this problem for me.

brew updatebrew install watchman

Edit on 26 June 2019:Github link to watchman

I did all the stuff above mentioned for the same problem but nothing worked. I tried below it worked 100%. Simple config changes.

Option 1: Set limit (It won't work most of the time)

user@ubuntu:~$ ulimit -n 65535

Check the current limit

user@ubuntu:~$ ulimit -n1024

Option 2: Increase the available limit to e.g. 65535

user@ubuntu:~$ sudo nano /etc/sysctl.conf

Add the following line to it

fs.file-max = 65535

Run this to refresh with new config

user@ubuntu:~$ sudo sysctl -p

Edit the following file

user@ubuntu:~$ sudo vim /etc/security/limits.conf

Add the following lines to it

root soft nproc 65535 root hard nproc 65535 root soft nofile 65535 root hard nofile 65535

Edit the following file

user@ubuntu:~$ sudo vim /etc/pam.d/common-session

Add this line to it

session required pam_limits.so

Logout and login and try the following command

user@ubuntu:~$ ulimit -n65535

Option 3: Just add this line

DefaultLimitNOFILE=65535

to /etc/systemd/system.conf and /etc/systemd/user.conf

I ran into this problem today, and finding no good solutions for it, I created a module to address it. I was inspired by @fbartho's snippet, but wanted to avoid overwriting the fs module.

The module I wrote is Filequeue, and you use it just like fs:

var Filequeue = require('filequeue');var fq = new Filequeue(200); // max number of files to open at oncefq.readdir('/Users/xaver/Downloads/xaver/xxx/xxx/', function(err, files) {if(err) {throw err;}files.forEach(function(file) {fq.readFile('/Users/xaver/Downloads/xaver/xxx/xxx/' + file, function(err, data) {// do something here}});});

You're reading too many files. Node reads files asynchronously, it'll be reading all files at once. So you're probably reading the 10240 limit.

See if this works:

var fs = require('fs')var events = require('events')var util = require('util')var path = require('path')var FsPool = module.exports = function(dir) {events.EventEmitter.call(this)this.dir = dir;this.files = [];this.active = [];this.threads = 1;this.on('run', this.runQuta.bind(this))};// So will act like an event emitterutil.inherits(FsPool, events.EventEmitter);FsPool.prototype.runQuta = function() {if(this.files.length === 0 && this.active.length === 0) {return this.emit('done');}if(this.active.length < this.threads) {var name = this.files.shift()this.active.push(name)var fileName = path.join(this.dir, name);var self = this;fs.stat(fileName, function(err, stats) {if(err)throw err;if(stats.isFile()) {fs.readFile(fileName, function(err, data) {if(err)throw err;self.active.splice(self.active.indexOf(name), 1)self.emit('file', name, data);self.emit('run');});} else {self.active.splice(self.active.indexOf(name), 1)self.emit('dir', name);self.emit('run');}});}return this};FsPool.prototype.init = function() {var dir = this.dir;var self = this;fs.readdir(dir, function(err, files) {if(err)throw err;self.files = filesself.emit('run');})return this};var fsPool = new FsPool(__dirname)fsPool.on('file', function(fileName, fileData) {console.log('file name: ' + fileName)console.log('file data: ', fileData.toString('utf8'))})fsPool.on('dir', function(dirName) {console.log('dir name: ' + dirName)})fsPool.on('done', function() {console.log('done')});fsPool.init()

Like all of us, you are another victim of asynchronous I/O. With asynchronous calls, if you loop around a lot of files, Node.js will start to open a file descriptor for each file to read and then will wait for action until you close it.

File descriptor remains open until resource is available on your server to read it. Even if your files are small and reading or updating is fast, it takes some time, but in the same time your loop don't stop to open new files descriptor. So if you have too many files, the limit will be soon reached and you get a beautiful EMFILE.

There is one solution, creating a queue to avoid this effect.

Thanks to people who wrote Async, there is a very useful function for that. There is a method called Async.queue, you create a new queue with a limit and then add filenames to the queue.

Note: If you have to open many files, it would be a good idea to store which files are currently open and don't reopen them infinitely.

const fs = require('fs')const async = require("async")var q = async.queue(function(task, callback) {console.log(task.filename);fs.readFile(task.filename,"utf-8",function (err, data_read) {callback(err,task.filename,data_read);});}, 4);var files = [1,2,3,4,5,6,7,8,9,10]for (var file in files) {q.push({filename:file+".txt"}, function (err,filename,res) {console.log(filename + " read");});}

You can see that each file is added to the queue (console.log filename), but only when the current queue is under the limit you set previously.

async.queue get information about availability of the queue through a callback, this callback is called only when data file is read and any action you have to do is achieved. (see fileRead method)

So you cannot be overwhelmed by files descriptor.

> node ./queue.js0.txt1.txt2.txt0.txt read3.txt3.txt read4.txt2.txt read5.txt4.txt read6.txt5.txt read7.txt1.txt read (biggest file than other)8.txt6.txt read9.txt7.txt read8.txt read9.txt read

I just finished writing a little snippet of code to solve this problem myself, all of the other solutions appear way too heavyweight and require you to change your program structure.

This solution just stalls any fs.readFile or fs.writeFile calls so that there are no more than a set number in flight at any given time.

// Queuing reads and writes, so your nodejs script doesn't overwhelm system limits catastrophicallyglobal.maxFilesInFlight = 100; // Set this value to some number safeish for your systemvar origRead = fs.readFile;var origWrite = fs.writeFile;var activeCount = 0;var pending = [];var wrapCallback = function(cb){return function(){activeCount--;cb.apply(this,Array.prototype.slice.call(arguments));if (activeCount < global.maxFilesInFlight && pending.length){console.log("Processing Pending read/write");pending.shift()();}};};fs.readFile = function(){var args = Array.prototype.slice.call(arguments);if (activeCount < global.maxFilesInFlight){if (args[1] instanceof Function){args[1] = wrapCallback(args[1]);} else if (args[2] instanceof Function) {args[2] = wrapCallback(args[2]);}activeCount++;origRead.apply(fs,args);} else {console.log("Delaying read:",args[0]);pending.push(function(){fs.readFile.apply(fs,args);});}};fs.writeFile = function(){var args = Array.prototype.slice.call(arguments);if (activeCount < global.maxFilesInFlight){if (args[1] instanceof Function){args[1] = wrapCallback(args[1]);} else if (args[2] instanceof Function) {args[2] = wrapCallback(args[2]);}activeCount++;origWrite.apply(fs,args);} else {console.log("Delaying write:",args[0]);pending.push(function(){fs.writeFile.apply(fs,args);});}};

I solved this by updating watchman

 brew install watchman

With bagpipe, you just need change

FS.readFile(filename, onRealRead);

=>

var bagpipe = new Bagpipe(10);bagpipe.push(FS.readFile, filename, onRealRead))

The bagpipe help you limit the parallel. more details: https://github.com/JacksonTian/bagpipe

Had the same problem when running the nodemon command so i reduced the name of files open in sublime text and the error dissappeared.

cwait is a general solution for limiting concurrent executions of any functions that return promises.

In your case the code could be something like:

var Promise = require('bluebird');var cwait = require('cwait');// Allow max. 10 concurrent file reads.var queue = new cwait.TaskQueue(Promise, 10);var read = queue.wrap(Promise.promisify(batchingReadFile));Promise.map(files, function(filename) {console.log(filename);return(read(filename));})

Building on @blak3r's answer, here's a bit of shorthand I use in case it helps other diagnose:

If you're trying to debug a Node.js script that is running out of file descriptors here's a line to give you the output of lsof used by the node process in question:

openFiles = child_process.execSync(`lsof -p ${process.pid}`);

This will synchronously run lsof filtered by the current running Node.js process and return the results via buffer.

Then use console.log(openFiles.toString()) to convert the buffer to a string and log the results.

For nodemon users:Just use the --ignore flag to solve the problem.

Example:

nodemon app.js --ignore node_modules/ --ignore data/

Use the latest fs-extra.

I had that problem on Ubuntu (16 and 18) with plenty of file/socket-descriptors space (count with lsof |wc -l). Used fs-extra version 8.1.0. After the update to 9.0.0 the "Error: EMFILE, too many open files" vanished.

I've experienced diverse problems on diverse OS' with node handling filesystems. Filesystems are obviously not trivial.

I did installing watchman, changing limit etc. and it didn't work in Gulp.

Restarting iterm2 actually helped though.

For anyone that might still be looking for solutions, using async-await worked fine for me:

fs.readdir(<directory path></directory>, async (err, filenames) => {if (err) {console.log(err);}try {for (let filename of filenames) {const fileContent = await new Promise((resolve, reject) => {fs.readFile(<dirctory path + filename>, 'utf-8', (err, content) => {if (err) {reject(err);}resolve(content);});});... // do things with fileContent}} catch (err) {console.log(err);}});

Here's my two cents: Considering a CSV file is just lines of text I've streamed the data (strings) to avoid this problem.

Easiest solution for me that worked in my usecase.

It can be used with graceful fs or standard fs. Just note that there won't be headers in the file when creating.

// import graceful-fs or normal fsconst fs = require("graceful-fs"); // or use: const fs = require("fs") // Create output file and set it up to receive streamed data// Flag is to say "append" so that data can be recursively added to the same file let fakeCSV = fs.createWriteStream("./output/document.csv", {flags: "a",});

and the data that needs to be streamed to the file i've done like this

// create custom streamer that can be invoked when neededconst customStreamer = (dataToWrite) => {fakeCSV.write(dataToWrite + "\n");};

Note that the dataToWrite is simply a string with a custom seperator like ";" or ",".i.e.

const dataToWrite = "batman" + ";" + "superman"customStreamer(dataToWrite);

This writes "batman;superman" to the file.

• Note that there's no error catching or whatsoever in this example.
• Docs: https://nodejs.org/api/fs.html#fs_fs_createwritestream_path_options

This will probably fix your problem if you're struggling to deploy a React solution that was created with the Visual Studio template (and has a web.config). In Azure Release Pipelines, when selecting the template, use:

Azure App Service deployment

Instead of:

Deploy a Node.js app to Azure App Service

It worked for me!

There's another possibility that hasn't been considered or discussed in any of the answers so far: symbolic link cycles.

Node's recursive filesystem watcher does not appear to detect and handle cycles of symlinks. So you can easily trigger this error with an arbitrarily high nfiles ulimit by simply running:

mkdir amkdir a/bcd a/b ln -s .. c

GNU find will notice the symlink cycle and abort:

$ find a -followaa/bfind: File system loop detected; ‘a/b/c’ is part of the same file system loop as ‘a’.

but node won't. If you set up a watch on the tree, it'll spew a EMFILE, too many open files error.

Amongst other things this can happen in node_modules where there's a containment relationship:

parent/package.jsonchild/package.json

which is how I encountered it in a project I was trying to build.

Note that you don't necessarily need to overcomplicate this issue, trying again works just fine.

import { promises as fs } from "fs";const filepaths = [];const errors = [];function process_file(content: string) {// logic here}await Promise.all(filepaths.map(function read_each(filepath) {return fs.readFile(filepath, "utf8").then(process_file).catch(function (error) {if (error.code === "EMFILE") return read_each(filepath);else errors.push({ file: filepath, error });});}),);

What you did is almost right:

sysctl -w kern.maxfiles=20480

On my macOS, the default is 491520 where what you were setting is actually lower value than of my system default, I just set it to 999999 and it worked perfectly. No more of that error.

Edit: I forgot to mention to restart afterwards.

Hope this helps.

I had this issue, and i solved it by running npm update and it worked.

In some cases you may need to remove node_modules rm -rf node_modules/

On Windows, there is seems that no the ulimit command to increase the number of open files. In graceful-fs, it maintains a queue to run I/O operations, eg: read/write file.

However, fs.readFile, fs.writeFile are based on fs.open, so you will need open/close files manually to solve this error.

import fs from 'fs/promises';const fd = await fs.open('path-to-file', 'r');await fd.readFile('utf-8'); // <== read through file handleawait fd.close(); // <== manually close it

This may happen after changing the Node versionERR emfile too many open files

• Restart the computer
• brew install watchman

It should be absolutely fixed the issue

first update your version of expo using expo update and then run yarn / npm install. This solved the issue for me!