, 2 min read
Opening lots and lots of files under Linux
Suppose you want a program, or a process to be precise, to open 10,000 files simultaneously. For some reason, I thought that, by default, this would be possible, but it seems that Linux sets the the default limit to 1024 files on most distributions we checked.
First of all, check how many files your system allows you to open simulateneously:
# cat /proc/sys/fs/file-max
101066
On my system, as you can see, a process should be able to open 100,000 files simultaneously without a problem. If your number is much lower, you may need to do some extra work. Unfortunately, there are security settings above and beyond this number. To get around them, add the following line to “/etc/security/limits.conf”:
* - nofile 100000
Then, you need to log in again (fresh, not within X). To make sure it worked, type
#ulimit -n
100000
As you can see, it worked for me.
To make double sure it works, you might try to run the following C++ program:
#include <fstream>
#include <string>
#include <iostream>
#include <fstream>
#include <sstream>
#include <vector>
using namespace std;
int main() {
fstream lfs[10000];
for (int k = 0 ; k < 10000; ++k) {
stringstream strs;
strs << "stupidtest" << k;
lfs[k].open(strs.str().c_str(),ios::out);
assert(lfs[k].good());
if(lfs[k].good()) {
cout << "file created "<<strs.str()<< " ok!" << endl;
}
}
for (int k = 0 ; k < 10000; ++k) {
stringstream strs;
strs << "stupidtest" << k;
if(lfs[k].good()) {
cout << "file "<<strs.str()<< " still ok!" << endl;
}
lfs[k] << k;
if(lfs[k].good()) {
cout << "file "<<strs.str()<< " still ok
after write!" << endl;
}
}
for (int k = 0 ; k < 10000; ++k) {
lfs[k].close();
}
}