Everything about inotify in Linux and MacOS

Sang

Aug 7, 2020 • 4 min read

Everything about inotify in Linux and macOS.

To check the current config:

$ sysctl fs.inotify
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 65536

$ cat /proc/sys/fs/inotify/max_queued_events
16384
$ cat /proc/sys/fs/inotify/max_user_instances
128
$ cat /proc/sys/fs/inotify/max_user_watches
65536

Explains:
+ max_user_instances is the maximum number of watch instances (= number of root dirs for watching).
+ max_user_watches is the maximum number of dirs across all watch instances.
+ max_queued_events the maximum number of events in the kernel queue. Usually, you should modify the 2 other values and keep this value as the default.

On macOS

$ sudo sysctl kern | grep maxfile
kern.maxfiles: 49152
kern.maxfilesperproc: 24576

To modify the configuration permanently. sudo vim /etc/sysctl.conf, modify these following lines or add new if they do not exist

fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 16384

Then run sudo sysctl -p to reload the changes.

On macOS, in the file path /etc/sysct.conf:

kern.maxfiles=49152
kern.maxfilesperproc=24576

The configuration will be used from the next reboot. To modify the setting in the current boot, see next.

To modify the configuration online which does not affect in the next restart. Use these command lines

sudo sysctl -n -w fs.inotify.max_queued_events=16384
sudo sysctl -n -w fs.inotify.max_user_instances=128
sudo sysctl -n -w fs.inotify.max_user_watches=16384

On macOS:

sudo sysctl -w kern.maxfiles=49152
sudo sysctl -w kern.maxfilesperproc=24576

Find the number of files being opened

$ lsof | grep inotify | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/121/gvfs
      Output information may be incomplete.
4378

To check if the inotify resource has already been limited. Use tail -f.
Because tail use inotify internally to watch for file update, the command will fail if the inotify resource is drained.
The error (if happens) would be tail: inotify cannot be used, reverting to polling: Too many open files.
For other programs, it usually shows this error instead

Error: ENOSPC: System limit for number of file watchers reached, watch ....

In IntelliJ editors, it shows

The current inotify(7) watch limit is too low

External file changes sync may be slow

Check which processes are consuming inotify resources

ps -p $(find /proc/*/fd/* -type l -lname 'anon_inode:inotify' -print 2> /dev/null | sed -e 's/^\/proc\/\([0-9]*\)\/.*/\1/')

List of processes id with the number of inotify instances they are consuming, sorted by the number of inotify instances being consumed.

$ for foo in /proc/*/fd/*; do readlink -f $foo; done | grep inotify | sort | uniq -c | sort -nr

      6 /proc/2183/fd/anon_inode:inotify
      2 /proc/6346/fd/anon_inode:inotify
      2 /proc/3077/fd/anon_inode:inotify
      2 /proc/2190/fd/anon_inode:inotify
      1 /proc/76491/fd/anon_inode:inotify
      1 /proc/6404/fd/anon_inode:inotify
      1 /proc/59641/fd/anon_inode:inotify
      1 /proc/51816/fd/anon_inode:inotify
      1 /proc/40172/fd/anon_inode:inotify
      1 /proc/3844/fd/anon_inode:inotify
      1 /proc/37830/fd/anon_inode:inotify
      1 /proc/3429/fd/anon_inode:inotify
      1 /proc/3323/fd/anon_inode:inotify
      1 /proc/3120/fd/anon_inode:inotify
      1 /proc/3060/fd/anon_inode:inotify
      1 /proc/2837/fd/anon_inode:inotify
      1 /proc/2797/fd/anon_inode:inotify
      1 /proc/2748/fd/anon_inode:inotify
      1 /proc/2729/fd/anon_inode:inotify
      1 /proc/2617/fd/anon_inode:inotify
      1 /proc/2580/fd/anon_inode:inotify
      1 /proc/2574/fd/anon_inode:inotify
      1 /proc/2561/fd/anon_inode:inotify
      1 /proc/2555/fd/anon_inode:inotify
      1 /proc/2550/fd/anon_inode:inotify
      1 /proc/2547/fd/anon_inode:inotify
      1 /proc/2539/fd/anon_inode:inotify
      1 /proc/2515/fd/anon_inode:inotify
      1 /proc/2477/fd/anon_inode:inotify
      1 /proc/2472/fd/anon_inode:inotify
      1 /proc/2455/fd/anon_inode:inotify
      1 /proc/2434/fd/anon_inode:inotify
      1 /proc/2416/fd/anon_inode:inotify
      1 /proc/2404/fd/anon_inode:inotify
      1 /proc/2392/fd/anon_inode:inotify
      1 /proc/2389/fd/anon_inode:inotify
      1 /proc/2384/fd/anon_inode:inotify
      1 /proc/2383/fd/anon_inode:inotify
      1 /proc/2382/fd/anon_inode:inotify
      1 /proc/2378/fd/anon_inode:inotify
      1 /proc/2263/fd/anon_inode:inotify
      1 /proc/2253/fd/anon_inode:inotify
      1 /proc/2245/fd/anon_inode:inotify
      1 /proc/2222/fd/anon_inode:inotify
      1 /proc/2195/fd/anon_inode:inotify
      1 /proc/2192/fd/anon_inode:inotify

Use sudo ps -aux | grep <pid> to get more info about the process. Or use the following commands to list them all.

find /proc/*/fd/* -type l -lname 'anon_inode:inotify' -exec sh -c 'cat $(dirname {})/../cmdline; echo ""' \; 2>/dev/null

Note that this only checks the number of inotify instances, which is not equal to the number of file descriptors being opened.

Tracing for a list of processes with the number of opening file descriptors is somewhat more complicated. Refer to this or this to get the step-by-step on how to do it, the screenshot of these two sources is appended at the end of this post.

Source

Sign up for more like this.