Everything about inotify in Ubuntu.

  • Check current config
$ sysctl fs.inotify
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 65536
  • To modify the configuration permanently. sudo vim /etc/sysctl.conf, modify these following lines or add new if they do not exist
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 16384

Then run sudo sysctl -p to reload the changes.

  • To modify the configuration online which does not affect in the next restart. Use these command lines
sudo sysctl -n -w fs.inotify.max_queued_events=16384
sudo sysctl -n -w fs.inotify.max_user_instances=128
sudo sysctl -n -w fs.inotify.max_user_watches=16384
  • Find number of files being opened
$ lsof | grep inotify | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/121/gvfs
      Output information may be incomplete.
4378
  • To check if the inotify resource has already been limited. Use tail -f.
    Because tail use inotify internally to watch for file update, the command will fail if the the inotify resource is drained.
    The error (if happens) would be tail: inotify cannot be used, reverting to polling: Too many open files.
    For other programs, it usually shows this error instead
Error: ENOSPC: System limit for number of file watchers reached, watch ....

In Intellij editors, it shows

The current inotify(7) watch limit is too low

or

External file changes sync may be slow
  • Check which processes are consuming inotify resources
ps -p $(find /proc/*/fd/* -type l -lname 'anon_inode:inotify' -print 2> /dev/null | sed -e 's/^\/proc\/\([0-9]*\)\/.*/\1/')
  • List of processes id with the number of inotify instances they are consuming, sorted by number of inotify instances being consumed.
$ for foo in /proc/*/fd/*; do readlink -f $foo; done | grep inotify | sort | uniq -c | sort -nr

      6 /proc/2183/fd/anon_inode:inotify
      2 /proc/6346/fd/anon_inode:inotify
      2 /proc/3077/fd/anon_inode:inotify
      2 /proc/2190/fd/anon_inode:inotify
      1 /proc/76491/fd/anon_inode:inotify
      1 /proc/6404/fd/anon_inode:inotify
      1 /proc/59641/fd/anon_inode:inotify
      1 /proc/51816/fd/anon_inode:inotify
      1 /proc/40172/fd/anon_inode:inotify
      1 /proc/3844/fd/anon_inode:inotify
      1 /proc/37830/fd/anon_inode:inotify
      1 /proc/3429/fd/anon_inode:inotify
      1 /proc/3323/fd/anon_inode:inotify
      1 /proc/3120/fd/anon_inode:inotify
      1 /proc/3060/fd/anon_inode:inotify
      1 /proc/2837/fd/anon_inode:inotify
      1 /proc/2797/fd/anon_inode:inotify
      1 /proc/2748/fd/anon_inode:inotify
      1 /proc/2729/fd/anon_inode:inotify
      1 /proc/2617/fd/anon_inode:inotify
      1 /proc/2580/fd/anon_inode:inotify
      1 /proc/2574/fd/anon_inode:inotify
      1 /proc/2561/fd/anon_inode:inotify
      1 /proc/2555/fd/anon_inode:inotify
      1 /proc/2550/fd/anon_inode:inotify
      1 /proc/2547/fd/anon_inode:inotify
      1 /proc/2539/fd/anon_inode:inotify
      1 /proc/2515/fd/anon_inode:inotify
      1 /proc/2477/fd/anon_inode:inotify
      1 /proc/2472/fd/anon_inode:inotify
      1 /proc/2455/fd/anon_inode:inotify
      1 /proc/2434/fd/anon_inode:inotify
      1 /proc/2416/fd/anon_inode:inotify
      1 /proc/2404/fd/anon_inode:inotify
      1 /proc/2392/fd/anon_inode:inotify
      1 /proc/2389/fd/anon_inode:inotify
      1 /proc/2384/fd/anon_inode:inotify
      1 /proc/2383/fd/anon_inode:inotify
      1 /proc/2382/fd/anon_inode:inotify
      1 /proc/2378/fd/anon_inode:inotify
      1 /proc/2263/fd/anon_inode:inotify
      1 /proc/2253/fd/anon_inode:inotify
      1 /proc/2245/fd/anon_inode:inotify
      1 /proc/2222/fd/anon_inode:inotify
      1 /proc/2195/fd/anon_inode:inotify
      1 /proc/2192/fd/anon_inode:inotify

Use sudo ps -aux | grep <pid> to get more info about the process. Or use the following commands to list them all.

find /proc/*/fd/* -type l -lname 'anon_inode:inotify' -exec sh -c 'cat $(dirname {})/../cmdline; echo ""' \; 2>/dev/null

Note that this only checks the number of inotify instances, which is not equal to the number of file descriptors being opened.

  • Tracing for list of processes with the number of opening file descriptors is somewhat more complicated. Refer to this or this to get the step by step on how to do, the screenshot of these two sources are appended at the end of this post.

Source