ulimits

Splunk has some specific requirements regarding the maximum file size, number of open files, user processes, and data segment sizes on the Linux platform—known as ulimits—that you will want to verify. From a Terminal command, type these commands (as root) and verify that the numbers meet or exceed the values given:

ulimit -f unlimited
ulimit -n 64000
ulimit -u 16000
ulimit -d 1073741824 (or unlimited)

(Or just type ulimit -a and read from the list.)

If your settings don't meet or exceed those values, make the following changes (for RHEL 7 running the systemd service):

Edit the /etc/systemd/system.conf file, un-comment (if needed), and change the following settings:

[Manager]
DefaultLimitFSIZE=-1
DefaultLimitNOFILE=64000
DefaultLimitNPROC=16000

Save the file and restart, then verify the settings with the ulimit -a command. If you're running something other than RHEL 7, you will need to search the Splunk installation and troubleshooting documents and/or the web for ulimit instructions for your installation.