Configuration file precedence

Splunk uses configuration (.conf) files to control nearly every aspect of its operation. There are numerous configuration files with the same name layered inside of different directories that affect users, an app, or the system as a whole, so upon startup, Splunk merges the contents of these files based upon a directory location-based prioritization scheme to achieve an overall working configuration in memory. The rules Splunk follows when merging these files are as follows:

• When different copies of the same file have conflicting attribute values (when they set the same attribute to different values), it uses the value from the file with the highest precedence.
• The precedence of configuration files is determined by their location in the directory structure – system, app, or user directory, in that order. To determine priority among the numerous collections of files in the app directory, it uses lexicographical order—app directories starting with A have higher priority than apps starting with B and so on.

There is also a concept of global (system) versus app or user contexts. App and user activities, such as searching, take place in an app (search) and user (your ID or role) context to allow control over access to knowledge objects and allowable actions. Activities like indexing data work within a global context, independent of any app or user.

You will most often need to consider precedence order within the global context:

• System local directory—highest priority
• App local directories
• App default directories
• System default directory—lowest priority

Precedence order within an app or user context will make more sense as you work with Splunk apps in later chapters:

• User directories for current user – highest priority
• App directories for currently running app (local, then default)
• App directories for all other apps (local, then default)
• System directories (local, then default) – lowest priority

Configuration file precedence is a fairly complex subject, covering more areas than discussed here; if you need to understand how it works in greater depth, you can read more in the Splunk admin manual starting at this link: http://docs.splunk.com/Documentation/Splunk/7.1.1/Admin/Wheretofindtheconfigurationfiles.

Most of the time, you'll just be working with system and app-level configuration files, so in summary, there are two simple rules that you should follow regarding configuration file precedence that will keep you out of trouble:

• Never change a .conf file in any /default directory. Splunk will overwrite these files (and your changes) when you upgrade Splunk.
• Always create a new, empty, .conf file, or edit an existing one, in the /local directory – this copy will have precedence over a file with the same name in the /default directory, so changes reflected here will always supersede identical settings in a .conf file in the /default directory, and Splunk upgrades don't touch any files in the /local directory. Make sure that Splunk has write permissions to any files you create in the /local directory, as this is also where any changes to the configuration from commands you run from the command-line interface (CLI) or by using Splunk Web will be stored here.