Hot/warm and cold buckets

There is a lot of discussion around index buckets when administering Splunk, for good reason—but it's a bit of a difficult subject to get your head around when you're just getting started, so here is a simplified, but accurate, introductory description of these concepts that you'll use the most in daily administration work, and which may warrant some consideration in the installation configuration process (regarding what disk storage to use). We'll also cover this subject again in the next chapter.

Again, incoming data is stored in indexes. Indexes have buckets, which is where event data is stored; buckets are directories organized by age. Hot buckets are the current files that are open and being written to; hot buckets eventually reach a size or age where they are closed and placed in a date-ranged directory, at which point they become warm buckets. Hot and warm buckets reside in the .../myindex/db directory; warm buckets that reach a certain age are moved to the /colddb directory and become cold buckets. Note that the cold buckets directory could reside on cheaper storage off the indexer, which comes into play when we look at sizing an indexing cluster. These cold buckets are still searchable, but searches will take longer – typically, older data is searched for less frequently, so this is not a huge issue. A final stage in the bucket lifespan is when cold buckets exceed a configured age. Then, they are moved to a frozen state and are either stored or deleted. If they were stored, they can be retrieved; when frozen buckets are opened and decompressed, they move to the .../thaweddb directory.

Hot and warm buckets are, by default, stored in /opt/splunk/var/lib/splunk (Linux) or C:\Program Files\Splunk\var\lib\splunk (Windows). Indexes reside in directories under this initial path, and under that are directories for hot and warm buckets (.../<index>/db/), cold buckets (.../<index>/colddb/), and a few other directories we won't worry about for now. For example:

hot bucket (files being written to)
/opt/splunk/var/lib/splunk/myindex/db/hot_v1_41
warm bucket (closed for writing, searchable)
/opt/splunk/var/lib/splunk/myindex/db/db_1530043376_1529957920_40/
cold bucket (searchable, may reside on different storage)
/opt/splunk/var/lib/splunk/myindex/colddb/db_1508276979_1508276438_0/