Performance considerations

Splunk provides an excellent guide to all the factors and considerations for hardware options selection in their capacity planning manual; this is a ~35-page document that you will want to read before making a final choice regarding hardware options: 
https://docs.splunk.com/Documentation/Splunk/latest/Capacity/IntroductiontocapacityplanningforSplunkEnterprise.

There are several takeaways from this document that warrant emphasizing:

• If you anticipate needing to scale your Splunk deployment to any extent, you will likely be building a distributed environment with an indexing cluster and a search head cluster.
• As discussed in previous sections, the primary factors affecting the performance of your Splunk solution are the volume of data being indexed and the type and number of concurrent searches that will be encountered. 
• A Splunk indexer has to split its disk resources between indexing incoming data and servicing search requests, both of which can be I/O-intensive. An indexer dedicates a CPU core to each search request, as well as taking care of other system requests and the indexing function. When the available number of available cores is consumed, all activities slow down as processing time is split between indexing and servicing search requests, and disk I/O is involved in both activities, so the practical solution to improving performance is adding more indexers.
• Splunk identifies four types of searches that impact the indexers in different ways; dense and sparse search types return between roughly 1 to 10 percent of the searched events, and tax the CPU from decompressing the raw data, while super-sparse and rare searches return a relatively smaller number of events and taxes disk I/O from the need to search through so many buckets looking for events that match the search criteria.
• Search heads aren't as disk-I/O-intensive, but in such a fashion to indexers, search heads do dedicate a CPU core to each search job. If all of the available cores are consumed, searches will be queued and search results will return more slowly; Splunk will warn you when the system reaches the maximum number of queued saved searches allowed, and it is possible that saved searches will be skipped on an overtaxed system.
• You can add more search heads to accommodate more concurrent searches, but indexer capability will likely need to be increased as well (perhaps before adding more search heads) to handle the higher number of concurrent search requests.
• If you plan to install any Splunk apps, you will want to review their installation manuals and consider the unique resource requirements for these apps in your design. A Splunk app is a prebuilt collection of dashboards and other UI elements and the saved searches that power them, packaged together for a specific technology. There are Splunk apps for a wide variety of technologies, including the Splunk app for Unix and Linux, the Splunk app for VMware, Splunk Enterprise Security, and so on. You can peruse all the Splunk apps that are available here: https://splunkbase.splunk.com/.

Splunk promotes the concept of a reference server specification for dedicated search heads and includes the following:

• 16 CPU cores at 2GHz or better per core on Intel x86 64-bit CPUs
• 12 GB of RAM
• A 1 GB Ethernet NIC
• 300 GB of RAID 1 (mirrored) disk storage
• A standard 64-bit distribution of Linux or Windows

The reference server specifications for indexers includes various performance levels (basic, midrange, and high-performance), such as the following:

• 12-48 CPU cores at 2GHz or better
• 12-128 GB of RAM
• A 1 GB Ethernet NIC
• A disk subsystem offering at least 800-1,200 average IOPS (input/output operations per second) with RAID 1+0 (mirroring and striping)
• A standard 64-bit distribution of Linux or Windows

For the best indexing performance, you could employ a solid-state disk (SSD) for storing hot and warm index buckets (which are instantly searchable), and relegate cold buckets (which can be searched but it takes longer) to a slower, cheaper, rotating disk.

A few more pertinent notes from the capacity manual include the following:

• You can also use Elastic Block Storage (EBS) on AWS instances, if you take care to select volume types and sizes that provide the necessary IOPS to handle Splunk Enterprise operations for searching and indexing; generally, the larger the EBS volume selected, the better the IOPS performance. You should also select an EBS-optimized EC2 instance to ensure adequate network bandwidth (10 GB) to support the needed throughput between the instance and EBS volume.
• The ratio of universal forwarders (UFs) to indexers that can be accommodated depends heavily on indexer performance in terms of CPUs and disk I/O, and the amount of data being sent to the indexers. Splunk conducted tests that indicated a ratio of between 2,000-5,000 UFs per indexer could be handled (keeping in mind the data ingestion volumes); performance increased when the server was configured for a higher number of Unix file descriptors – that is, three to four times the number of forwarders the indexer could accept. We will touch on Unix file descriptors later in this chapter.
• If you have indexers and search heads that exceed the reference server specifications in terms of the number of CPUs and disk I/O performance, you may be able to use parallelization settings that improve search and indexing performance and better leverage your hardware investment.