Blog Devops notes

Elasticsearch for Logstash

2013-06-24 00:00:00 -0700


Open Files

Elasticsearch requires that many files can be kept open, so upping ulimit is a good pre-emptive measure. This can be done on Ubuntu simply with pam-limits on a per-user basis.

echo "session required" >> /etc/pam.d/common-session
echo "elasticsearch  hard  nofile  265535" >> /etc/security/limits.conf

Confirm by logging in as user elasticsearch and running:

ulimit -Hn


Give half of your physical memory over to Elasticsearch in the JavaVM. The stock scripts look for the ES_MAX_MEM environment variable. So if you set

export ES_MAX_MEM="16g" 

on a 32g box things should run smoothly.


Using the Elasticsearch API to manage your logstash cluster makes life much simpler

Checking health

$ curl http://localhost:9200/_cluster/health?pretty=true
  "cluster_name" : "shokunin-logstash",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 4,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 36,
  "active_shards" : 72,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0

Checking nodes

 $ curl http://localhost:9200/_nodes?pretty=true
   "ok" : true,
   "cluster_name" : "shokunin-logstash",
   "nodes" : {
     "QibmzJ8pF9x8Szu9xk-DHA" : {
       "name" : "shokunin-logstash04",
       "transport_address" : "inet[/]",
       "hostname" : "",
       "version" : "0.20.5",
       "http_address" : "inet[/]"

Checking indices

By default Logstash creates one index per day like logstash-2013.06.24 . This can be changed similar to create hourly indices. I prefer this so in the event that there are data issues, only one hour worth of data is lost.

#logstash.conf output section
elasticsearch {
  embedded => false
  max_inflight_requests => 1000
  host => "localhost"
  index => "logstash-%{+YYYY.MM.dd.HH}" 
  cluster => "shokunin-logstash"

API Call

$curl http://localhost:9200/logstash-2013.
"ok" : true,
"_shards" : {
  "total" : 10,
  "successful" : 10,
  "failed" : 0
"indices" : {
  "logstash-2013.06.24.19" : {
    "index" : {
      "primary_size" : "8.5gb",
      "primary_size_in_bytes" : 9138037628,
      "size" : "17gb",

Dropping indicies

Getting rid of old logs is simply a matter of deleting the index

curl -XDELETE 'http://localhost:9200/logstash-2013.03.17/'

Clean shutdown

To shutdown the Elasticsearch cluster cleanly flush the data and stop all nodes. Clean shutdowns are necessary when upgrading the version of elasticsearch.

curl -XPOST 'http://localhost:9200/_flush'
#shutdown all cluster nodes
curl -XPOST 'http://localhost:9200/_cluster/nodes/_all/_shutdown'
#shtudown a single node in the cluster
curl -XPOST 'http://localhost:9200/_cluster/nodes/_local/_shutdown'



Checking to be sure that data is updating using this nagios check which checks for updates in the past two minutes by default.

Confirming issues

Running the health check against the API

$ curl http://localhost:9200/_cluster/health?pretty=true
  "cluster_name" : "shokunin-logstash",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 4,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 18,
  "active_shards" : 32,
  "relocating_shards" : 0,
  "initializing_shards" : 4,
  "unassigned_shards" : 28

Shards move from unassigned to initializing to active during the recovery process. Check the number of data nodes matches the expected number. Often a data node will run out of memory and drop out of the cluster.


  • Stop all logstash processes sending data into the cluster
  • Shut all nodes down one by one ( NOTE: you cannot flush when shards are unassigned )
  • Start up all nodes cleanly
  • Run the recovery script in a loop and wait patiently for the status to go back to green. It will give you status and information on how quickly the shards are recovering.
  • Start up all logstash processes sending data to elasticsearch