Shokunin Consulting LLC.
Blog Devops notes
Elasticsearch for Logstash
2013-06-24 00:00:00 -0700
Installation
Open Files
Elasticsearch requires that many files can be kept open, so upping ulimit is a good pre-emptive measure. This can be done on Ubuntu simply with pam-limits on a per-user basis.
echo "session required pam_limits.so" >> /etc/pam.d/common-session
echo "elasticsearch hard nofile 265535" >> /etc/security/limits.conf
Confirm by logging in as user elasticsearch and running:
ulimit -Hn
Memory
Give half of your physical memory over to Elasticsearch in the JavaVM. The stock scripts look for the ES_MAX_MEM environment variable. So if you set
export ES_MAX_MEM="16g"
on a 32g box things should run smoothly.
Management
Using the Elasticsearch API to manage your logstash cluster makes life much simpler
Checking health
$ curl http://localhost:9200/_cluster/health?pretty=true
{
"cluster_name" : "shokunin-logstash",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 2,
"active_primary_shards" : 36,
"active_shards" : 72,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}
Checking nodes
$ curl http://localhost:9200/_nodes?pretty=true
{
"ok" : true,
"cluster_name" : "shokunin-logstash",
"nodes" : {
"QibmzJ8pF9x8Szu9xk-DHA" : {
"name" : "shokunin-logstash04",
"transport_address" : "inet[/10.23.63.13:9300]",
"hostname" : "shokunin-logstash04.int",
"version" : "0.20.5",
"http_address" : "inet[/10.23.63.13:9200]"
},
CUT...
Checking indices
By default Logstash creates one index per day like logstash-2013.06.24 . This can be changed similar to create hourly indices. I prefer this so in the event that there are data issues, only one hour worth of data is lost.
#logstash.conf output section
elasticsearch {
embedded => false
max_inflight_requests => 1000
host => "localhost"
index => "logstash-%{+YYYY.MM.dd.HH}"
cluster => "shokunin-logstash"
}
API Call
$curl http://localhost:9200/logstash-2013.06.24.19.00/_status?pretty=true
"ok" : true,
"_shards" : {
"total" : 10,
"successful" : 10,
"failed" : 0
},
"indices" : {
"logstash-2013.06.24.19" : {
"index" : {
"primary_size" : "8.5gb",
"primary_size_in_bytes" : 9138037628,
"size" : "17gb",
Dropping indicies
Getting rid of old logs is simply a matter of deleting the index
curl -XDELETE 'http://localhost:9200/logstash-2013.03.17/'
Clean shutdown
To shutdown the Elasticsearch cluster cleanly flush the data and stop all nodes. Clean shutdowns are necessary when upgrading the version of elasticsearch.
curl -XPOST 'http://localhost:9200/_flush'
#shutdown all cluster nodes
curl -XPOST 'http://localhost:9200/_cluster/nodes/_all/_shutdown'
#shtudown a single node in the cluster
curl -XPOST 'http://localhost:9200/_cluster/nodes/_local/_shutdown'
Trouble
Monitoring
Checking to be sure that data is updating using this nagios check which checks for updates in the past two minutes by default.
Confirming issues
Running the health check against the API
$ curl http://localhost:9200/_cluster/health?pretty=true
{
"cluster_name" : "shokunin-logstash",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 2,
"active_primary_shards" : 18,
"active_shards" : 32,
"relocating_shards" : 0,
"initializing_shards" : 4,
"unassigned_shards" : 28
}
Shards move from unassigned to initializing to active during the recovery process. Check the number of data nodes matches the expected number. Often a data node will run out of memory and drop out of the cluster.
Recovering
- Stop all logstash processes sending data into the cluster
- Shut all nodes down one by one ( NOTE: you cannot flush when shards are unassigned )
- Start up all nodes cleanly
- Run the recovery script in a loop and wait patiently for the status to go back to green. It will give you status and information on how quickly the shards are recovering.
- Start up all logstash processes sending data to elasticsearch