Unallocated shard in Elasticsearch, and often unavailable

[BurningDog]

After setting up and running on a Digital Ocean droplet (4 GB RAM) I had multiple problems. Logstash was able to set up an index in Elasticsearch - logstash-2020.12.09-000001 - but Elasticsearch stayed in yellow health status because of an unassigned shard. Querying Elasticsearch with GET /_cat/shards?v showed

{
  "cluster_name": "docker-cluster",
  "status": "yellow",
  "timed_out": false,
  "number_of_nodes": 1,
  "number_of_data_nodes": 1,
  "active_primary_shards": 1,
  "active_shards": 1,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 1,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 87.5,
  "indices": {
    "logstash-2020.12.09-000001": {
      "status": "yellow",
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "active_primary_shards": 1,
      "active_shards": 1,
      "relocating_shards": 0,
      "initializing_shards": 0,
      "unassigned_shards": 1,
      "shards": {
        "0": {
          "status": "yellow",
          "primary_active": true,
          "active_shards": 1,
          "relocating_shards": 0,
          "initializing_shards": 0,
          "unassigned_shards": 1
        }
      }
    }
  }
}

Notice "unassigned_shards": 1, - the value should be 0 as this is a single node cluster.

Doing

GET _cluster/allocation/explain -d '{
  "index": "logstash-2020.12.09-000001",
  "shard": 0,
  "primary": true
}'

showed:

{
  "index": "logstash-2020.12.09-000001",
  "shard": 0,
  "primary": true,
  "current_state": "started",
  "current_node": {
    "id": "9d_i9BLtTQGndsrZqY2CIQ",
    "name": "a8c5e5ec8685",
    "transport_address": "172.20.0.3:9300",
    "attributes": {
      "ml.machine_memory": "4127453184",
      "xpack.installed": "true",
      "transform.node": "true",
      "ml.max_open_jobs": "20"
    },
    "weight_ranking": 1
  },
  "can_remain_on_current_node": "yes",
  "can_rebalance_cluster": "no",
  "can_rebalance_cluster_decisions": [
    {
      "decider": "rebalance_only_when_active",
      "decision": "NO",
      "explanation": "rebalancing is not allowed until all replicas in the cluster are active"
    },
    {
      "decider": "cluster_rebalance",
      "decision": "NO",
      "explanation": "the cluster has unassigned shards and cluster setting [cluster.routing.allocation.allow_rebalance] is set to [indices_all_active]"
    }
  ],
  "can_rebalance_to_other_node": "no",
  "rebalance_explanation": "rebalancing is not allowed"
}

I'm not an Elasticsearch expert, and now I'm having to read documentation on what all of this stuff means, and what to do about it. Eventually I did

PUT /_settings
{
    "index" : {
        "number_of_replicas" : 0
    }
}

Ok great! Elasticsearch is now green!

However, logstash just can't talk to Elasticsearch. From the logs:

logstash         | [2020-12-09T19:06:00,332][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

What now?! Sshing into the logstash container, I did a number of curl http://elasticsearch:9200/ and only the first one worked:

curl http://elasticsearch:9200/
{
  "name" : "a8c5e5ec8685",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "iT6MF_DsTNSOylme1dMgPg",
  "version" : {
    "number" : "7.9.3",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "c4138e51121ef06a6404866cddc601906fe5c868",
    "build_date" : "2020-10-16T10:36:16.141335Z",
    "build_snapshot" : false,
    "lucene_version" : "8.6.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
bash-4.2$ curl http://elasticsearch:9200/
curl: (7) Failed connect to elasticsearch:9200; Connection refused
bash-4.2$ curl http://elasticsearch:9200/
curl: (7) Failed connect to elasticsearch:9200; Connection refused
bash-4.2$ curl http://elasticsearch:9200/
curl: (7) Failed connect to elasticsearch:9200; Connection refused
bash-4.2$ curl http://elasticsearch:9200/
curl: (7) Failed connect to elasticsearch:9200; Connection refused
bash-4.2$

And Elasticsearch is throwing this exception on boot:

"stacktrace": ["org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [.kibana][_doc][space:default]: routing [null]]",

It's probably because I only have 4 GB RAM on my Digital Ocean droplet; https://elk-docker.readthedocs.io/#prerequisites says that it needs A minimum of 4GB RAM assigned to Docker. If this is the case, it should be mentioned in this project's readme.