Example dynamic cluster membership with tests. #155

mgwidmann · 2019-08-15T02:24:03Z

Currently the tests fail due to the issue #154

@derekkraan Can you get the crashes to stop happening?

I know its a lot of output, but this is everything output from running mix test. There are only 2 tests.

✗ mix test

22:06:02.822 [info]  Application logger started at :nonode@nohost

22:06:02.825 [info]  Application jason started at :nonode@nohost

22:06:02.825 [info]  Application libcluster started at :nonode@nohost

22:06:02.826 [info]  Application local_cluster started at :nonode@nohost

22:06:02.833 [info]  Child :telemetry_handler_table of Supervisor :telemetry_sup started
Pid: #PID<0.192.0>
Start Call: :telemetry_handler_table.start_link()
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:02.833 [info]  Application telemetry started at :nonode@nohost

22:06:02.837 [info]  Child :telemetry_poller_default of Supervisor :telemetry_poller_sup started
Pid: #PID<0.197.0>
Start Call: :telemetry_poller.start_link([measurements: [:memory, :total_run_queue_lengths], name: :telemetry_poller_default])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:02.837 [info]  Application telemetry_poller started at :nonode@nohost

22:06:02.842 [info]  Application merkle_map started at :nonode@nohost

22:06:02.842 [info]  Application delta_crdt started at :nonode@nohost

22:06:02.844 [info]  Application libring started at :nonode@nohost

22:06:02.844 [info]  Application horde started at :nonode@nohost

22:06:02.863 [info]  Child Dynamic.EntityRegistry.Crdt of Supervisor Dynamic.EntityRegistry.Supervisor started
Pid: #PID<0.210.0>
Start Call: DeltaCrdt.start_link(DeltaCrdt.AWLWWMap, [sync_interval: 300, max_sync_size: :infinite, shutdown: 30000, crdt: DeltaCrdt.AWLWWMap, on_diffs: #Function<0.13952030/1 in Horde.RegistrySupervisor.delta_crdt_options/1>, name: Dynamic.EntityRegistry.Crdt])
Restart: :permanent
Shutdown: 30000
Type: :worker

22:06:02.863 [info]  Starting Horde.RegistryImpl with name Dynamic.EntityRegistry

22:06:02.870 [info]  Child Dynamic.EntityRegistry of Supervisor Dynamic.EntityRegistry.Supervisor started
Pid: #PID<0.211.0>
Start Call: Horde.RegistryImpl.start_link([name: Dynamic.EntityRegistry, members: [Dynamic.EntityRegistry]])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:02.870 [info]  Child Horde.Registry of Supervisor Dynamic.Supervisor started
Pid: #PID<0.209.0>
Start Call: Horde.Registry.start_link([id: Horde.Registry, name: Dynamic.EntityRegistry, keys: :unique])
Restart: :permanent
Shutdown: :infinity
Type: :supervisor

22:06:02.877 [info]  Child Dynamic.EntitySupervisor.Crdt of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.213.0>
Start Call: DeltaCrdt.start_link(DeltaCrdt.AWLWWMap, [sync_interval: 300, max_sync_size: :infinite, shutdown: 30000, crdt: DeltaCrdt.AWLWWMap, on_diffs: #Function<0.83009929/1 in Horde.SupervisorSupervisor.delta_crdt_options/1>, name: Dynamic.EntitySupervisor.Crdt])
Restart: :permanent
Shutdown: 30000
Type: :worker

22:06:02.877 [info]  Starting Horde.SupervisorImpl with name Dynamic.EntitySupervisor

22:06:02.877 [info]  Child Horde.SupervisorImpl of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.214.0>
Start Call: Horde.SupervisorImpl.start_link([name: Dynamic.EntitySupervisor, root_name: Dynamic.EntitySupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:02.877 [info]  Child Horde.GracefulShutdownManager of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.215.0>
Start Call: GenServer.start_link(Horde.GracefulShutdownManager, Dynamic.EntitySupervisor.Crdt, [name: Dynamic.EntitySupervisor.GracefulShutdownManager])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:02.878 [info]  Child Dynamic.EntitySupervisor.ProcessesSupervisor of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.216.0>
Start Call: Horde.DynamicSupervisor.start_link([shutdown: :infinity, graceful_shutdown_manager: Dynamic.EntitySupervisor.GracefulShutdownManager, root_name: Dynamic.EntitySupervisor, type: :supervisor, name: Dynamic.EntitySupervisor.ProcessesSupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1])
Restart: :permanent
Shutdown: :infinity
Type: :supervisor

22:06:02.878 [info]  Child Horde.SignalShutdown of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.217.0>
Start Call: GenServer.start_link(Horde.SignalShutdown, [Dynamic.EntitySupervisor.GracefulShutdownManager, Dynamic.EntitySupervisor])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:02.880 [info]  Child Dynamic.EntitySupervisor_telemetry_poller of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.218.0>
Start Call: :telemetry_poller.start_link([measurements: [{:process_info, [name: Dynamic.EntitySupervisor, event: [:horde, :supervisor], keys: [:message_queue_len]]}, {Horde.SupervisorTelemetryPoller, :poll, [Dynamic.EntitySupervisor]}], period: 5000, name: Dynamic.EntitySupervisor_telemetry_poller])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:02.880 [info]  Child Horde.Supervisor of Supervisor Dynamic.Supervisor started
Pid: #PID<0.212.0>
Start Call: Horde.Supervisor.start_link([name: Dynamic.EntitySupervisor, strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1])
Restart: :permanent
Shutdown: :infinity
Type: :supervisor

22:06:02.880 [info]  Child NodeListener of Supervisor Dynamic.Supervisor started
Pid: #PID<0.219.0>
Start Call: NodeListener.start_link([])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:02.886 [info]  Child Dynamic.ClusterConnector of Supervisor Dynamic.Supervisor started
Pid: #PID<0.220.0>
Start Call: Task.start_link(#Function<0.1039468/0 in Dynamic.Application.start/2>)
Restart: :transient
Shutdown: 5000
Type: :worker

22:06:02.886 [info]  Application dynamic started at :nonode@nohost

22:06:02.901 [info]  Child :erl_epmd of Supervisor :net_sup started
Pid: #PID<0.224.0>
Start Call: :erl_epmd.start_link()
Restart: :permanent
Shutdown: 2000
Type: :worker

22:06:02.904 [info]  Child :auth of Supervisor :net_sup started
Pid: #PID<0.225.0>
Start Call: :auth.start_link()
Restart: :permanent
Shutdown: 2000
Type: :worker

22:06:02.907 [info]  Child :net_kernel of Supervisor :net_sup started
Pid: #PID<0.226.0>
Start Call: :net_kernel.start_link([:"[email protected]"], false)
Restart: :permanent
Shutdown: 2000
Type: :worker

22:06:02.907 [info]  Child :net_sup_dynamic of Supervisor :kernel_sup started
Pid: #PID<0.223.0>
Start Call: :erl_distribution.start_link([:"[email protected]"], false)
Restart: :permanent
Shutdown: 1000
Type: :supervisor

22:06:02.921 [info]  Child ExUnit.Server of Supervisor ExUnit.Supervisor started
Pid: #PID<0.236.0>
Start Call: ExUnit.Server.start_link([])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:02.921 [info]  Child ExUnit.CaptureServer of Supervisor ExUnit.Supervisor started
Pid: #PID<0.237.0>
Start Call: ExUnit.CaptureServer.start_link([])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:02.921 [info]  Child ExUnit.OnExitHandler of Supervisor ExUnit.Supervisor started
Pid: #PID<0.238.0>
Start Call: ExUnit.OnExitHandler.start_link([])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:02.921 [info]  Application ex_unit started at :"[email protected]"
warning: variable "node1_entity" is unused (if the variable is not meant to be used, prefix it with an underscore)
  test/dynamic_test.exs:58

warning: variable "node2_entity" is unused (if the variable is not meant to be used, prefix it with an underscore)
  test/dynamic_test.exs:62


22:06:03.009 [info]  Child of Supervisor :inet_gethost_native_sup started
Pid: #PID<0.252.0>
Start Call: :inet_gethost_native.init([])

22:06:03.009 [info]  Child :inet_gethost_native_sup of Supervisor :kernel_safe_sup started
Pid: #PID<0.251.0>
Start Call: :inet_gethost_native.start_link()
Restart: :temporary
Shutdown: 1000
Type: :worker

22:06:05.240 [info]  Starting Horde.RegistryImpl with name Dynamic.EntityRegistry

22:06:05.240 [info]  Starting Horde.RegistryImpl with name Dynamic.EntityRegistry

22:06:05.278 [info]  Starting Horde.SupervisorImpl with name Dynamic.EntitySupervisor

22:06:05.278 [info]  Starting Horde.SupervisorImpl with name Dynamic.EntitySupervisor
warning: redefining module Dynamic.EntityTest (current version defined in memory)
  test/dynamic_test.exs:1

warning: variable "node1_entity" is unused (if the variable is not meant to be used, prefix it with an underscore)
  test/dynamic_test.exs:58

warning: variable "node2_entity" is unused (if the variable is not meant to be used, prefix it with an underscore)
  test/dynamic_test.exs:62

Test nodes that are now online: [:"[email protected]", :"[email protected]"]


  1) test multiple nodes a node goes down (Dynamic.EntityTest)
     test/dynamic_test.exs:85
     ** (CaseClauseError) no case clause matching: []
     code: assert Dynamic.get_entity(node1_entity_str) == node1_entity_data
     stacktrace:
       (dynamic) lib/dynamic.ex:17: Dynamic.get_entity_pid/1
       (dynamic) lib/dynamic.ex:11: Dynamic.get_entity/1
       test/dynamic_test.exs:99: (test)


22:06:06.351 [error] Child Dynamic.EntitySupervisor.ProcessesSupervisor of Supervisor Dynamic.EntitySupervisor.Supervisor terminated
** (exit) normal
Pid: #PID<0.216.0>
Start Call: Horde.DynamicSupervisor.start_link([shutdown: :infinity, graceful_shutdown_manager: Dynamic.EntitySupervisor.GracefulShutdownManager, root_name: Dynamic.EntitySupervisor, type: :supervisor, name: Dynamic.EntitySupervisor.ProcessesSupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1])
Restart: :permanent
Shutdown: :infinity
Type: :supervisor

22:06:06.361 [error] GenServer Dynamic.EntitySupervisor terminating
** (stop) exited in: GenServer.stop(Dynamic.EntitySupervisor.ProcessesSupervisor, :normal, :infinity)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir) lib/gen_server.ex:938: GenServer.stop/3
    (horde) lib/horde/supervisor_impl.ex:581: Horde.SupervisorImpl.shut_down_all_processes/1
    (horde) lib/horde/supervisor_impl.ex:358: Horde.SupervisorImpl.handle_info/2
    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:crdt_update, [{:add, {:member, {Dynamic.EntitySupervisor, :"[email protected]"}}, 1}]}
State: %Horde.SupervisorImpl{distribution_strategy: Horde.UniformQuorumDistribution, local_process_count: 4, members: %{{Dynamic.EntitySupervisor, :"[email protected]"} => 1, {Dynamic.EntitySupervisor, :"[email protected]"} => 1}, members_info: %{{Dynamic.EntitySupervisor, :"[email protected]"} => %Horde.Supervisor.Member{name: {Dynamic.EntitySupervisor, :"[email protected]"}, status: :alive}, {Dynamic.EntitySupervisor, :"[email protected]"} => %Horde.Supervisor.Member{name: {Dynamic.EntitySupervisor, :"[email protected]"}, status: :uninitialized}}, name: Dynamic.EntitySupervisor, name_to_supervisor_ref: %{{Dynamic.EntitySupervisor, :"[email protected]"} => #Reference<0.2045498330.1376518145.173581>, {Dynamic.EntitySupervisor, :nonode@nohost} => #Reference<0.2045498330.1376518145.173478>}, process_pid_to_id: %{#PID<17550.251.0> => 203344144639827332175378368881323364659, #PID<17550.252.0> => 336577234915540442433944746061623993228, #PID<17550.253.0> => 230434561701108535677139095368514670192, #PID<0.283.0> => 1308682357906732534861808327661391099, #PID<0.294.0> => 203344144639827332175378368881323364659, #PID<0.297.0> => 230434561701108535677139095368514670192, #PID<0.298.0> => 336577234915540442433944746061623993228}, processes_by_id: %{1308682357906732534861808327661391099 => {{Dynamic.EntitySupervisor, :"[email protected]"}, %{id: 1308682357906732534861808327661391099, restart: :transient, shutdown: 10000, start: {Dynamic.Entity, :start_link, ["one", %{one: "data"}]}}, #PID<0.283.0>}, 203344144639827332175378368881323364659 => {{Dynamic.EntitySupervisor, :"[email protected]"}, %{id: 203344144639827332175378368881323364659, restart: :transient, shutdown: 10000, start: {Dynamic.Entity, :start_link, ["two", %{two: "data"}]}}, #PID<0.294.0>}, 230434561701108535677139095368514670192 => {{Dynamic.EntitySupervisor, :"[email protected]"}, %{id: 230434561701108535677139095368514670192, restart: :transient, shutdown: 10000, start: {Dynamic.Entity, :start_link, ["four", %{four: "data"}]}}, #PID<0.297.0>}, 336577234915540442433944746061623993228 => {{Dynamic.EntitySupervisor, :"[email protected]"}, %{id: 336577234915540442433944746061623993228, restart: :transient, shutdown: 10000, start: {Dynamic.Entity, :start_link, ["three", %{three: "data"}]}}, #PID<0.298.0>}}, shutting_down: false, supervisor_options: [name: Dynamic.EntitySupervisor, root_name: Dynamic.EntitySupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1], supervisor_ref_to_name: %{#Reference<0.2045498330.1376518145.173478> => {Dynamic.EntitySupervisor, :nonode@nohost}, #Reference<0.2045498330.1376518145.173581> => {Dynamic.EntitySupervisor, :"[email protected]"}}, waiting_for_quorum: []}

22:06:06.361 [error] Process Dynamic.EntitySupervisor (#PID<0.214.0>) terminating
** (exit) exited in: GenServer.stop(Dynamic.EntitySupervisor.ProcessesSupervisor, :normal, :infinity)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir) lib/gen_server.ex:938: GenServer.stop/3
    (horde) lib/horde/supervisor_impl.ex:581: Horde.SupervisorImpl.shut_down_all_processes/1
    (horde) lib/horde/supervisor_impl.ex:358: Horde.SupervisorImpl.handle_info/2
    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Initial Call: Horde.SupervisorImpl.init/1
Ancestors: [Dynamic.EntitySupervisor.Supervisor, Dynamic.Supervisor, #PID<0.207.0>]
Message Queue Length: 1
Messages: [{:"$gen_call", {#PID<0.217.0>, #Reference<0.2045498330.1376518145.174133>}, :horde_shutting_down}]
Links: [#PID<0.212.0>]
Dictionary: [rand_seed: {%{bits: 58, jump: #Function<8.10897371/1 in :rand.mk_alg/1>, next: #Function<5.10897371/1 in :rand.mk_alg/1>, type: :exrop, uniform: #Function<6.10897371/1 in :rand.mk_alg/1>, uniform_n: #Function<7.10897371/2 in :rand.mk_alg/1>, weak_low_bits: 1}, [31528902473244349 | 842880235163533]}]
Trapping Exits: true
Status: :running
Heap Size: 6772
Stack Size: 27
Reductions: 93712

22:06:06.362 [error] GenServer #PID<0.217.0> terminating
** (stop) exited in: GenServer.call(Dynamic.EntitySupervisor, :horde_shutting_down, 5000)
    ** (EXIT) exited in: GenServer.stop(Dynamic.EntitySupervisor.ProcessesSupervisor, :normal, :infinity)
        ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir) lib/gen_server.ex:989: GenServer.call/3
    (horde) lib/horde/signal_shutdown.ex:21: anonymous fn/1 in Horde.SignalShutdown.terminate/2
    (elixir) lib/enum.ex:769: Enum."-each/2-lists^foreach/1-0-"/2
    (elixir) lib/enum.ex:769: Enum.each/2
    (stdlib) gen_server.erl:673: :gen_server.try_terminate/3
    (stdlib) gen_server.erl:858: :gen_server.terminate/10
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:EXIT, #PID<0.212.0>, :shutdown}
State: [Dynamic.EntitySupervisor.GracefulShutdownManager, Dynamic.EntitySupervisor]

22:06:06.362 [error] Process #PID<0.217.0> terminating
** (exit) exited in: GenServer.call(Dynamic.EntitySupervisor, :horde_shutting_down, 5000)
    ** (EXIT) exited in: GenServer.stop(Dynamic.EntitySupervisor.ProcessesSupervisor, :normal, :infinity)
        ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir) lib/gen_server.ex:989: GenServer.call/3
    (horde) lib/horde/signal_shutdown.ex:21: anonymous fn/1 in Horde.SignalShutdown.terminate/2
    (elixir) lib/enum.ex:769: Enum."-each/2-lists^foreach/1-0-"/2
    (elixir) lib/enum.ex:769: Enum.each/2
    (stdlib) gen_server.erl:673: :gen_server.try_terminate/3
    (stdlib) gen_server.erl:858: :gen_server.terminate/10
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Initial Call: Horde.SignalShutdown.init/1
Ancestors: [Dynamic.EntitySupervisor.Supervisor, Dynamic.Supervisor, #PID<0.207.0>]
Message Queue Length: 0
Messages: []
Links: []
Dictionary: []
Trapping Exits: true
Status: :running
Heap Size: 6772
Stack Size: 27
Reductions: 4727

22:06:06.362 [error] Child Horde.SignalShutdown of Supervisor Dynamic.EntitySupervisor.Supervisor shut down abnormally
** (exit) exited in: GenServer.call(Dynamic.EntitySupervisor, :horde_shutting_down, 5000)
    ** (EXIT) exited in: GenServer.stop(Dynamic.EntitySupervisor.ProcessesSupervisor, :normal, :infinity)
        ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
Pid: #PID<0.217.0>
Start Call: GenServer.start_link(Horde.SignalShutdown, [Dynamic.EntitySupervisor.GracefulShutdownManager, Dynamic.EntitySupervisor])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:06.362 [error] Child Horde.SupervisorImpl of Supervisor Dynamic.EntitySupervisor.Supervisor shut down abnormally
** (exit) exited in: GenServer.stop(Dynamic.EntitySupervisor.ProcessesSupervisor, :normal, :infinity)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
Pid: #PID<0.214.0>
Start Call: Horde.SupervisorImpl.start_link([name: Dynamic.EntitySupervisor, root_name: Dynamic.EntitySupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:06.372 [info]  Child Dynamic.EntitySupervisor.Crdt of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.307.0>
Start Call: DeltaCrdt.start_link(DeltaCrdt.AWLWWMap, [sync_interval: 300, max_sync_size: :infinite, shutdown: 30000, crdt: DeltaCrdt.AWLWWMap, on_diffs: #Function<0.83009929/1 in Horde.SupervisorSupervisor.delta_crdt_options/1>, name: Dynamic.EntitySupervisor.Crdt])
Restart: :permanent
Shutdown: 30000
Type: :worker

22:06:06.372 [info]  Starting Horde.SupervisorImpl with name Dynamic.EntitySupervisor

22:06:06.372 [info]  Child Horde.SupervisorImpl of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.308.0>
Start Call: Horde.SupervisorImpl.start_link([name: Dynamic.EntitySupervisor, root_name: Dynamic.EntitySupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:06.372 [info]  Child Horde.GracefulShutdownManager of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.309.0>
Start Call: GenServer.start_link(Horde.GracefulShutdownManager, Dynamic.EntitySupervisor.Crdt, [name: Dynamic.EntitySupervisor.GracefulShutdownManager])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:06.372 [info]  Child Dynamic.EntitySupervisor.ProcessesSupervisor of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.310.0>
Start Call: Horde.DynamicSupervisor.start_link([shutdown: :infinity, graceful_shutdown_manager: Dynamic.EntitySupervisor.GracefulShutdownManager, root_name: Dynamic.EntitySupervisor, type: :supervisor, name: Dynamic.EntitySupervisor.ProcessesSupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1])
Restart: :permanent
Shutdown: :infinity
Type: :supervisor

22:06:06.372 [info]  Child Horde.SignalShutdown of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.311.0>
Start Call: GenServer.start_link(Horde.SignalShutdown, [Dynamic.EntitySupervisor.GracefulShutdownManager, Dynamic.EntitySupervisor])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:06.373 [info]  Child Dynamic.EntitySupervisor_telemetry_poller of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.312.0>
Start Call: :telemetry_poller.start_link([measurements: [{:process_info, [name: Dynamic.EntitySupervisor, event: [:horde, :supervisor], keys: [:message_queue_len]]}, {Horde.SupervisorTelemetryPoller, :poll, [Dynamic.EntitySupervisor]}], period: 5000, name: Dynamic.EntitySupervisor_telemetry_poller])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:07.868 [info]  Starting Horde.RegistryImpl with name Dynamic.EntityRegistry

22:06:07.868 [info]  Starting Horde.RegistryImpl with name Dynamic.EntityRegistry

22:06:07.903 [info]  Starting Horde.SupervisorImpl with name Dynamic.EntitySupervisor

22:06:07.903 [info]  Starting Horde.SupervisorImpl with name Dynamic.EntitySupervisor
warning: redefining module Dynamic.EntityTest (current version defined in memory)
  test/dynamic_test.exs:1

warning: variable "node1_entity" is unused (if the variable is not meant to be used, prefix it with an underscore)
  test/dynamic_test.exs:58

warning: variable "node2_entity" is unused (if the variable is not meant to be used, prefix it with an underscore)
  test/dynamic_test.exs:62

Test nodes that are now online: [:"[email protected]", :"[email protected]"]

22:06:08.327 [error] Child Dynamic.EntitySupervisor.ProcessesSupervisor of Supervisor Dynamic.EntitySupervisor.Supervisor terminated
** (exit) normal
Pid: #PID<0.310.0>
Start Call: Horde.DynamicSupervisor.start_link([shutdown: :infinity, graceful_shutdown_manager: Dynamic.EntitySupervisor.GracefulShutdownManager, root_name: Dynamic.EntitySupervisor, type: :supervisor, name: Dynamic.EntitySupervisor.ProcessesSupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1])
Restart: :permanent
Shutdown: :infinity
Type: :supervisor

22:06:08.328 [error] GenServer Dynamic.EntitySupervisor terminating
** (stop) exited in: GenServer.stop(Dynamic.EntitySupervisor.ProcessesSupervisor, :normal, :infinity)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir) lib/gen_server.ex:938: GenServer.stop/3
    (horde) lib/horde/supervisor_impl.ex:581: Horde.SupervisorImpl.shut_down_all_processes/1
    (horde) lib/horde/supervisor_impl.ex:358: Horde.SupervisorImpl.handle_info/2
    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:crdt_update, [{:add, {:member_node_info, {Dynamic.EntitySupervisor, :"[email protected]"}}, %Horde.Supervisor.Member{name: {Dynamic.EntitySupervisor, :"[email protected]"}, status: :shutting_down}}]}
State: %Horde.SupervisorImpl{distribution_strategy: Horde.UniformQuorumDistribution, local_process_count: 1, members: %{{Dynamic.EntitySupervisor, :"[email protected]"} => 1}, members_info: %{{Dynamic.EntitySupervisor, :"[email protected]"} => %Horde.Supervisor.Member{name: {Dynamic.EntitySupervisor, :"[email protected]"}, status: :alive}}, name: Dynamic.EntitySupervisor, name_to_supervisor_ref: %{{Dynamic.EntitySupervisor, :"[email protected]"} => #Reference<0.2045498330.1376518146.173040>, {Dynamic.EntitySupervisor, :"[email protected]"} => #Reference<0.2045498330.1376518145.174676>}, process_pid_to_id: %{#PID<0.337.0> => 30956948180949210312828241188152995010}, processes_by_id: %{30956948180949210312828241188152995010 => {{Dynamic.EntitySupervisor, :"[email protected]"}, %{id: 30956948180949210312828241188152995010, restart: :transient, shutdown: 10000, start: {Dynamic.Entity, :start_link, ["one", %{one: "data"}]}}, #PID<0.337.0>}}, shutting_down: true, supervisor_options: [name: Dynamic.EntitySupervisor, root_name: Dynamic.EntitySupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1], supervisor_ref_to_name: %{#Reference<0.2045498330.1376518145.174676> => {Dynamic.EntitySupervisor, :"[email protected]"}, #Reference<0.2045498330.1376518146.173040> => {Dynamic.EntitySupervisor, :"[email protected]"}}, waiting_for_quorum: []}

22:06:08.329 [error] Process Dynamic.EntitySupervisor (#PID<0.308.0>) terminating
** (exit) exited in: GenServer.stop(Dynamic.EntitySupervisor.ProcessesSupervisor, :normal, :infinity)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir) lib/gen_server.ex:938: GenServer.stop/3
    (horde) lib/horde/supervisor_impl.ex:581: Horde.SupervisorImpl.shut_down_all_processes/1
    (horde) lib/horde/supervisor_impl.ex:358: Horde.SupervisorImpl.handle_info/2
    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Initial Call: Horde.SupervisorImpl.init/1
Ancestors: [Dynamic.EntitySupervisor.Supervisor, Dynamic.Supervisor, #PID<0.207.0>]
Message Queue Length: 0
Messages: []
Links: [#PID<0.212.0>]
Dictionary: [rand_seed: {%{bits: 58, jump: #Function<8.10897371/1 in :rand.mk_alg/1>, next: #Function<5.10897371/1 in :rand.mk_alg/1>, type: :exrop, uniform: #Function<6.10897371/1 in :rand.mk_alg/1>, uniform_n: #Function<7.10897371/2 in :rand.mk_alg/1>, weak_low_bits: 1}, [260826826645588017 | 17108248010956880]}]
Trapping Exits: true
Status: :running
Heap Size: 4185
Stack Size: 27
Reductions: 65646

22:06:08.329 [error] Child Horde.SupervisorImpl of Supervisor Dynamic.EntitySupervisor.Supervisor shut down abnormally
** (exit) exited in: GenServer.stop(Dynamic.EntitySupervisor.ProcessesSupervisor, :normal, :infinity)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
Pid: #PID<0.308.0>
Start Call: Horde.SupervisorImpl.start_link([name: Dynamic.EntitySupervisor, root_name: Dynamic.EntitySupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:08.329 [info]  Child Dynamic.EntitySupervisor.Crdt of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.341.0>
Start Call: DeltaCrdt.start_link(DeltaCrdt.AWLWWMap, [sync_interval: 300, max_sync_size: :infinite, shutdown: 30000, crdt: DeltaCrdt.AWLWWMap, on_diffs: #Function<0.83009929/1 in Horde.SupervisorSupervisor.delta_crdt_options/1>, name: Dynamic.EntitySupervisor.Crdt])
Restart: :permanent
Shutdown: 30000
Type: :worker

22:06:08.329 [info]  Starting Horde.SupervisorImpl with name Dynamic.EntitySupervisor

22:06:08.329 [info]  Child Horde.SupervisorImpl of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.342.0>
Start Call: Horde.SupervisorImpl.start_link([name: Dynamic.EntitySupervisor, root_name: Dynamic.EntitySupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:08.330 [info]  Child Horde.GracefulShutdownManager of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.345.0>
Start Call: GenServer.start_link(Horde.GracefulShutdownManager, Dynamic.EntitySupervisor.Crdt, [name: Dynamic.EntitySupervisor.GracefulShutdownManager])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:08.330 [info]  Child Dynamic.EntitySupervisor.ProcessesSupervisor of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.346.0>
Start Call: Horde.DynamicSupervisor.start_link([shutdown: :infinity, graceful_shutdown_manager: Dynamic.EntitySupervisor.GracefulShutdownManager, root_name: Dynamic.EntitySupervisor, type: :supervisor, name: Dynamic.EntitySupervisor.ProcessesSupervisor, members: [Dynamic.EntitySupervisor], strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 100000, max_seconds: 1])
Restart: :permanent
Shutdown: :infinity
Type: :supervisor

22:06:08.330 [info]  Child Horde.SignalShutdown of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.347.0>
Start Call: GenServer.start_link(Horde.SignalShutdown, [Dynamic.EntitySupervisor.GracefulShutdownManager, Dynamic.EntitySupervisor])
Restart: :permanent
Shutdown: 5000
Type: :worker

22:06:08.330 [info]  Child Dynamic.EntitySupervisor_telemetry_poller of Supervisor Dynamic.EntitySupervisor.Supervisor started
Pid: #PID<0.348.0>
Start Call: :telemetry_poller.start_link([measurements: [{:process_info, [name: Dynamic.EntitySupervisor, event: [:horde, :supervisor], keys: [:message_queue_len]]}, {Horde.SupervisorTelemetryPoller, :poll, [Dynamic.EntitySupervisor]}], period: 5000, name: Dynamic.EntitySupervisor_telemetry_poller])
Restart: :permanent
Shutdown: 5000
Type: :worker


  2) test multiple nodes entity data fetch from other nodes (Dynamic.EntityTest)
     test/dynamic_test.exs:56
     ** (CaseClauseError) no case clause matching: []
     stacktrace:
       (dynamic) lib/dynamic.ex:17: Dynamic.get_entity_pid/1
       test/dynamic_test.exs:28: Dynamic.EntityTest.__ex_unit_setup_0/1
       test/dynamic_test.exs:1: Dynamic.EntityTest.__ex_unit__/2



Finished in 5.3 seconds
2 tests, 2 failures

Randomized with seed 925875

mgwidmann · 2019-08-17T02:23:14Z

Have you had a chance to look at this test? I'm seeing shutdowns occur while my system is running at full load, not sure if its related but it certainly causes a lot of noise when it does since it all I can find is this same error message. Perhaps there is a further root cause I cannot find on why the system is shutting down...

derekkraan · 2019-08-17T05:55:07Z

I haven't had a chance to look at this yet. Work is busy these days.

Example dynamic cluster membership with tests.

741790d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Example dynamic cluster membership with tests. #155

Example dynamic cluster membership with tests. #155

mgwidmann commented Aug 15, 2019 •

edited

mgwidmann commented Aug 17, 2019

derekkraan commented Aug 17, 2019

Example dynamic cluster membership with tests. #155

Are you sure you want to change the base?

Example dynamic cluster membership with tests. #155

Conversation

mgwidmann commented Aug 15, 2019 • edited

mgwidmann commented Aug 17, 2019

derekkraan commented Aug 17, 2019

mgwidmann commented Aug 15, 2019 •

edited