Khepri 0.4.0

At this point, Khepri should be considered Alpha and not production ready. The API will likely evolve and future releases will likely introduce breaking changes. The goal of this release is to make it easy for anyone to try it and possibly give feedback.

What's new in Khepri 0.4.0

Add a logo :-)

Let's start with the fanciest change: we added the logo you can spot on the right ;-)

This logo is inspired by the drawing representing the god Khepri: a dung beetle holding the Sun. Here we use a rabbit (for the RabbitMQ-originated project) holding a database diagram. I know, creativity turned to 11.

Revamp of the clustering API

The clustering API was completely revamped in Khepri 0.4.0. Like the rest of the public API in Khepri 0.3.0, the goal is again to make it more straightforward and consistent.

The #105 pull request gives more details about the changes and how existing code can be adapted. But here are some of the highlights:

• It is possible to start a Khepri store with a data directory as argument. No need to configure a Ra system yourself.
• Most functions can now wait for a Ra leader to be elected. They take a timeout as argument to limit that wait. You don't have to handle a retry loop anymore.
• There is a khepri:stop() function now to stop a store.

Here is an example of an old code and its newer version:

• Up to Khepri 0.3.0:

  %% `my_ra_system' is configured using `ra_system:start/1'.
  DataDir = "/var/lib/khepri",
  RaSystemConfig = RaSystemConfig0#{name => my_ra_system,
                                    data_dir => DataDir,
                                    wal_data_dir => DataDir},
  {ok, _} = ra_system:start(RaSystemConfig),
  
  %% The Khepri store is started using that Ra system.
  {ok, StoreId} = khepri:start(my_ra_system).

• Starting from Khepri 0.4.0:

  %% The Khepri store is started with a data directory. A Ra system will be
  %% configured and started automatically.
  {ok, StoreId} = khepri:start("/var/lib/khepri").

See #105.

Several improvements to make it easier to use Khepri for Elixir developers

We had great feedback on the Erlang forums following a question we asked about what should Khepri provide to make it nice to Elixir developers.

As a result, we implemented the following improvements:

1. Khepri path can be Erlang binaries (#93). Elixir and other languages like Gleam running on top of the BEAM runtime use Erlang binaries to implement the "string" type. With this change, their native strings can be used directly in the Khepri API.

   iex> :khepri.get_data("/:foo")
   :value

2. The khepri module exports "bang functions" such as khepri:'get!'(StoreId, Path) (#98). They don't look as nice in Erlang, but in Elixir, they are a common practice to have a version of a function which returns its result directly (if successful) or throws an error if not. They are especially handy when using pipelines.

   iex> :khepri.get!(~p"/:foo")
   %{
     [:foo] => %{
       child_list_length: 0,
       child_list_version: 1,
       data: :value,
       payload_version: 1
     }
   }

   iex> :khepri.get!(:non_existing_store, ~p"/:foo")
   ** (ErlangError) Erlang error: :noproc
       (khepri 0.3.0) /home/dumbbell/Projects/pivotal/khepri/src/khepri.erl:2629: :khepri.unwrap_result/1

3. The khepri_path modules exports ~p and ~P sigils (#98). They are constructs which make the parsing of Khepri Unix-like string/binary-based paths very easy in Elixir.

   iex> import :khepri_path
   
   iex> ~p"/stock/wood/:oak"
   ["stock", "wood", :oak]
   
   iex> ~P"/stock/wood/:oak"
   ["stock", "wood", :oak]

See #93, #98.

EEP-48 doc chunks

In addition to rebar3_edoc_extensions, the plugin we use to produce documentations which are nicer to the eye compared to the default EDoc HTML output, we now also use the EDoc doclet and layout modules to produce EEP-48 doc chunks. People using Khepri should now get documentation and specs right from their compatible IDE and text editors.

► rebar3 shell
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling khepri
Erlang/OTP 24 [erts-12.3.2] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit] [dtrace]

Eshell V12.3.2  (abort with ^G)
1> h(khepri).

   khepri

  Khepri database API.

  This module exposes the database API to manipulate data.

  The API is mainly made of the functions used to perform simple direct atomic operations and queries on the database. In addition to that, transaction/1 are the starting point to run transaction functions. However the API to
  use inside transaction functions is provided by khepri_tx.

  This module also provides functions to start and stop a simple unclustered Khepri store. For more advanced setup and clustering, see khepri_cluster.

Other changes

• Added khepri:count() functions (#102).
• Added support for ETS pattern conditions in #if_data_matches{} conditions (97).
• Improved support for Erlang/OTP 25 (#101).
• Ra was updated from 2.0.9 from 2.0.13 (#110).
• Clarified the difference between RDBMS and Khepri's triggers (#65).

Download

Khepri 0.4.0 is available from Hex.pm: https://hex.pm/packages/khepri/0.4.0

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.4.0"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.4.0

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.4.0 is available on Hex.pm.

Khepri 0.3.0

At this point, Khepri should be considered Alpha and not production ready. The API will likely evolve and future releases will likely introduce breaking changes. The goal of this release is to make it easy for anyone to try it and possibly give feedback.

What's new in Khepri 0.3.0

Revamp of the public API

The public API was completely revamped in Khepri 0.3.0, with the goal to make it more straightforward and consistent. Hopefully it will be easier to understand for both old and new users.

The #79 pull request gives more details about the changes and how existing code can be adapted. But here are some of the highlights:

• The high- vs low-level API distinction is now gone. The public API is now exposed by khepri only. khepri_machine becomes an internal private module. As part of that khepri grew several new functions for common use cases and we will certainly add more in the future, based on the feedback.
• Unix-like path are first-class citizen: all functions taking a native path ([stock, wood, <<"oak">>]) now accept Unix-like paths ("/:stock/:wood/oak"). In the process, the syntax of Unix-like paths evolved: atoms are represented as :atom and binaries are represented as-is, binary. The main reason is that using <<binary>> for binaries was difficult to read and type. See #73 and #74.
• Payload and event filter records are now private. Payload types are automatically detected now, likewise for event filters. That said, it is still possible to use functions to construct the internal structures, but it should rarely be necessary.
• khepri_tx, the module to perform Khepri calls inside transactions, will now expose the same API as khepri, except when functions don't make sense in a transaction.

Here is an example of an old code and its newer version:

• Up to Khepri 0.2.1:

  %% `khepri_machine' had to be used for "advanced" use cases, though
  %% `khepri' would have been fine in this example.
  case khepri_machine:get(StoreId, [stock, wood, <<"oak">>]) of
      %% Accessing the data in the node payload was a bit complicated,
      %% requiring to pattern-match on the node properties map inside the
      %% result map.
      {ok, #{Path := #{data := Quantity}}} when Quantity < 100 ->
          %% We would also have to construct a payload record.
          Payload = #kpayload_data{data = 500},
          {ok, _} = khepri_machine:put(StoreId, [orders, wood, <<"oak">>], Payload),
          ok
      _ ->
          ok
  end.

• Starting from Khepri 0.3.0:

  %% Now we have helpers for common use cases like simply accessing the data
  %% of a single tree node. The piece of data is returned directly, returning
  %% a default value if there is no data and bypassing error handling if we
  %% don't care.
  %%
  %% Unix-like paths are used for the demonstration. Native paths, like in
  %% the previous example, would work as well.
  Quantity = khepri:get_data_or(StoreId, "/:stock/:wood/oak", 0),
  if
      Quantity < 100 ->
          %% The payload record is automatically constructed internally. No
          %% need to mess with records.
          {ok, _} = khepri:put(StoreId, "/:orders/:wood/oak", 500),
          ok;
      true ->
          ok
  end.

See #79.

Options to play with consistency and latency

Queries, including read-only transactions, now accept a favor option to select if Khepri should favor consistency or low latencies:

• consistency: it means that a "consistent query" will be used in Ra. It will return the most up-to-date piece of data the cluster agreed on. Note that it could block and time out if there is no quorum in the Ra cluster.
• compromise: it performs "leader queries" most of the time to reduce latency (though latency will still be higher than "local queries"; see below), but uses "consistent queries" every 10 seconds to verify that the cluster is healthy. It should be faster but may block and time out like consistency and still return slightly out-of-date data.
• low_latency: it means that "local queries" are used exclusively. They are the fastest and have the lowest latency. However, the returned data is whatever the local Ra server has. It could be out-of-date if it has troubles keeping up with the Ra cluster. The chance of blocking and timing out is very small.

%% Favor low latency, even if it means out-of-date data compared to what the consensus is.
khepri:get(StoreId, Path, #{favor => low_latency})

See #64.

Asynchronous updates

Updates, including read-write transactions, now accept an async option to configure asynchronous commands:

• true to perform an asynchronous low-priority command without a correlation ID.
• false to perform a synchronous command.
• A correlation ID to perform an asynchronous low-priority command with that correlation ID.
• A priority to perform an asynchronous command with the specified priority but without a correlation ID.
• A combination of a correlation ID and a priority to perform an asynchronous command with the specified parameters.

%% Configure a correlation ID for this update.
Ref = erlang:make_ref(),
ok = khepri:put(StoreId, Path, Data, #{async => Ref}),

%% Later, consume the reply from Khepri/Ra.
receive
    {ra_event, _, {applied, [{Ref, Reply}]}} ->
        do_things(Reply)
end.

See #69.

Other changes

• Introduced support for Erlang/OTP 25 (#81).
• Ra was updated from 2.0.2 to 2.0.9 (#70, #71, #80, #89).
• Introduced a compiled standalone functions cache (#72). Note that this cache is never cleaned up for now!
• Numerous fixes and improvements to the function extraction code (khepri_fun) from @the-mikedavis (#63, #66, #67, #68 and probably more).

Download

Khepri 0.3.0 is available from Hex.pm: https://hex.pm/packages/khepri/0.3.0

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.3.0"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.3.0

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.3.0 is available on Hex.pm.

Contributors

A warm thank you to contributors outside of the RabbitMQ team, this is invaluable for such a young project!

• Kian-Meng Ang (@kianmeng)
• Michael Davis (@the-mikedavis)

Khepri 0.2.1

At this point, Khepri should be considered Alpha and not production ready. The API will likely evolve and future releases will likely introduce breaking changes. The goal of this release is to make it easy for anyone to try it and possibly give feedback.

What's new in Khepri 0.2.1

This release brings bug fixes only.

• Many new assembly instructions supported in the function extraction code (commit b140d8c, commit 0836441, commit a801495, #54).
• Complete "Line" beam chunk decoding (#55).
• Use a valid SPDX license identifier in rebar.app.

Download

Khepri 0.2.1 is available from Hex.pm: https://hex.pm/packages/khepri/0.2.1

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.2.1"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.2.1

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.2.1 is available on Hex.pm.

Khepri 0.2.0

At this point, Khepri should be considered Alpha and not production ready. The API will likely evolve and future releases will likely introduce breaking changes. The goal of this release is to make it easy for anyone to try it and possibly give feedback.

What's new in Khepri 0.2.0

Stored procedures and triggers

This version introduces stored procedures and triggers. They allow to store code in the database itself and automatically execute it after some event occurs.

1. Store an anonymous function in the tree:

   StoredProcPath = [path, to, stored_procedure],
   
   Fun = fun(Props) ->
             #{path := Path},
               on_action => Action} = Props
         end,
   
   khepri_machine:put(
     StoreId,
     StoredProcPath,
     #kpayload_sproc{sproc = Fun}))}.

2. Register a trigger using an event filter:

   EventFilter = #kevf_tree{path = [stock, wood, <<"oak">>]},
   
   ok = khepri_machine:register_trigger(
          StoreId,
          TriggerId,
          EventFilter,
          StoredProcPath))}.

In the example above, as soon as the [stock, wood, <<"oak">>] node is created, updated or deleted, the anonymous function will be executed.

See #47.

Use records to describe payloads instead of macros

Payloads are now described as Erlang records. This allows to benefit from more compile-time checks in particular.

Here are the modifications you will have to do in your code:

% Up until 0.1.1:
khepri_machine:put(StoreId, Path, ?DATA_PAYLOAD(Data)).

% Starting from 0.2.0:
khepri_machine:put(StoreId, Path, #kpayload_data{data = Data}).

See #20.

keep_until conditions renamed to keep_while

The keep_until option of khepri_machine:put/3 was renamed to keep_while. The reason is that the tree node is kept while the condition is met. As soon as the condition evaluates to false, the tree node is deleted.

Here are the modifications you will have to do in your code:

% Up until 0.1.1:
khepri_machine:put(StoreId, Path, Payload, #{keep_until => #{Path => Condition}}).

% Starting from 0.2.0:
khepri_machine:put(StoreId, Path, Payload, #{keep_while => #{Path => Condition}}).

See #18.

Stacktrace frames pointing to an extracted function code have line numbers

Before, when a transaction function encountered an exception, the stacktrace would miss the source file and line number for frames pointing to the extracted code:

%% Based on the following function:
%%     crashing_fun() ->
%%         throw("Expected crash").

1> Fun = khepri_fun:to_standalone_fun(fun mod:crashing_fun/0, #{}).
2> khepri_fun:exec(Fun, []).
** exception throw: "Expected crash"
     in function  kfun__mod__crashing_fun__33048370:run/0

Now, khepri_fun decode lines information and the stacktrace frames provide the source file and the line number:

1> Fun = khepri_fun:to_standalone_fun(fun mod:crashing_fun/0, #{}).
2> khepri_fun:exec(Fun, []).
** exception throw: "Expected crash"
     in function  kfun__mod__crashing_fun__33048370:run/0 (.../mod.erl, line 30)

See #51.

Other changes

• Ra was updated from 2.0.2 to 2.0.4.
• Several functions of the re Erlang module are allowed in transaction functions (#41).
• Patch several assembly instructions incorrectly decoded by beam_disasm (#39).
• Many new assembly instructions supported in the function extraction code.
• Annotate registers with type information to fix compilation failures related to function extracted (#36, #49).
• The workaround to make it possible to run Dialyzer on testsuite was removed (#23). Now, one just has to use rebar as test dialyzer.
• Modules to generate the nicer documentation were moved to https://hex.pm/packages/rebar3_edoc_extensions.

Download

Khepri 0.2.0 is available from Hex.pm: https://hex.pm/packages/khepri/0.2.0

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.2.0"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.2.0

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.2.0 is available on Hex.pm.

Contributors

A warm thank you to contributors outside of the RabbitMQ team, this is invaluable for such a young project!

• Michael Davis (@the-mikedavis)
• Viacheslav Katsuba (@vkatsuba)

Khepri 0.1.1

At this point, Khepri should be considered Alpha and not production ready. The API will likely evolve and future releases will likely introduce breaking changes. The goal of this release is to make it easy for anyone to try it and possibly give feedback.

What's new in Khepri 0.1.1

ReadWrite argument to khepri_tx:transaction/2 changes

The transaction API changed: khepri_tx:transaction/2 now takes the atoms ro, rw or auto as its second argument, instead of a boolean or auto. It makes it a bit clearer what the argument means out of context, compared to a boolean. This impacts the following APIs:

• khepri:transaction/2
• khepri:transaction/3
• khepri_machine:transaction/2
• khepri_machine:transaction/3

Here are the modifications you will have to do in your code:

%% Read/write transaction:
khepri:transaction(StoreId, Fun, true). % Up until 0.1.0
khepri:transaction(StoreId, Fun, rw).   % Starting from 0.1.1

%% Read/only transaction:
khepri:transaction(StoreId, Fun, false). % Up until 0.1.0
khepri:transaction(StoreId, Fun, ro).    % Starting from 0.1.1

The use of the auto argument remains unchanged.

See commit d7893f3.

Auto-detection of R/W transaction fixed

When a transaction function used another anonymous function which was declared outside of the transaction, that second anonymous function was not analyzed to determine the R/W nature. Therefore, if the transaction function didn't use khepri_tx:put/2 or khepri_tx:delete/1, it would be considered a read-only transaction. Unfortunately, if the second anonymous function did use one of those two APIs, the transaction would abort with a store_update_denied error.

The transaction below would be detected as read-only even though it calls khepri_tx:delete/1:

Fun = fun() -> khepri_tx:delete([foo]) end,
khepri_tx:to_standalone_fun(
  fun() -> Fun() end,
  auto),

Pull request #7 fixes this problem.

Other changes

• Ra was updated from 2.0.1 to 2.0.2.
• Several improvements and fixes to the testsuites and the specifications.
• To be able to run Dialyzer on the testsuites, they are compiled at the same time as the actual source code of Khepri. Unfortunately, it means that PropEr becomes a regular dependency (instead of a test dependency), even though it is not used at runtime. That said it is a regular dependency from Rebar point of view, but not Erlang and the PropEr application won't be started or used at runtime.

Download

Khepri 0.1.1 is available from Hex.pm: https://hex.pm/packages/khepri/0.1.1

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.1.1"}]}.

2. Run rebar3 upgrade.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.1.1

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.1.1 is available on Hex.pm.

Khepri 0.1.0

This is the first release of Khepri.

At this point, Khepri should be considered Alpha and not production ready. The API will likely evolve and future releases will likely introduce breaking changes. The goal of this release is to make it easy for anyone to try it and possibly give feedback.

Khepri 0.1.0 is available from Hex.pm:
https://hex.pm/packages/khepri/0.1.0