Khepri 0.10.1

At this point, Khepri should be considered Beta and not production ready. The API will likely evolve and future releases will likely introduce breaking changes. The goal of this release is to make it easy for anyone to try it and possibly give feedback.

What's new in Khepri 0.10.1

This release fixes the list of files published to Hex.pm: the LICENSE-* files were missing.

See #237.

Download

Khepri 0.10.1 is available from Hex.pm: https://hex.pm/packages/khepri/0.10.1

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.10.1"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.10.1

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.10.1 is available on Hex.pm.

Khepri 0.10.0

At this point, Khepri should be considered Beta and not production ready. The API will likely evolve and future releases will likely introduce breaking changes. The goal of this release is to make it easy for anyone to try it and possibly give feedback.

What's new in Khepri 0.10.0

This release brings a single API change around the async operations.

Unify khepri:handle_async_ret/2 return values

The new khepri:handle_async_ret/2 introduced in Khepri 0.9.0 is improved in this new release: it converts the "raw" return values contained in the ra_event to a format similar to all other Khepri sync commands. This allows the function to handle specific cases like exceptions and aborted transactions.

Here is an example of the change to make to your code:

• Up-to Khepri 0.9.x:

  receive
      {ra_event, _,
       {applied, [{CorrelationId, {ok, _} = Ret}]}} = AsyncRet ->
          %% Call `handle_async_ret/2' for every entries;
          %% here only one:
          ok = khepri:handle_async_ret(StoreId, AsyncRet),
  
          %% Do something with each `Ret'.
          Ret
  end

• Starting from Khepri 0.10.0:

  receive
      {ra_event, _, _} = AsyncRet ->
          %% Call `handle_async_ret/2' once and get a list of correlation
          %% ID/return value pairs.
          [{CorrelationId, Ret}] = khepri:handle_async_ret(StoreId, AsyncRet),
  
          %% Do something with each `Ret'.
          Ret
  end

See #234.

Download

Khepri 0.10.0 is available from Hex.pm: https://hex.pm/packages/khepri/0.10.0

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.10.0"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.10.0

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.10.0 is available on Hex.pm.

Khepri 0.9.0

At this point, Khepri should be considered Beta and not production ready. The API will likely evolve and future releases will likely introduce breaking changes. The goal of this release is to make it easy for anyone to try it and possibly give feedback.

What's new in Khepri 0.9.0

This release focuses on a couple improvements and bug fixes to async commands.

Improvements to async commands

• Correctly use the cached leader ID if it is present to send the command (#228).
• Replace khepri:wait_for_async_ret/{1,2} with khepri:handle_async_ret/{1,2} (#229). The previous API didn't match the underlying Ra API. It's better for the caller to receive the message(s) from the Ra server, then call khepri:handle_async_ret/{1,2} to let Khepri do internal things such as updating the cached leader ID.

Other changes

• Expose the ?IS_KHEPRI_STORE_ID/1 macro (#231).
• Ra was bumped from 2.6.3 to 2.7.0 (#232).

Download

Khepri 0.9.0 is available from Hex.pm: https://hex.pm/packages/khepri/0.9.0

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.9.0"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.9.0

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.9.0 is available on Hex.pm.

Khepri 0.8.0

At this point, Khepri should be considered Beta and not production ready. The API will likely evolve and future releases will likely introduce breaking changes. The goal of this release is to make it easy for anyone to try it and possibly give feedback.

What's new in Khepri 0.8.0

This release focuses on several important bug fixes and improvements to projections.

Many improvements to projections

Projections are a new mechanism introduced in Khepri 0.6.0 to allow fast queries with a low latency.

Here are five pull requests related to projections:

• Add a khepri:unregister_projection() function to unregister projections (#220).
• Make "copy"-style projections more efficient and straightforward to use (#204).
• Handle errors from ETS gracefullly (#205).
• Allow to pass options to Horus (#209).

Other changes

• New khepri:is_empty/1 public function (#210).
• khepri_cluster:join/3 is now idempotent (#219).
• Fix some corner cases in khepri_cluster:join/3 (#225).
• Handle missing default applications when determining the list of modules to skip when extracting a transaction function (#215).
• Handle Ra server shutdown gracefully (#211, #223).
• Improve logging for member queries (#208, #216).
• Fix documentation of put() (#206).
• Ra was bumped from 2.5.1 to 2.6.3 (#224).
• Horus was bumped from 0.2.3 to 0.2.4 (#224).

Download

Khepri 0.8.0 is available from Hex.pm: https://hex.pm/packages/khepri/0.8.0

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.8.0"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.8.0

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.8.0 is available on Hex.pm.

Khepri 0.7.0

At this point, Khepri should be considered Beta and not production ready. The API will likely evolve and future releases will likely introduce breaking changes. The goal of this release is to make it easy for anyone to try it and possibly give feedback.

What's new in Khepri 0.7.0

This release focuses on several important bug fixes and improvements to projections.

We promoted this release from Alpha to Beta as we believe the API is a lot more stable now. We are not ready to commit to this API yet though and may introduce more breaking changes if the need arises.

Many improvements to projections

Projections are a new mechanism introduced in Khepri 0.6.0 to allow fast queries with a low latency.

Here are five pull requests related to projects:

• The deletion of an entire subtree is reflected in projections (#178).
• Projections' ETS tables are correctly recreated when a Ra server is restarted and enters the recovered state (#182, #193)).
• Query speed improvements (#188).
• Reduce memory footprint (#190).

Creation of the Horus library

The previous khepri_fun module was extracted from Khepri and the Horus library was created out of it. It makes this code easier to maintain and test, and it can be reused outside of Khepri.

See #192.

Split of the khepri_machine code into several modules

The khepri_machine module was really large because it hosted the entire implementation of the internal tree data structure. This module was refactored and split into multiple modules to improve the readability of the code and its maintenance.

See #180, #181, #185, #186, #187, #189 and #194.

Other changes

• Documentation improvements (#173).
• Supervision tree fixes (#174).
• New khepri_cluster:is_store_running/1 helper (#175).
• Work around Rebar's use of cth_readable parse_transform (#176).
• Support for new Erlang/OTP 26 assembler instructions (#183).
• Ra was bumped from 2.4.0 to 2.5.1 (#191, #199).

Download

Khepri 0.7.0 is available from Hex.pm: https://hex.pm/packages/khepri/0.7.0

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.7.0"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.7.0

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.7.0 is available on Hex.pm.

Khepri 0.6.0

At this point, Khepri should be considered Alpha and not production ready. The API will likely evolve and future releases will likely introduce breaking changes. The goal of this release is to make it easy for anyone to try it and possibly give feedback.

What's new in Khepri 0.6.0

This release focuses on several important new features.

Very fast queries thanks to "projections"

Projections are a new mechanism to allow fast queries with a low latency.

Projections use ETS tables to work as a cache. The user is in control of the way data is stored in these ETS tables:

1. He selects what is to be cached using path patterns.
2. He provides an anonymous function to transform the data payload to a record ready to be stored in ETS.

This obviously comes at a cost. The cache being implemented as ETS tables means data duplication between the store itself and the ETS tables and an increase in memory footprint. Also, queries are therefore eventually consistent (because of the time it may take to reflect a store change in the cache).

Here is a copy of the exemple in the pull request:

1. You need to declare the projection first:

   %% `ProjectionName' will be the name of the ETS table.
   ProjectionName = wood_stocks,
   
   %% `ProjectionFun' defines how a data payload is transformed into an ETS
   %% record.
   ProjectionFun = fun([stock, wood, Kind], Stock) -> {Kind, Stock} end,
   
   %% This line creates an opaque term from the variables above.
   Options = #{type => set, read_concurrency => true},
   Projection = khepri_projection:new(ProjectionName, ProjectionFun, Options),
   
   %% You can finally associate the created projection with a path pattern.
   StoreId = stock,
   PathPattern = "/:stock/:wood/*",
   RegisterOptions = #{},
   khepri:register_projection(StoreId, PathPattern, Projection, RegisterOptions).

   The ETS table named wood_stocks will be created and filled with the existing tree nodes matching the path pattern, after they are transformed using the given anonymous function.

2. You can start to query the ETS table for cached values:

   %% Imagine you write a new value in the store, whether it is before or
   %% after creating the projection.
   khepri:put(StoreId, "/:stock/:wood/Oak", 100),
   
   %% After the projection was defined, you can query the ETS table:
   100 = ets:lookup_element(ProjectionName, <<"Oak">>, 2).

See #135 for more information.

Import and export of a store

This release introduces an import/export mechanism.

This feature relies on import/export backend modules. These modules follow the same API as the "backup and restore" Mnesia API, therefore it is possible to re-use existing backup modules for Mnesia.

Unlike Mnesia however, it is possible to export a part of the store by specifying a path pattern to select tree nodes to export.

During import, everything is written to the store. Note that other existing tree nodes are left untouched: the store is not emptied before an import.

Khepri 0.6.0 comes with an import/export backend module called khepri_export_erlang which uses a plaintext file where tree nodes are formatted as Erlang terms. The benefit is it doesn't add any dependencies to the library.

Here is an example:

1. Export the wood stocks from an existing store:

   ok = khepri:export(
      StoreId,
      "/:stock/:wood/**",
      khepri_export_erlang,
      "export-wood-stock-1.erl").

   export-wood-stock-1.erl contains:

   {put, [stock, wood, <<"Oak">>], {p_data, 100}, #{}}.
   {put, [stock, wood, <<"Maple">>], {p_data, 230}, #{}}.

2. Import the wood stocks back:

   ok = khepri:export(
      StoreId,
      khepri_export_erlang,
      "export-wood-stock-1.erl").

See #157 for more information.

New maps:fold/3-like functions

To make it easier to work on many tree nodes, Khepri 0.6.0 brings the common iterative functions we are used to use in lists and maps modules:

• khepri:fold/5
• khepri:foreach/4
• khepri:map/4
• khepri:filter/4

They will all loop on tree nodes matching the given path pattern and run the given anonymous function for each one of them. Except for khepri:foreach/4, the return value of this anonymous function will be used to build a new Erlang term return by the khepri:fold/5 and friends.

Example:

Fun = fun
          (Path, #{data := Stock}, Acc) ->
              Wood = lists:last(Path),
              Acc#{Wood => Stock};
          (_Path, _NodeProps, Acc) ->
              Acc
      end,

#{<<"Oak">> := 100},
 {<<"Maple">> := 230}} = khepri:fold(StoreId, "/:stock/:wood/*", Fun, #{}).

See #163 for more information.

Stored procedures as transaction functions

Transactions in Khepri can be expensive in terms of CPU and memory because of the need to extract the given anonymous function, verify what it does and create a standalone module out of it to be executed on all Ra nodes.

To "pay" this extraction cost once, it is now possible to use stored procedures as transaction functions. In other words, a caller needs to:

1. store the transaction anonymous function once in the store
2. call khepri:transaction/2 with the path to that stored procedure instead of passing the anonymous function directly

To make transactions more flexible and allow the use of more generic stored procedures, transaction functions can now take arguments.

See #150 for more information.

Other changes

• Automatically reclaim useless tree nodes which were automatically created (#155).
• Support empty binaries in string-based paths (#156).
• Fix handling of khepri:delete_many("**") (#158).
• Rename delete_payload() back to clear_payload().
• Fix application definition when building Khepri with mix(1) (#146).
• Add module_info/{0,1} functions to extracted stored procedures and transaction functions, as required by debugging tools (#148).
• Fix several issues around function extraction: #149, #161, #165, #168, #169, #170.
• Acknowledge trigger executions asynchronously to fix a race with store shutdown (#151).
• Fix regression with the use of #if_has_data{} with node props (#154).
• Implement ra_machine:overview/1 for khepri_machine (#159).
• Reduce the size of the "put" Ra commands by merging two maps (#166).
• Accept machine_config in a Ra server config map (#167).

Download

Khepri 0.6.0 is available from Hex.pm: https://hex.pm/packages/khepri/0.6.0

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.6.0"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.6.0

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.6.0 is available on Hex.pm.

Khepri 0.5.0

At this point, Khepri should be considered Alpha and not production ready. The API will likely evolve and future releases will likely introduce breaking changes. The goal of this release is to make it easy for anyone to try it and possibly give feedback.

What's new in Khepri 0.5.0

Rework the public API again

In Khepri 0.3.0, the public API already saw a lot of changes to make it more straightforward. We were still unhappy with the result and this release is another attempt at providing a nice API. The goals are:

1. to have a simple API with no surprises when you read a program using Khepri
2. to keep the ability to handle more advanced use cases

Now, the khepri module exposes the simple API mentionned above, while a new module, khepri_adv covers the advanced use cases. Likewise inside transactions with khepri_tx and khepri_adv.

Here is an example of the changed API:

• Up-to Khepri 0.4.x:

  %% Store and get a value:
  {ok, _} = khepri:put("/:stock/:wood/Oak", 120),
  {ok, #{[stock, wood, <<"Oak">>] := #{data := 120}}} = khepri:get("/:stock/:wood/Oak"),
  
  %% Get the entire stock:
  {ok, #{[stock, wood, <<"Oak">>] := #{data := 120},
         [stock, wood, <<"Maple">>] := #{data := 45}}} = khepri:get_many("/:stock/:wood/*"),
  
  %% Update the stock after a fire in the warehouse...
  %% -> Use a transaction going through all elements.
  
  %% Or:
  {ok, _} = khepri:delete("/:stock/:wood").

• Starting from Khepri 0.5.0:

  %% Store and get a value:
  ok = khepri:put("/:stock/:wood/Oak", 120),
  {ok, 120} = khepri:get("/:stock/:wood/Oak"),
  
  %% Get the entire stock:
  {ok, #{[stock, wood, <<"Oak">>] := 120,
         [stock, wood, <<"Maple">>] := 45}} = khepri:get_many("/:stock/:wood/*"),
  
  %% Update the stock after a fire in the warehouse...
  ok = khepri:put_many("/:stock/:wood/*", 0),
  
  %% Or:
  ok = khepri:delete("/:stock/:wood").

Everything is described in greater details in pull request #145.

Errors from khepri:exists(), khepri:has_data() and khepri:is_sproc()

The functions khepri:exists(), khepri:has_data() and khepri:is_sproc() are checking a particular property of the targeted tree node. They used to return a boolean(). However, when there was an error in the process, like an issue with the network, the errors were ignored and false was returned.

This is not the truth and the functions can't know. Therefore, they now return that error.

See #126.

Other changes

• Handle nodedown when handling queries (#125).
• Several bug fixes around function extraction: #130, #131, 134, #138 and #139.
• Fix a bug with triggers on grandchildren (#137).
• Improve documentation (#128 and #133).
• Drop support for Erlang 23 (#132).

Download

Khepri 0.5.0 is available from Hex.pm: https://hex.pm/packages/khepri/0.5.0

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.5.0"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.5.0

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.5.0 is available on Hex.pm.

Khepri 0.4.3

At this point, Khepri should be considered Alpha and not production ready. The API will likely evolve and future releases will likely introduce breaking changes. The goal of this release is to make it easy for anyone to try it and possibly give feedback.

Bugfixes in Khepri 0.4.3

• Allow badrecord instructions (#120).
• Work around several badly decoded instructions from beam_disasm (#119, #121, #122).

Download

Khepri 0.4.3 is available from Hex.pm: https://hex.pm/packages/khepri/0.4.3

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.4.3"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.4.3

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.4.3 is available on Hex.pm.

Khepri 0.4.2

At this point, Khepri should be considered Alpha and not production ready. The API will likely evolve and future releases will likely introduce breaking changes. The goal of this release is to make it easy for anyone to try it and possibly give feedback.

Bugfixes in Khepri 0.4.2

• Fix a possible deadlock when a transaction is executed during the start or stop of Erlang applications (#117).

Download

Khepri 0.4.2 is available from Hex.pm: https://hex.pm/packages/khepri/0.4.2

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.4.2"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.4.2

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.4.2 is available on Hex.pm.

Khepri 0.4.1

At this point, Khepri should be considered Alpha and not production ready. The API will likely evolve and future releases will likely introduce breaking changes. The goal of this release is to make it easy for anyone to try it and possibly give feedback.

Bugfixes in Khepri 0.4.1

• Fix a crash when a Khepri store joins a cluster after being restarted, i.e. was created by a previous call to khepri:start() (#113).
• Update Ra from 2.0.13 to 2.1.0; this fixes a race condition in Ra which affects clustered Khepri stores when one of them is stopped (#115).

Other changes

• Improve cluster_SUITE integration testsuite (#114, #116).

Download

Khepri 0.4.1 is available from Hex.pm: https://hex.pm/packages/khepri/0.4.1

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.4.1"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.4.1

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.4.1 is available on Hex.pm.