Skip to content

Commit

Permalink
Remove QLC from member & guild caches
Browse files Browse the repository at this point in the history
See #625 and #629 for further information.
  • Loading branch information
jchristgit committed Aug 18, 2024
1 parent 4179b3a commit 387ee43
Show file tree
Hide file tree
Showing 12 changed files with 223 additions and 494 deletions.
161 changes: 0 additions & 161 deletions benchmarks/qlc_bench.exs

This file was deleted.

91 changes: 0 additions & 91 deletions guides/cheat-sheets/qlc.cheatmd

This file was deleted.

98 changes: 7 additions & 91 deletions guides/functionality/state.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,101 +13,17 @@ free to suggest it [on GitHub](https://github.com/Kraigie/nostrum/issues).

Should the default ETS-based caching not be enough for you - for instance, you
want to integrate to some external caching mechanism or want to distribute your
bot across multiple nodes, please see the [pluggable
bot across multiple nodes, and the built-in Mnesia-based caching is not enough
for you either, please see the [pluggable
caching](../advanced/pluggable_caching.md) documentation.


## Query list comprehensions

nostrum's built-in functions to query the cache should be sufficient to cover
common use cases. If you need more involved queries, it is recommended to use
nostrum's [qlc](https://www.erlang.org/doc/man/qlc.html) support.

### Examples

Below you can find some example queries using QLC.

```erl
% src/nostrum_queries.erl

-module(nostrum_queries).
-export([find_role_users/4, find_large_communities/2]).

-include_lib("stdlib/include/qlc.hrl").

% Find the Nostrum.Struct.User and Member objects of all members in a specific guild role.
find_role_users(RequestedGuildId, RoleId, MemberCache, UserCache) ->
qlc:q([{User, Member} || {{GuildId, MemberId}, Member} <- MemberCache:query_handle(),
% Filter to member objects of the selected guild
GuildId =:= RequestedGuildId,
% Filter to members of the provided role
lists:member(RoleId, map_get(roles, Member)),
% Get a handle on the UserCache table
{UserId, User} <- UserCache:query_handle(),
% Find the User struct that matches the found Member
MemberId =:= UserId]).

% Find all communities in the Guild cache with the COMMUNITY guild feature
% that are over a certain threshold in user size
find_large_communities(Threshold, GuildCache) ->
qlc:q([Guild || {_, Guild} <- GuildCache:query_handle(),
% Filter for guilds that are over the provided size
map_get(member_count, Guild) > Threshold,
% Filter for guilds that have COMMUNITY in the features field
lists:member(<<"COMMUNITY">>, map_get(features, Guild))]).
```

`nostrum_queries:find_role_users/4` fetches all users in the specified guild
(`RequestedGuildId`) with the role `RoleId`. The code is annotated, but
step-by-step the flow is: the member cache is filtered down to all members in
the guild, then using `lists:member/2` we check for role membership, and finally
we join against the user cache to return full user objects in the result.

`nostrum_queries:find_large_communities/2` fetches all guilds in the guild cache
that meet the criteria of having at least `Threshold` members *and* having the
`COMMUNITY` guild feature set to true in the Discord UI. It is easy to follow
the flow of this query by the annotations, with only some minor things to note
such as needing to use `<<"bitstring">>` bit syntax to represent the strings,
which is implicit in Elixir.

In Elixir, you can call these queries like so using `:qlc.eval/1`:

```elixir
matching_guilds = :qlc.eval(:nostrum_queries.find_large_communities(50, Nostrum.Cache.GuildCache))
```

### Implementing your own queries and caches

By [implementing a QLC
table](https://www.erlang.org/doc/man/qlc.html#implementing_a_qlc_table), all
read operations from nostrum will be performed over your QLC table
implementation alone, and nostrum's dispatcher modules can easily be expanded
for more queries in the future. If you've never heard of QLC before, the
[`beam-lazy` repository](https://github.com/savonarola/beam-lazy) contains a
good introduction.

Using QLC bring a plethora of benefits. Implementation of a QLC table is
relatively simple, and gives us compile-time query optimization and compilation
in native Erlang list comprehension syntax. Furthermore, should you wish to
perform queries on your caches beyond what nostrum offers out of the box, you
can write your queries using the `query_handle/0` functions on our caches,
without having to investigate their exact API.

There is one caveat to be aware of when writing cache adapters in Elixir that
build on this functionality: While Erlang's QLC can perform intelligent query
optimization, a lot of it is implemented via a parse transform and thus only
available at compile time in Erlang modules. It is therefore recommended to
write your QLC queries in Erlang modules: in Mix projects this can be achieved
easily via the `src/` directory. Read the [QLC module
documentation](https://www.erlang.org/doc/man/qlc.html) for more details on the
optimizations done.

The reason why QLC is being used as opposed to the Elixir-traditional stream API
is that the stream API does not support a number of features we are using here.
Apart from that, nostrum's previous API (`select` and friends) gave users a
false impression that nostrum was doing an efficient iteration under the hood,
which caused issues for large bots.
## Implementing your own caches

To implement custom caches, implement the behaviour defined by the cache
module, such as `Nostrum.Cache.GuildCache`. For ease of use, these modules
define both a user-facing API to obtain objects from the configured cache, as
well as the developer-facing behaviour description.


## Internal state
Expand Down
Loading

0 comments on commit 387ee43

Please sign in to comment.