Skip to content

Commit

Permalink
Add exist tools compare
Browse files Browse the repository at this point in the history
  • Loading branch information
unkcpz committed Nov 13, 2024
1 parent 76d9439 commit 633eff3
Showing 1 changed file with 15 additions and 0 deletions.
15 changes: 15 additions & 0 deletions 010_replace_rabbitmq/readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,21 @@ However, it is not trivial to implement all the communication needed from handsh
The `kiwipy` in AiiDA servers as the role to provide the wrappered methods to talk between different components over RabbitMQ.
Therefore, `kiwipy` is the interface for actioner and worker on bundle the operations to talk to RMQ, as the replacement, in `tatzelwurm` I should provide the python interface with methods ready to be used for sending and consuming certain type of MessagePack meesage.

## Other solution or exist tools?

It is discussed in https://github.com/aiidateam/AEP/pull/30#discussion_r813895745, better to look for tools that ready to use.
What mentioned are [Apache Kafka](https://kafka.apache.org/), [faust-streaming](https://github.com/faust-streaming), [MQTT](https://mqtt.org/) and [Celery](https://docs.celeryq.dev/en/stable/getting-started/introduction.html).

- Apach Kafka: for event streaming, require service to start. (servirce required)
- MQTT: is a messaging protocol in low bandwidth environments. (no queue system + overkill for low bandwidth)
- faust-streaming: require Kafka as event streaming backend. (Kafka services required, it is like kiwipy but for Kafka)
- Celery: this is a task queue. (For event streaming it requires a message broker, RMQ, Redis, SQS .., For persistent, it requires a results store.)

The `tatzelwurm` is tailored for AiiDA use cases specifically with message streaming/broke, lightwight queue system and persistent with KV store in disk in design.
It is lightweight but will meet all the requirements for AiiDA.
The tools above are for machines to machines scenario which required for huge message passing around millios of microservices and that is why those are overkill.
We are not targeting to that but for AiiDA with required architecture as described above.

## Design details

In this section, I futher extend on the details of the new architecture.
Expand Down

0 comments on commit 633eff3

Please sign in to comment.