Clojurians-log v2 dreams!

[oxalorg]

👋 Welcome clojurians!

This project has received funding for 3 months by Clojurists Together. Thanks to the amaznig Clojurists Together team and the awesome folks of the clojure community for their support 🥳 🌸

The wealth of knowledge being shared on the clojurians slack server is immense. Capturing, conserving, and making this discourse complete, easily accessible, and searchable should greatly benefit the community as a whole.

There is a LOT of potential features listed on the README page. But those are just my personal dreams. 💭

This discussion is about what dreams YOU (an awesome member of the clojure community) have!

I'm writing down a small action plan which I intend to start working on from 20th September 2021, any feedback / discussion / suggestions / rants are welcome. Let me know if I would be wasting my time on a feature no one would use 🐍 🐌

Action plan ⚡

This is the bare minimum I need to do get an MVP up and running. Like @plexus always says, "Always start with a walking skeleton", and that's what I intend to do first.

Walking Skeleton

The walking skeleton 💀🦴 would look something like this:

• I have purchased a Hetzner VPS (4gb ram)
• A bunch of ansible scripts to provision the server and deploy the code
  • setup user, fail2ban, firewall, postgres, java, clojure, git, caddy, SSL, etc.
  • run one-off commands?
  • run migrations
  • deploy & update the code / systemd / caddy services
• Ship it live on a dummy staging domain🌏
• [Possibly] GitHub actions to continually deploy changes from master
  • I am not sure how safe GITHUB secrects would be in a public repo as anyone who creates a PR could possibly print/echo them 🤔
  • Low priority, might skip this
• Setup sql migrations 🛰️
  • https://github.com/yogthos/migratus
  • or https://github.com/weavejester/ragtime ?

MVP / critical features

Once the walking skeleton is done, the plan is to work on these:

• slack imports from the beginning of time ⌛
  • I don't want to start off with a real-time archiving solution
  • first goal is to make clojurians-log-v2 idempotent to data imports (incremental / ad-hoc)
  • given a slack export, and a single repl command (import "path-to-slack-export-dir"), all data should be populated properly
  • this would give us huge flexibility in the future and should be really easy to run up a new instance independently at anytime
  • instance was down for a week? No problem, just export the weeks data from slack admin ui and import it. DONE!
  • this is the BIGGEST task (messages, users, channels, reactions, threads, replies etc)
• real time full text search for entire logs 🔍
  • I'm most excited to work on this, my first attempt is going to be using postgres' inbuilt FTS
  • 2nd choice would be MeiliSearch vs typsense
  • creating a frontend interface for this search (it would be wise to start off with a server rendered search result, maybe in the future we can do "type as you search")
• mobile responsive and a brand new UI + UX

Once this is done the MVP would be ready and we can discuss further from there! 😁 🙏

Additional

Once the above critical features are done, 2 very important features would still be pending to really get this off the ground!

• Full SEO compatibility with existing clojurians-log so that Google and others keep indexing it as is (if and when the switch happens)
• Implementing real-time socket API of slack to process events and log messages instantly 🚥
  • making sure this stays compatible and idempotent with our static data import / backfilling

Preliminary screenshot:

Feedback

Am I missing something really important or obvious? Please feel free to discuss and post your thoughts / feedback / suggestions / rant below, I am super excited to hear them! 😁

Also do not shy away from creating issues, or contributing! Issues and PR's are always welcome 🌸

If you're as excited about this as I am, please leave a star ⭐ ! It would greatly help in keeping me motivated 🌞