New Rust re-write #27

Stridsvagn69420 · 2022-10-24T22:49:43Z

This will introduce the Rust re-write with v1.2.0, the managaement tools and obviously the hot new stuff as well as a re-write of the Wiki and better docs.

Closes [Core] Make the code base prettier or switch to new language entirely #20 both in terms of codebase and also in terms of documentation.
Resolves #17 and the easier management of metadata files and registered owners.

Checklist

And then obviously the server stuff:

Meta headers (license, source code, Server Version)
Meta redirects (e.g. source code)
CORS everywhere
Hostinfo generator
Hostinfo and file serving routes with optional Basic Auth (directory listing will be simple <a href="filename.mp3">filename.mp3</a> stacks)

In the end, both cyrkensia and cyrkensia-cfg binaries should be finished in order to merge this PR!

cyrkensia - the Cyrkensia server
cyrkensia-cfg - tool for managing Cyrkensia-related files and folders

src/artist.rs

Stridsvagn69420 · 2022-11-01T02:49:54Z

Only the Server Routes and Root Docs needed now!

Stridsvagn69420 · 2022-11-01T02:50:46Z

...for the raw library of course, not the binaries. But still, pretty neat!

Stridsvagn69420 · 2022-11-12T18:46:52Z

Use actix-web shared state for the routes. Probably use a trait for it if possible.

Basically what's missing:

Shared state via Actix-Web (Hostinfo, Config, hostinfo timestamp, Htpasswd HashMap)
CORS everywhere (via middleware)
Password authentication (in file serving, via shared state)
The damn file serving routes
(NEW) Multiple root paths in config

Stridsvagn69420 · 2022-11-15T21:45:42Z

The Password Authentication thingy will be done in the 3rd point on the checklist.

Stridsvagn69420 · 2022-11-25T17:20:23Z

Only the file index and then actual file serving is missing. Thankfully these will be just two route functions, one for index, one for files.
Plus there are the index response (basically just formatting) and a 401 error response, similar to the sever_500() response.

Also, the file serving route should always include the size in the header, but also the SHA-256 and BLAKE3 hashes when there's a HEAD request (authentication has to be successful for this; if it isn't, always return the 401 response).

Stridsvagn69420 · 2022-11-25T18:29:15Z

Using BLAKE3 is a good idea ...for checksums though.
I'll switch the password hashing to argon2 asap. BLAKE3 will still be used to create the file checksums, and SHA-256 will be removed.

Stridsvagn69420 · 2022-12-03T01:15:28Z

Apart from bundling everything together as an executable and testing everything, only the file serving with the HEAD meta and optional HTTP Basic Auth is missing!

Stridsvagn69420 · 2022-12-03T01:18:05Z

Aw shit, I made an "it's 2 AM and I'm sick and tired"-oopsies

It should read from the config's roots and check where that folder is.

Stridsvagn69420 · 2022-12-03T13:59:39Z

It's only the actual fileserving-basicauth-headmeta route thingy now.

Stridsvagn69420 · 2022-12-04T13:42:47Z

I should probably also remake the wiki before this gets merged...

Stridsvagn69420 · 2022-12-04T22:43:15Z

Header stuff for the file serving:

ETag and Digest headers will contain BLAKE3 hash
Last-Modified header will use file metadata for time
Implement If-Modified-Since header behavior (client can send request with date to check if new data has to be downloaded)
And HEAD request will only be exclusive to the fact that you can get the file size and the other metadata without actually dealing with megabytes or even gigabytes of wasted data. Plus, I want the HTTP codes to align with that's actually happening. So 200 for everything normal and 304 for when there's nothing new.

I have the feeling Actix-Web already does a lot of things for you, but I want to be sure and implement it my way, plus only something like the Actix-Web-Static-Fileserver thingy can really do things like hashing, ETag and Last-Modified, plus there's the user authentication (without auth, it won't send any information).
I originally also wanted to cache all of these things, but BLAKE3 is blazingly fast and the other caching mechanics get near the level of what NGINX and Apache can kindly do for you.

src/server/routes.rs

Basic Auth WIP. CLI tool wip, server message wip, logger wip.

src/server/mod.rs

+ 	let mut range = binding.split('-');
+
+	// Parse range
+	if let (Some(x), Some(y)) = (range.nth(0), range.nth(0)) {


src/server/mod.rs

+ 	let mut range = binding.split('-');
+
+	// Parse range
+	if let (Some(x), Some(y)) = (range.nth(0), range.nth(0)) {


Stridsvagn69420 · 2023-01-24T19:10:31Z

Random stuff, but make the digest and partial content thing a middleware (has to be combined because digest depends on content).

Stridsvagn69420 · 2023-03-10T13:13:38Z

Honestly, I might just make a new Rust rewrite.
I wrote it down on the Nucleus server, so time to post it here too.

was messing around with GTA's Self Radio, FFmpeg, audio codecs and tags in audii files, and I might make a rust-new-version of my WIP rust-version of Cyrkensia

you see, aside from having to redo most things because mixing basic file serving, sending metadata and password protecting everything together being complicated, I might do the storing part different

I'll probably use SQL and will give the option to use SQLite or RDBMS. There will still be a config file in Json, but it's very simple.
The database will have tables for:
-maintainers and server admins
-user database (Auth can be toggled in settings)
-soundtrack table (lots of metadata and the filesystem info, maybe also hashes for integrity and stuff)
I'll keep the current updating system for the soundtrack table btw, but the idea is that instead of having 300 MB of hostinfo.json, there will be a REST api endpoint for getting the file metadata and in general to know what the server even offers
and 3. (I forgot this because slewpy)
there should be a little tool that updates the files' tags baaed on the database

I have a more elaborate version on this now.
So, I'm gonna make it based on the current architecture, BUT the hostinfo won't be so fucking huge anymore. It merely lists the audio files, maybe even just the groups (what I called albums), and of course the server meta.
And then there's another endpoint, the songinfo, that allows the client to read more data about a file. And this part just reads the available data from the database (probably SQL then). The database can be manually filled, but if you already have metadata on your file, another tool tries to write it to the database on command. Idk if I will include caching, probably yes if I figure out how.

So things will get debloated and the architecture will get better too. I just have to find time when I can actually do it, school and stuff you know.

Stridsvagn69420 · 2023-03-10T13:15:50Z

Auth and Content hashing meta thingy will most likely be a middleware btw.

Stridsvagn69420 · 2023-04-16T01:49:27Z

Gonna rewrite the entire thing again...

Cargo init

758a6e5

Stridsvagn69420 added bug Something isn't working documentation Improvements or additions to documentation enhancement New feature or request help wanted Extra attention is needed dependencies Pull requests that update a dependency file labels Oct 24, 2022

Stridsvagn69420 self-assigned this Oct 24, 2022

Stridsvagn69420 added 2 commits October 25, 2022 01:16

Lib files WIP. Will add private submodule for the server.

610fa1b

More basic files and a lot of TODOs

174cb79

Stridsvagn69420 force-pushed the rust-variant branch from 9d416f7 to 174cb79 Compare October 25, 2022 10:51

Stridsvagn69420 added 2 commits October 25, 2022 14:04

Structs added. Trait impls wip. Main library docs wip.

1032fcf

Workflows and README.md

4bdf5ec

github-advanced-security bot found potential problems Oct 26, 2022

View reviewed changes

src/artist.rs Fixed Show fixed Hide fixed

Checklist #2 to #5 done

739246b

Stridsvagn69420 force-pushed the rust-variant branch from 87a677c to 739246b Compare October 26, 2022 08:33

Stridsvagn69420 added 3 commits October 27, 2022 01:52

Owner and Metadata finished

54b2020

Config done basically

31ab7d9

Hostinfo done, including generation!!

dfae17b

Stridsvagn69420 added 4 commits November 1, 2022 16:56

Cleaner Hostinfo code and docs

a4b1a40

Middleware and redirects

faf564e

Lib part basically done. Routes can only be set in the binary itself.

7fd6412

Rust 1.65 and let-else is awesome

31461bc

Stridsvagn69420 added 2 commits November 14, 2022 00:09

Add shared state for Actix-Web routes

c08a02b

Add multi-root option

a6fdf19

Added hostinfo route and route utils

4bafa1e

Stridsvagn69420 force-pushed the rust-variant branch from c6e4b58 to 4bafa1e Compare November 15, 2022 21:48

Add CORS everywhere middleware

307473a

Stridsvagn69420 added 3 commits November 25, 2022 20:10

Switch password hashing to Argon2

5cb32cd

Small improvements to keep this alive

c16c6c2

Album indexing route done

ec29995

Album finder done and route updated

e89b77f

Stridsvagn69420 added 2 commits December 4, 2022 04:24

Structural tuning and server binary

b49ab10

Small fixes and better code

90b5a12

Stridsvagn69420 force-pushed the rust-variant branch from f29e138 to 90b5a12 Compare December 4, 2022 04:18

Timestamp logging and more macros

1ceca0a

Artists live in central artists.json file now

b79cfa3

Stridsvagn69420 force-pushed the rust-variant branch from 8feaba4 to b79cfa3 Compare December 4, 2022 16:52

"Verbose error output"

036e323

Stridsvagn69420 force-pushed the rust-variant branch from 420fe0d to 036e323 Compare December 4, 2022 18:56

CLI tool wip, server message wip, logger wip, file routes wip

e1da6b5

github-advanced-security bot found potential problems Dec 28, 2022

View reviewed changes

src/server/routes.rs Fixed Show fixed Hide fixed

src/server/routes.rs Fixed Show fixed Hide fixed

src/server/routes.rs Fixed Show fixed Hide fixed

Stridsvagn69420 added 2 commits December 29, 2022 15:54

CLI tool wip, server message wip, logger wip. file routes DONE

0dcabf0

Partial content doesn't work. Time to revert.

669d0dc

Basic Auth WIP. CLI tool wip, server message wip, logger wip.

github-advanced-security bot found potential problems Dec 29, 2022

View reviewed changes

Stridsvagn69420 closed this Apr 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New Rust re-write #27

New Rust re-write #27

Stridsvagn69420 commented Oct 24, 2022 •

edited

Loading

Stridsvagn69420 commented Nov 1, 2022

Stridsvagn69420 commented Nov 1, 2022

Stridsvagn69420 commented Nov 12, 2022 •

edited

Loading

Stridsvagn69420 commented Nov 15, 2022

Stridsvagn69420 commented Nov 25, 2022

Stridsvagn69420 commented Nov 25, 2022

Stridsvagn69420 commented Dec 3, 2022

Stridsvagn69420 commented Dec 3, 2022 •

edited

Loading

Stridsvagn69420 commented Dec 3, 2022

Stridsvagn69420 commented Dec 4, 2022

Stridsvagn69420 commented Dec 4, 2022

Stridsvagn69420 commented Jan 24, 2023

Stridsvagn69420 commented Mar 10, 2023

Stridsvagn69420 commented Mar 10, 2023

Stridsvagn69420 commented Apr 16, 2023

New Rust re-write #27

New Rust re-write #27

Conversation

Stridsvagn69420 commented Oct 24, 2022 • edited Loading

Checklist

Stridsvagn69420 commented Nov 1, 2022

Stridsvagn69420 commented Nov 1, 2022

Stridsvagn69420 commented Nov 12, 2022 • edited Loading

Stridsvagn69420 commented Nov 15, 2022

Stridsvagn69420 commented Nov 25, 2022

Stridsvagn69420 commented Nov 25, 2022

Stridsvagn69420 commented Dec 3, 2022

Stridsvagn69420 commented Dec 3, 2022 • edited Loading

Stridsvagn69420 commented Dec 3, 2022

Stridsvagn69420 commented Dec 4, 2022

Stridsvagn69420 commented Dec 4, 2022

Stridsvagn69420 commented Jan 24, 2023

Stridsvagn69420 commented Mar 10, 2023

Stridsvagn69420 commented Mar 10, 2023

Stridsvagn69420 commented Apr 16, 2023

Stridsvagn69420 commented Oct 24, 2022 •

edited

Loading

Stridsvagn69420 commented Nov 12, 2022 •

edited

Loading

Stridsvagn69420 commented Dec 3, 2022 •

edited

Loading