stats endpoint

[arrowwood49]

I'm running Akumuli in a test environment. It is running in a docker with number of volumes as 6 and volume size as 500 MB:
sudo docker run -d --memory="300m" -p 8181:8181 -p 8282:8282 -p 8383:8383 -p 4242:4242 -v /tmp/akumuli:/Akumuli -e nvolumes=6 -e volume_size=500MB akumuli/akumuli:0.8.72-skylake

Used OpenTSDB protocol to ingest around 12 millions time series:
time cat tsdb_mock.txt > /dev/tcp/127.0.0.1/4242

I verified that data was loaded by running query. However, stats end point still shows 512MB of free space for all 6 volumes. Is there a different query to get current available or free space?

curl -s --url localhost:8181/api/stats

{
    "volume_0":
    {
        "free_space": "524288000",
        "file_name": "\/Akumuli\/db_0.vol"
    },
    "volume_1":
    {
        "free_space": "524288000",
        "file_name": "\/Akumuli\/db_1.vol"
    },
    "volume_2":
    {
        "free_space": "524288000",
        "file_name": "\/Akumuli\/db_2.vol"
    },
    "volume_3":
    {
        "free_space": "524288000",
        "file_name": "\/Akumuli\/db_3.vol"
    },
    "volume_4":
    {
        "free_space": "524288000",
        "file_name": "\/Akumuli\/db_4.vol"
    },
    "volume_5":
    {
        "free_space": "524288000",
        "file_name": "\/Akumuli\/db_5.vol"
    }
}

[Lazin]

Depending on your data it could be OK or not.
If you're writing many time-series and each individual series gets small number of datapoints no writes to the storage volumes will be made. Your data will be stored in memory and WAL. But WAL capacity is not reflected by the stats endpoint. You can restart akumulid and the /stats will be updated.

If the input file is ill-formed you won't see any error message while running this command time cat tsdb_mock.txt > /dev/tcp/127.0.0.1/4242. In case of error in the file OpenTSDB endpoint will log the error and close the connection. I believe that bash won't print any warning or error message in this case.

[arrowwood49]

I restarted docker container, stats endpoint now shows volume_0 free space changed.
Does this mean Akumuli used 175.5 MB (524288000 total space on volume_0 - 352784384 free space) of disk space to store the time-series?

Where is WAL stored? How do I check the disk space used by WAL? Can WAL size be configured?

{
    "volume_0": {
        "free_space": "352784384",
        "file_name": "/Akumuli/db_0.vol"
    },
    "volume_1": {
        "free_space": "524288000",
        "file_name": "/Akumuli/db_1.vol"
    },
    "volume_2": {
        "free_space": "524288000",
        "file_name": "/Akumuli/db_2.vol"
    },
    "volume_3": {
        "free_space": "524288000",
        "file_name": "/Akumuli/db_3.vol"
    },
    "volume_4": {
        "free_space": "524288000",
        "file_name": "/Akumuli/db_4.vol"
    },
    "volume_5": {
        "free_space": "524288000",
        "file_name": "/Akumuli/db_5.vol"
    }
}

This is what I see under /tmp/akumuli:

ls -lR /tmp/akumuli/
/tmp/akumuli/:
total 175380
-rw-r--r-- 1 root root 524288000 Aug  4 09:46 db_0.vol
-rw-r--r-- 1 root root 524288000 Aug  3 15:41 db_1.vol
-rw-r--r-- 1 root root 524288000 Aug  3 15:41 db_2.vol
-rw-r--r-- 1 root root 524288000 Aug  3 15:41 db_3.vol
-rw-r--r-- 1 root root 524288000 Aug  3 15:41 db_4.vol
-rw-r--r-- 1 root root 524288000 Aug  3 15:41 db_5.vol
-rw-r--r-- 1 root root   8085504 Aug  4 09:46 db.akumuli
drwxr-xr-x 2 root root        53 Aug  3 22:33 logs

/tmp/akumuli/logs:
total 4052
-rw-r--r-- 1 root root 4138135 Aug  4 09:47 akumuli.log
-rw-r--r-- 1 root root    5692 Aug  3 15:50 akumuli.log.2020-08-03

Looks like timeseries tags and other metadata are stored in db.akumuli which is a SQLite db. Will the metadata be auto deleted if the associated time-series expires and no new series for that object is added?

[Lazin]

WAL is configured in the config file. There is a WAL section in the automatically generated config and it has reasonalbe defaults. It also has a path to WAL storage. But normally you won't be able to locate any files if the database is stopped. You will be able to see the files if the database is running or crashed.

Will the metadata be auto deleted if the associated time-series expires and no new series for that object is added?
No. As for now it's not deleted.

[arrowwood49]

WAL is disabled by default. I'm using docker version "0.8.72-skylake". I looked at the config file ~/.akumulid docker but didn't see any config related to WAL.

This is from the log:

Command line: /usr/bin/akumulid


2020-08-05 17:21:33,888 main [INFO] WAL is disabled in configuration
2020-08-05 17:21:33,888 akumuli-storage [INFO] Open database at: /Akumuli/db.akumuli

[Lazin]

I see. The config in docker image is hardcoded and it doesn't have a wal section. I probably should enable it by default in docker. I'll look into that. You can try to run akumulid locally if you need to use WAL. Generally, it's preferable because Akumuli provides better persistency guarantees with WAL, also, it requires less RAM if you have a lot of empty stats.

[arrowwood49]

Thanks @Lazin, I will try to run akumulid locally with WAL settings.