Filer Stores

Comparing Storage Options

The Filer Store persists all file metadata and directory information.

Filer Store Name	Lookup	number of entries in a folder	Scalability	Directory Renaming	TTL	Fast Bucket Deletion	Note
memory	O(1)	limited by memory	Local, Fast		Yes		for testing only, no persistent storage
leveldb	O(logN)	unlimited	Local, Very Fast		Yes		Default, fairly scalable
leveldb2	O(logN)	unlimited	Local, Very Fast		Yes		Similar to leveldb, part of the lookup key is 128bit MD5 instead of the long full file path
leveldb3	O(logN)	unlimited	Local, Very Fast		Yes	Yes	Similar to leveldb2, separate leveldb instance for each bucket
RocksDB	O(logN)	unlimited	Local, Very Fast		Native		Default, fairly scalable
Sqlite	O(logN)	unlimited	Local, Very Fast	Atomic	Yes	Yes	Default, fairly scalable, Stream backup
Mongodb	O(logN)	unlimited	Local or Distributed, Fast		Yes		Easy to manage
Redis2	O(1)	limited	Local or Distributed, Fastest		Native		one directory's children are stored in one key~value entry
Redis3	O(1)	unlimited	Local or Distributed, Fastest		Native		one directory's children are spread into multiple key~value entries
Cassandra	O(logN)	unlimited	Local or Distributed, Very Fast		Native
MySql	O(logN)	unlimited	Local or Distributed, Fast	Atomic	Yes		Easy to manage
MySql2	O(logN)	unlimited	Local or Distributed, Fast	Atomic	Yes	Yes	Easy to manage
Postgres	O(logN)	unlimited	Local or Distributed, Fast	Atomic	Yes		Easy to manage
Postgres2	O(logN)	unlimited	Local or Distributed, Fast	Atomic	Yes	Yes	Easy to manage
MemSql	O(logN)	unlimited	Distributed, Fast	Atomic	Yes	Yes	Scalable
TiDB	O(logN)	unlimited	Distributed, Fast	Atomic	Yes	Yes	Scalable
TiKV	O(logN)	unlimited	Distributed, Fast	Atomic	Yes	Yes	Scalable
CockroachDB	O(logN)	unlimited	Distributed, Fast	Atomic	Yes	Yes	Scalable
YugabyteDB	O(logN)	unlimited	Distributed, Fast	Atomic	Yes	Yes	Scalable
Etcd	O(logN)	unlimited	Distributed, 10,000 writes/sec		Yes		No SPOF. High Availability. Limited Capacity.
ElasticSearch	O(logN)	unlimited	Distributed, Fast		Yes		Scalable, Searchable
HBase	O(logN)	unlimited	Distributed, Fast		Native		Scalable

Switching between different Stores

It is easy to switch between different filer stores.

For example:

# first save current filer meta data

$ weed shell
> fs.cd /
> fs.meta.save
...
total 65 directories, 292 files
meta data for http://localhost:8888/ is saved to localhost-8888-20190417-005421.meta
> exit

# now switch to a new filer, and load the previously saved metadata
$ weed shell
> fs.meta.load localhost-8888-20190417-005421.meta
...
total 65 directories, 292 files
localhost-8888-20190417-005421.meta is loaded to http://localhost:8888/

Extending Storage Options

For any new storage option, please implement the FilerStore interface. It should be fairly straight forward to implement. Welcome to contribute back.