Question about Clover's query model

[fbilhaut]

Hi Stefano,

I had a look on your implementation because I'm experiencing performance problems even on a subset of my data.

I saw that when evaluating a query the process consists in listing all documents and checking if the constraints are satisfied. At least it seems to be the case in the default storage implementation:

func (s *storageImpl) FindAll(q *Query) ([]*Document, error) {
	docs := make([]*Document, 0)
	err := s.IterateDocs(q, func(doc *Document) error {
		if q.satisfy(doc) {
			docs = append(docs, doc)
		}
		return nil
	})
	return docs, err
}

I understand the constraints given by a straightforward use of the underlying key-value store, and that Clover aims to remain simple and easily maintainable.

However, this will not be sufficiently efficient for some volume and time constraints. For example it is not on my use cases starting from a few hundred thousands docs.

So my questions is : do you intend to implement some kind of indexing in the default storage implementation (on specified fields for example), or is it something you consider out of scope of your project ?

[ostafen]

Hi, @fbilhaut, index support is definitely something which will be added in the future.

As a workaround, for now, I could suggest you to partition your documents across multiple collections based on some partition key

[fbilhaut]

Great news, thanks for your quick answer.

As for workarounds, for my use case partitioning is not practical. However if it was possible to specify client-generated IDs for documents (freely formatted strings) instead of generated UUIDS, it would allow to leverage at least the efficiency of the underlying key-value store.

[ostafen]

I think this can be easily done. You can open a new issue for this if you want

[ostafen]

Hi, @fbilhaut, I'm currently implementing indexes in the v2 branch. Would you like to do some tests and give me some feedback about performance?

[ostafen]

it's very simple. If you want to create an index, you have to just use the method:

db.CreateIndex(collectionName, fieldName)

Currently, the index will be used only if your criteria doesn't contain OR conditions

[fbilhaut]

Thanks Stefano,

Tested it on my data and very simple queries (just an exact match on the indexed field, for now), and for that matter it works flawlessly. Don't have the numbers, but needless to say that for that simple use case, switching from linear search to indexed access, it's something like 1000x faster :-)

Just a few remarks / questions:

• Do you intend to merge Resolve #64 (Allow to discover available fields for a document) #65 into the v2 branch ? Would be convenient for me to do further testing.
• Could you expose the HasIndex() method at DB level ?
• Could you clarify a bit on the index lifecycle ? I mean does it have to be created before inserts, etc.

Thanks a lot !

[ostafen]

Do you intend to merge #65 into the v2 branch ? Would be convenient for me to do further testing.

Yes, I will merge it soon.

Could you expose the HasIndex() method at DB level ?

Of course!

Could you clarify a bit on the index lifecycle ? I mean does it have to be created before inserts, etc.

You can create the index at any moment of the db's lifecycle. When you call the CreateIndex() method, your collection's documents will be inserted into the index. Since then, the index will be automatically kept updated on every subsequent operation (insert, update, delete). However, if you have an high volume of data, it's better to create it before inserting any document, because otherwise index creation could take too much.

Note that this is still in development, so this could take some weeks before reaching a stable version. In every case, feel free to submit your feedback

[fbilhaut]

Great, thanks !

[ostafen]

@fbilhaut, sorry for the delay, I merged your changes into the v2 branch. Since last time, it also contains many important speed-ups