v1.25.0 - RAFT-based Schema, Batch Vectorization, Hybrid Search Improvements, Implicit Tenant Creation, Dynamic Index Switching
Breaking Changes
none
New Features
RAFT-based Schema
We're excited to announce the release of our RAFT-based schema! With this, Weaviate now supports concurrent schema updates, eliminating bottlenecks and significantly improving performance in large-scale and dynamic settings.
- Update schema manager interface to expose schema version (Part 1) by @reyreaud-l in #4659
- Update schema manager interface to expose schema version (Part 2) by @reyreaud-l in #4660
- Implement schema updates using RAFT consensus by @redouan-rhazouani in #3944
- Rebuild GQL just once when reloading local db by @redouan-rhazouani in #4670
- Update local_dev script RAFT_JOIN config to bootstrap cluster correctly by @moogacs in #4676
- Refactor consensus dev script on nodes name weaviate-* by @moogacs in #4677
- Update object interface to include version by @reyreaud-l in #4675
- RAFT store Apply tests calling mocks expectations assertion by @moogacs in #4684
- Save copy of the current RAFT schema into the old format by @moogacs in #4679
- Trigger schema callbacks on AddProperty only on success by @moogacs in #4685
- Check error on apply to avoid panics by @moogacs in #4687
- Add schema version to batch write client and server by @reyreaud-l in #4681
- Added store-specific statistics to existing RAFT statistics by @nathanwilk7 in #4689
- Client rpc reuse the same conn by @moogacs in #4686
- Update batch delete with schema version by @reyreaud-l in #4690
- Schema v2 props upsert by @aliszka in #4680
- Autoschema optimizations - cached class schema by @aliszka in #4662
- RAFT store.Service.Ready is flaky in cluster/store/store_test.go, poll for 2s instead of 1s wait to improve robustness by @nathanwilk7 in #4706
- Implement schema querying based on a specific version for enhanced version control by @redouan-rhazouani in #4693
- Update put, merge and add ref with schema version by @reyreaud-l in #4704
- Update UpdateShardStatus to use schema version by @reyreaud-l in #4713
- Fix nil ptr panic when closing RAFT leader client by @reyreaud-l in #4723
- RAFT GRPC: tweak the client service config by @moogacs in #4715
- Add query shard tenant and update usages by @reyreaud-l in #4727
- Enable cancelation for HTTP replication requests and reduce timeout for faster node failure detection by @redouan-rhazouani in #4730
- Schema v2 idempotent tenants by @aliszka in #4699
- Make batch operation retrieve and pass schema version to client by @reyreaud-l in #4738
- Update auto schema to return the schema version by @reyreaud-l in #4742
- Refactor usage of index.getOrInitLocalShard() to the minimum by @redouan-rhazouani in #4744
- Schema V2 writes with schema version by @aliszka in #4745
- Allow migration from RAFT to take as much time as necessary by @redouan-rhazouani in #4710
- Add missing CloudService to ClusterService rename by @aliszka in #4747
- Update RAFT subsystem to use the same log format by @redouan-rhazouani in #4746
- Convert
TenantExists
to handle eventual consistency by @tsmith023 in #4750 - Add an external gRPC method for getting tenant information by @tsmith023 in #4741
- update object endpoints to fetch and propagate schema version by @reyreaud-l in #4752
- Replace slog with logrus for logs consistency by @moogacs in #4753
- Add action logrus field to RAFT logs by @moogacs in #4755
- Autoschema fix to correctly return max schemaVersion by @aliszka in #4757
- Make default CLUSTER_HOSTNAME consistent between memberlist and RAFT by @moogacs in #4758
- Cluster log formatting by @nathanwilk7 in #4759
- Handle possible nil classes in autoSchema by @moogacs in #4761
- Refresh class's schema when props added by autoschema by @aliszka in #4762
- Specialize voter nodes for metadata storage by @redouan-rhazouani in #4734
- Update the schema version parsing to not always return 0 by @reyreaud-l in #4767
- Refactor leader error distinguish between election and network issues by @moogacs in #4784
- Update query shard tenant to get multiple shards at once by @moogacs in #4731
- Avoid schema updating race with DB update by @reyreaud-l in #4769
- Backup restore: return err class exists if it does by @moogacs in #4792
- Make coordinator node wait for schema version if changed on validate step by @reyreaud-l in #4794
- Add RAFT_GRPC_MESSAGE_MAX_SIZE Environment Variable to set maximum GRPC message size for RAFT by @nathanwilk7 in #4799
- Only involve leader for tenant status when necessary by @etiennedi in #4803
- Shift add tenant to the RAFT leader by @reyreaud-l in #4801
- Fix RAFT shutdown, force it for dependencies and convert to enterrors.GoWrapper by @moogacs in #4810
- Allow Leader QueryShardingState type by @moogacs in #4817
- Update replica usecase to wait for EC on writes with schema version by @reyreaud-l in #4814
- Schema v2 prevent shutdown by @aliszka in #4821
- Aggregate querying the leader for classes with variadic by @moogacs in #4787
- Observe durations of schema reads and writes for local and leader reads by @etiennedi in #4844
- Reserve RAFT (all casing permutations) as a class name by @moogacs in #4874
- Make bootstrap exit early on RAFT store reporting ready by @reyreaud-l in #4871
Batch Vectorization
Avoid vectorization API rate-limiting and enjoy faster insertion.
- Batch vectorization openai by @dirkkul in #4384
- Factor out batch vectorization and add Cohere support by @dirkkul in #4515
- Factor out batching in modules by @dirkkul in #4564
- Vectorizer batching for VoyageAI by @dirkkul in #4578
Dynamic Index Switching
Automatically switch vector index types to achieve peak performance and efficiency.
- Dynamic vector index type by @abdelr in #4350
- Fix merge conflict on schema validation of dynamic index by @trengrj in #4829
- Adding more tests around the dynamic index and fixing an existing bug by @abdelr in #4836
Implicit Tenant Creation
Create tenants on the fly by simply including the tenant name during batch inserts.
- Create nonexistent tenants in batch by default by @parkerduckworth in #3741
- Support auto-tenant toggling on MT-enabled classes by @parkerduckworth in #4813
Hybrid Search Improvements
- Add nearvector and neartext to hybrid search by @donomii in #4462
- Batch vectorization with custom rate limits by @dirkkul in #4546
- Add groupby to hybrid search and bm25f, add moveto/movefrom etc to aggregate hybrid search by @donomii in #4477
- Add tests, improve parameter extraction in hybrid aggregate by @donomii in #4809
Other
- Add endpoint for checking if tenant exists by @litlig in #4155
- Allow for the go profiler to be hosted on ports other than 6060 by @bennycortese in #4289
- Custom linter to report all files before exit by @moogacs in #4485
- Migrate compression helper to wrapper functions by @dirkkul in #4506
- Improve lazy shard loading initialization logging by @parkerduckworth in #4705
- Add cluster aware /cluster/statistics endpoint by @antas-marcin in #4789
- Adds the created tenants to the response of
POST /tenants
by @tsmith023 in #4802 - Add Tenant Activity "Metrics" by @etiennedi in #4739
- Increase put and batch timeout to 60 sec by @amourao in #4820
- use flag for consistency wait timeout and rename it by @reyreaud-l in #4864
- add data dir to be removed in restart_dev_environment script by @moogacs in #4867
- GRPC: Clean up target vectors in hybrid by @dirkkul in #4879
Module Improvements
- Refactor skipping module vectorization by @dirkkul in #4330
- Replace Generate with Chat for
generative-cohere
, Add Command R support by @CShorten in #4542 - Introduce text2vec-ollama module by @trengrj in #4595
- Introduce generative-ollama module by @CShorten in #4071
- Add Command R+ to
generative-cohere
by @CShorten in #4613 - Unify modules class settings by @dirkkul in #4581
- Add Llama3 to
generative-ollama
by @CShorten in #4717 - Adding OctoAI generative and text2vec modules by @tmoreau89 in #4771
- Remove model list in generative ollama module by @antas-marcin in #4847
- Change modelId setting to model in Ollama modules by @antas-marcin in #4870
Performance Improvements
- Lock-free DB::ratePerSecond counter by @arturmelanchyk in #4249
- SIMD: AVX256 int8 dot product by @asdine in #4566
- SIMD: add l2 byte AVX256 implementation by @asdine in #4596
- Add missing generic types on ARM64 by @asdine in #4606
- Refactor generic dot product function into two functions by @asdine in #4628
- Hamming ARM64 SIMD implementation by @ajit283 in #4630
- Hamming AMD64 SIMD implementation by @ajit283 in #4678
- SIMD: add SIMD SVE implementation of L2 distance function by @asdine in #4748
Testing Improvements
- Fix randomly failing hybrid test by @donomii in #4304
- Update otel to avoid race in tests by @reyreaud-l in #4479
- Add text2vec-ollama module e2e tests by @antas-marcin in #4599
- Solve sharded lock test flakiness by @jeroiraz in #4763
- Auto-cancel old GHA run by @reyreaud-l in #4815
- EC acceptance test sleep waiting for replication by @moogacs in #4825
- Add new GHA ci check to detect protobuf breaking changes by @reyreaud-l in #4833
- Index queue: fix race in test by @asdine in #4852
Fixes
- Thread-safe bucket creation and loading by @jeroiraz in #4390
- Thread saftey on runJobOnBuckets() by @moogacs in #4421
- Delete existent precalculated tmp bloom filter by @jeroiraz in #4469
- Thread safety for bucket creation and loading by @moogacs in #4422
- Fix panic with named vectors and generative module by @dirkkul in #4577
- Fix parsing of tenant in
/schema/{className}/shards?tenant={tenant}
by @tsmith023 in #4777 - Errgroup wait correction on idx add property by @moogacs in #4768
- Update schema when PQ is enabled for a named vector index by @parkerduckworth in #4779
- Class info error not found instead of exists by @moogacs in #4788
- Update error message for class already exists to include class name by @reyreaud-l in #4791
- Change
ListValue
message to avoid gRPCN+1
problem by @tsmith023 in #4565 - Fix backup restore with
include
&exclude
by @tsmith023 in #4804 - Fix node status indication in /v1/cluster/statistics response by @antas-marcin in #4811
- Don't return shutdown error incase the shard was already shutdown by @moogacs in #4839
- Max length field validation during object marshalling by @jeroiraz in #4877
Docs & Chores
- Update resource links in README by @victorialslocum in #4208
- DR-214 update git hub readme by @daveatweaviate in #4486
- Update references to old default branch by @parkerduckworth in #4568
- Update data local nodes dir name for cleaning on restart dev env by @moogacs in #4773
- Fix log typo on delete objects replicator by @moogacs in #4774
- Added error handling to extractSchemaVersionFromUrlQuery by @jeroiraz in #4775
- Fix function names in comment by @mountcount in #4627
- Remove unused code from
schemaManager
by @etiennedi in #4843 - Update v1 endpoint doc links by @daveatweaviate in #4812
- Remove debug logs by @etiennedi in #4889
New Contributors
- @litlig made their first contribution in #4155
- @arturmelanchyk made their first contribution in #4249
- @victorialslocum made their first contribution in #4208
- @daveatweaviate made their first contribution in #4486
- @ajit283 made their first contribution in #4630
- @nathanwilk7 made their first contribution in #4689
- @mountcount made their first contribution in #4627
- @tmoreau89 made their first contribution in #4771
Full Changelog: v1.24.12...v1.25.0