[Bug]: Using v1.0.3 client/server with a database created by v0.6.3 client/server causes chromadb.errors.InternalError
#4217
Comments
|
@Davidyz, try to build a new server image with the following dockerfile: FROM rust:1.81.0 AS builder
ARG RELEASE_MODE=release
WORKDIR /chroma/
ENV PROTOC_ZIP=protoc-25.1-linux-x86_64.zip
RUN curl -OL https://github.com/protocolbuffers/protobuf/releases/download/v25.1/$PROTOC_ZIP \
&& unzip -o $PROTOC_ZIP -d /usr/local bin/protoc \
&& unzip -o $PROTOC_ZIP -d /usr/local 'include/*' \
&& rm -f $PROTOC_ZIP
RUN git clone --branch trayan-04-08-fix_improving_backfil_error_propagation https://github.com/chroma-core/chroma.git .
RUN apt-get update && \
apt-get install -y python3.11-dev
# Build dependencies first (for better caching)
RUN cargo build --bin chroma --release
#FROM gcr.io/distroless/cc-debian12
FROM debian:bookworm-slim
# Copy the binary from the build stage
COPY --from=builder /chroma/target/release/chroma /usr/local/bin/
RUN apt-get update && \
apt-get install -y python3.11-dev
EXPOSE 8000
ENTRYPOINT ["chroma"]
CMD ["run","--path","/data","--host","0.0.0.0"]Build it: docker build -t chromarust -f Dockerfile .Then run your server: docker run -v ./<local_dir>:/data -p 8000:8000 chromarustTest it out and let me know what error you get then. |
|
Hi @tazarov , thanks for the instructions and apologies for replying late. Here's the error output using v1.0.3 client (the server is built from the dockerfile provided above): EDIT: this error persists on v1.0.4 client |
|
hey @Davidyz, thanks for providing the logs. I see that now we do get, as anticipated, more feedback from the error messages. Looking at the code, this seems to be related to sqlite3. Can I bother you to build another image from a new branch where I've added propagation of sqlite3 related errors: FROM rust:1.81.0 AS builder
ARG RELEASE_MODE=release
WORKDIR /chroma/
ENV PROTOC_ZIP=protoc-25.1-linux-x86_64.zip
RUN curl -OL https://github.com/protocolbuffers/protobuf/releases/download/v25.1/$PROTOC_ZIP \
&& unzip -o $PROTOC_ZIP -d /usr/local bin/protoc \
&& unzip -o $PROTOC_ZIP -d /usr/local 'include/*' \
&& rm -f $PROTOC_ZIP
RUN git clone --branch trayan-04-10-chore_local_compation_manager_error_propagation_for_sqlite https://github.com/chroma-core/chroma.git .
RUN apt-get update && \
apt-get install -y python3.11-dev
# Build dependencies first (for better caching)
RUN cargo build --bin chroma --release
#FROM gcr.io/distroless/cc-debian12
FROM debian:bookworm-slim
# Copy the binary from the build stage
COPY --from=builder /chroma/target/release/chroma /usr/local/bin/
RUN apt-get update && \
apt-get install -y python3.11-dev
EXPOSE 8000
ENTRYPOINT ["chroma"]
CMD ["run","--path","/data","--host","0.0.0.0"] |
|
Hi @tazarov thanks for the patch. Here's the new error message: This also applies to both 1.0.3 and 1.0.4 |
|
hey @Davidyz, thanks so much for the quick turnaround. Once again, much appreciate you sticking with this and I can understand your frustration with this. As you point out things should "just work" and indeed this is the intent here :). Reality though is that bugs do creep in from time to time. The error above starts to make sense now. My hypothesis at this point is misbehaving DB migration. Can I bother you for one last bit of feedback: sqlite3 <persist_dir>/chroma.sqlite3 'select segment_id, hex(seq_id) from max_seq_id;'The above command will output something like this: Aslo this: sqlite3<persist_dir>/chroma.sqlite3 ' 'select dir,version,filename,hash from migrations' Resulting in something like this: |
|
@tazarov, here is the output for the commands that you requested: |
|
Hey @Davidyz, much appreciated. This tells me two things:
Good news is that I am able to reproduce your error 100% and the fix for it is relatively simple. Just run: sqlite3 <persist_dir>/chroma.sqlite3 "delete from migrations where dir ='metadb' and filename='00005-max-seq-id-int.sqlite.sql';" Then run your query again. While the above will solve the issue you are having it doesn't explain how the migration script failed. Possibly due to another error that you've encountered along the way that may have prevented the migration from applying. I'll keep digging. |
|
Thanks! I'm glad I was able to help! I'm maintaining a code repository indexing tool that acts as a context provider (MCP tools) for LLM applications, and the current version pinned chromadb to 0.6.3. Should I stay at 0.6.3 until this is fixed in a future release of chromadb? (If I unpin chromadb now, it'll work for new users, but for existing user,s it might break their setup.) |
|
@Davidyz, Chroma 1.0 brings lots of improvements, main of which is performance however, if you have a large based of existing users on 0.6.3 I would recommend pining the version for the time being until we figure the root cause of the failure and possibly fix it. I will try to reproduce the upgrade error. Any particular steps you followed in your upgrade process that might be useful? Or did you just unpin and installed the latest available? |
|
Yes I just unpinned the version number in my One thing that may be unusual is that I personally deployed chromadb by |
|
@Davidyz should we keep this open or is everything ironed out for now? |
|
@Davidyz, I tried to reproduce it with 0.6.3 -> 1.0.x upgrade with both server (docker) and CLI but in vain. I think there maybe either a step or a separate process which can hold locks to the sqlite3, but even then the whole application startup should generally fail as the db will be locked. Perhaps it is worth following your exact process:
Correct the above to match your sequence. |
|
Yes these steps are what I used, except that I used a user service that can be created and managed without sudo: and then |
## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - We also propagate errors from sqlite metadata reader - related to debugging of #4217 ## Test plan *How are these changes tested?* - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?*
What happened?
A
collection.get()call using v1.0.3 client and v1.0.3 server on a collection created by v0.6.3 chromadb produced achromadb.errors.InternalError: Error executing plan: Error sending backfill request to compactor.This is the relevant code snippet:
Aside from this error, I'm wondering whether a database created by 0.x.x chromadb is supposed to "just work"? Also, what's the expected outcome if a user has a mismatched chromadb client and server? I'm experiencing errors, and at the same time, I couldn't find many useful guides on what to expect over the upgrade (from 0.x.x to 1.x.x). The Discord channel (migrations) hasn't been updated for a few months. I'm a bit lost at this point because I don't know what to expect when trying to bump the chromadb version for my project.
Versions
Collection created by chromadb v0.6.3 and accessed by chromadb v1.0.3, on python 3.13, arch Linux
Relevant log output
The text was updated successfully, but these errors were encountered: