Issue with deleting collections

[virajone]

I currently do not see an option on the front end to delete collections. These collections are also somehow persisted despite restarting the containers.
Further, on trying to use the existing collections, I get the following error :

Here's the trace from the terminal :

cognita-backend | DEBUG: 2024-07-28 17:42:11,419 - indexer:ingest_data:343 - Starting ingestion for data source fqn: web::https://aws.plainenglish.io/deploying-a-docker-container-in-aws-using-fargate-5a19a140b018
cognita-backend | DEBUG: 2024-07-28 17:42:11,430 - qdrant:list_data_point_vectors:194 - [Qdrant] Listing all data point vectors for collection test
qdrant | 2024-07-28T17:42:11.433558Z INFO actix_web::middleware::logger: 172.18.0.5 "POST /collections/test/points/scroll HTTP/1.1" 404 100 "-" "python-httpx/0.27.0" 0.000507
cognita-backend | ERROR: 2024-07-28 17:42:11,434 - indexer:sync_data_source_to_collection:84 - Unexpected Response: 404 (Not Found)
cognita-backend | Raw response content:
cognita-backend | b'{"status":{"error":"Not found: Collection test doesn't exist!"},"time":0.000032875}'
cognita-backend | Traceback (most recent call last):
cognita-backend | File "/app/backend/indexer/indexer.py", line 72, in sync_data_source_to_collection
cognita-backend | existing_data_point_vectors = VECTOR_STORE_CLIENT.list_data_point_vectors(
cognita-backend | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cognita-backend | File "/app/backend/modules/vector_db/qdrant.py", line 201, in list_data_point_vectors
cognita-backend | records, next_offset = self.qdrant_client.scroll(
cognita-backend | ^^^^^^^^^^^^^^^^^^^^^^^^^^
cognita-backend | File "/virtualenvs/venv/lib/python3.11/site-packages/qdrant_client/qdrant_client.py", line 917, in scroll
cognita-backend | return self._client.scroll(
cognita-backend | ^^^^^^^^^^^^^^^^^^^^
cognita-backend | File "/virtualenvs/venv/lib/python3.11/site-packages/qdrant_client/qdrant_remote.py", line 1255, in scroll
cognita-backend | ] = self.openapi_client.points_api.scroll_points(
cognita-backend | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cognita-backend | File "/virtualenvs/venv/lib/python3.11/site-packages/qdrant_client/http/api/points_api.py", line 1338, in scroll_points
cognita-backend | return self._build_for_scroll_points(
cognita-backend | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cognita-backend | File "/virtualenvs/venv/lib/python3.11/site-packages/qdrant_client/http/api/points_api.py", line 534, in build_for_scroll_points
cognita-backend | return self.api_client.request(
cognita-backend | ^^^^^^^^^^^^^^^^^^^^^^^^
cognita-backend | File "/virtualenvs/venv/lib/python3.11/site-packages/qdrant_client/http/api_client.py", line 76, in request
cognita-backend | return self.send(request, type)
cognita-backend | ^^^^^^^^^^^^^^^^^^^^^^^^^
cognita-backend | File "/virtualenvs/venv/lib/python3.11/site-packages/qdrant_client/http/api_client.py", line 99, in send
cognita-backend | raise UnexpectedResponse.for_response(response)
cognita-backend | qdrant_client.http.exceptions.UnexpectedResponse: Unexpected Response: 404 (Not Found)
cognita-backend | Raw response content:
cognita-backend | b'{"status":{"error":"Not found: Collection test doesn't exist!"},"time":0.000032875}'
cognita-backend | ERROR: 2024-07-28 17:42:11,475 - indexer:ingest_data:412 - Unexpected Response: 404 (Not Found)
cognita-backend | Raw response content:
cognita-backend | b'{"status":{"error":"Not found: Collection test doesn't exist!"},"time":0.000032875}'
cognita-backend | Traceback (most recent call last):
cognita-backend | File "/app/backend/indexer/indexer.py", line 358, in ingest_data
cognita-backend | await sync_data_source_to_collection(
cognita-backend | File "/app/backend/indexer/indexer.py", line 89, in sync_data_source_to_collection
cognita-backend | raise e
cognita-backend | File "/app/backend/indexer/indexer.py", line 72, in sync_data_source_to_collection
cognita-backend | existing_data_point_vectors = VECTOR_STORE_CLIENT.list_data_point_vectors(
cognita-backend | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cognita-backend | File "/app/backend/modules/vector_db/qdrant.py", line 201, in list_data_point_vectors
cognita-backend | records, next_offset = self.qdrant_client.scroll(
cognita-backend | ^^^^^^^^^^^^^^^^^^^^^^^^^^
cognita-backend | File "/virtualenvs/venv/lib/python3.11/site-packages/qdrant_client/qdrant_client.py", line 917, in scroll
cognita-backend | return self._client.scroll(
cognita-backend | ^^^^^^^^^^^^^^^^^^^^
cognita-backend | File "/virtualenvs/venv/lib/python3.11/site-packages/qdrant_client/qdrant_remote.py", line 1255, in scroll
cognita-backend | ] = self.openapi_client.points_api.scroll_points(
cognita-backend | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cognita-backend | File "/virtualenvs/venv/lib/python3.11/site-packages/qdrant_client/http/api/points_api.py", line 1338, in scroll_points
cognita-backend | return self._build_for_scroll_points(
cognita-backend | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cognita-backend | File "/virtualenvs/venv/lib/python3.11/site-packages/qdrant_client/http/api/points_api.py", line 534, in build_for_scroll_points
cognita-backend | return self.api_client.request(
cognita-backend | ^^^^^^^^^^^^^^^^^^^^^^^^
cognita-backend | File "/virtualenvs/venv/lib/python3.11/site-packages/qdrant_client/http/api_client.py", line 76, in request
cognita-backend | return self.send(request, type)
cognita-backend | ^^^^^^^^^^^^^^^^^^^^^^^^^
cognita-backend | File "/virtualenvs/venv/lib/python3.11/site-packages/qdrant_client/http/api_client.py", line 99, in send
cognita-backend | raise UnexpectedResponse.for_response(response)
cognita-backend | qdrant_client.http.exceptions.UnexpectedResponse: Unexpected Response: 404 (Not Found)
cognita-backend | Raw response content:
cognita-backend | b'{"status":{"error":"Not found: Collection test doesn't exist!"},"time":0.000032875}'
cognita-backend | INFO: 192.168.65.1:49862 - "POST /v1/collections/ingest HTTP/1.1" 500 Internal Server Error
cognita-backend | INFO: 192.168.65.1:49862 - "GET /v1/collections/test HTTP/1.1" 200 OK
cognita-backend | INFO: 192.168.65.1:58793 - "POST /v1/collections/data_ingestion_runs/list HTTP/1.1" 200 OK
cognita-backend | INFO: 127.0.0.1:38696 - "GET /health-check HTTP/1.1" 200 OK
cognita-backend | INFO: 127.0.0.1:47780 - "GET /health-check HTTP/1.1" 200 OK

Can you please help me identify :

1. How can I delete the collections from the frontend ? If not, what are the alternatives ?
2. How can I make sure that no data is carried over between 2 different runs (docker compose) of the containers ? (I already tried removing the containers, deleting the volumes but it did not work)

[mnvsk97]

Hello @virajone

I'm sorry for not getting back to you sooner. We now have a button on the front end to delete collections.

Please give it a try and let us know if you have issues. We are constantly improving the UX and would love to hear from you if you see any scope for improvement. Thanks!