There appear to be 1 leaked semaphore objects to clean up at shutdown

[jakubLangr]

Summary

I've tried various workarounds but unfortunately all of them seem very old (due to lib versions).
Have also tried different models.
etc.

Platform

OS: MacOS 10.15
8 GB

Successfully installed Pillow-11.0.0 filelock-3.16.1 fsspec-2024.10.0 huggingface-hub-0.26.1 mpmath-1.3.0 networkx-3.4.2 safetensors-0.4.5 scikit-learn-1.5.2 scipy-1.14.1 sentence-transformers-3.2.1 sympy-1.13.1 threadpoolctl-3.5.0 tokenizers-0.20.1 torch-2.5.0 transformers-4.45.2

Running on:

• [ x] CPU
• GPU

Interface:

• C++
• [x ] Python

Reproduction instructions

model = SentenceTransformer('all-MiniLM-L6-v2', device='cpu')  # Force CPU usage
print(f"Encoding query: {query}")
print(f"Size of RAG data: {len(rag_data)}")
import sys
print(f"Memory usage: {sys.getsizeof(rag_data)} bytes")
query_embedding = model.encode(query, show_progress_bar=False)

Size of RAG data: 272
Memory usage: 2520 bytes
/opt/miniconda3/envs/wapi/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown

FYI Memory usage: 83.7%

Any help would be appreciated!

[sunnsi]

+1, I meet the same issue.

INFO:sentence_transformers.SentenceTransformer:Load pretrained SentenceTransformer: sentence-transformers/all-mpnet-base-v2
INFO - docarray - DB config created
INFO:docarray:DB config created
INFO - docarray - Runtime config created
INFO:docarray:Runtime config created
INFO - docarray - No docs or index file provided. Initializing empty InMemoryExactNNIndex.
INFO:docarray:No docs or index file provided. Initializing empty InMemoryExactNNIndex.
Batches: 0%| | 0/1 [00:00<?, ?it/s][1] 98819 segmentation fault python --port 8510
/opt/anaconda3/envs/llm/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

OS: MacOS 15.0
Memory: 192GB

[tomaarsen]

Hello!

I'd like to chase this down. What Python version are you on? @jakubLangr @sunnsi

There's a handful of potential causes as far as I can tell, for example:

1. Python 3.8 seems to have some issues?
2. High memory usage on Mac seems to cause it?

---

• Tom Aarsen