-
Notifications
You must be signed in to change notification settings - Fork 943
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Query vector dimension 76800 does not match the dimension of the index 768 #153
Comments
the portion of your code that is creating the actual vectors is the issue where it is producing the 76800 value as dimensions when it should be generating vectors with dimensions of 768. It is multiplied by the batch_size |
if the batch_size is set to 1. In this case, it throws an error in line "for query, pos_passage, query_res in zip(queries, pos_passages, res['results']):" |
I'm having the same problem, have you solved it yet? |
I get the following error
ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'content-type': 'application/json', 'date': 'Mon, 06 Mar 2023 17:34:00 GMT', 'x-envoy-upstream-service-time': '2', 'content-length': '110', 'server': 'envoy', 'connection': 'close'})
HTTP response body: {"code":3,"message":"Query vector dimension 76800 does not match the dimension of the index 768","details":[]}
For the code snippet:
import random
batch_size = 100
triplets = []
for i in tqdm(range(0, len(pairs), batch_size)):
# embed queries and query pinecone in batches to minimize network latency
i_end = min(i+batch_size, len(pairs))
queries = [pair[0] for pair in pairs[i:i_end]]
print(len(queries))
pos_passages = [pair[1] for pair in pairs[i:i_end]]
#print((pos_passages))
# create query embeddings
query_embs = model.encode(queries, convert_to_tensor=True, show_progress_bar=False)
# search for top_k most similar passages
res = index.query(query_embs.tolist(), top_k=1)
# iterate through queries and find negatives
for query, pos_passage, query_res in zip(queries, pos_passages, res['results']):
top_results = query_res['matches']
# shuffle results so they are in random order
random.shuffle(top_results)
for hit in top_results:
neg_passage = pairs[int(hit['id'])][1]
# check that we're not just returning the positive passage
if neg_passage != pos_passage:
# if not we can add this to our (Q, P+, P-) triplets
triplets.append(query+'\t'+pos_passage+'\t'+neg_passage)
break
The text was updated successfully, but these errors were encountered: