You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
import{readFileSync}from"fs";import{resolve}from"path";import{fileURLToPath}from"url";import{create,insertMultiple,search}from"./dist/index.js";// function to print the used memoryfunctionprintUsedMemory(){constused=process.memoryUsage().heapUsed/1024/1024;console.log(`The script uses approximately ${Math.round(used*100)/100} MB`);}const__dirname=resolve(fileURLToPath(import.meta.url),'..');constdata=JSON.parse(readFileSync(__dirname+'/large-object.json','utf8'),);printUsedMemory();// create index and add dataconstdb=awaitcreate({schema: {id: "string",name: "string",surname: "string",fiscalCode: "string",season: "number",},});awaitinsertMultiple(db,data);console.log("Index created");printUsedMemory();// search the indexconstresults=awaitsearch(db,{term: "john",properties: "*",});printUsedMemory();
The results with internalId are:
The script uses approximately 44.61 MB
Index created
The script uses approximately 546.78 MB
The script uses approximately 549.3 MB
But, if we disable/remove that feature:
The script uses approximately 44.65 MB
Index created
The script uses approximately 502.32 MB
The script uses approximately 504.65 MB
We can reduce the memory usage by ~8%.
New Behavior
Can we have both scenarios where we store the IDs as internal and then remap them? Or is it too much work?
Can we give some flag to the user to choose between lower index size or reduced memory usage?
The text was updated successfully, but these errors were encountered:
Current Behavior
After #441 be merged, we had ~34% reduce of index size when saving to
.json
.But the memory usage increased, so I rewrote the test to generate truly random data (to not take advantage of v8 string cache):
And then I see the memory usage:
The results with
internalId
are:But, if we disable/remove that feature:
We can reduce the memory usage by ~8%.
New Behavior
Can we have both scenarios where we store the IDs as internal and then remap them? Or is it too much work?
Can we give some flag to the user to choose between lower index size or reduced memory usage?
The text was updated successfully, but these errors were encountered: