Running LocalGPT on M1 Mac - no support for int64 reduction ops #503

Elearnn1 · 2023-09-20T11:39:08Z

Elearnn1
Sep 20, 2023

When I get to the "enter a query" stage after running Python run_LocalGPT.py --device_type mps I get the below error:

/Users/user/miniforge3/envs/LocalGPT/lib/python3.10/site-packages/InstructorEmbedding/instructor.py:278: UserWarning: MPS: no support for int64 reduction ops, casting it to int32 (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/native/mps/operations/ReduceOps.mm:144.)
assert torch.sum(attention_mask[local_idx]).item() >= context_masks[local_idx].item(),
ggml_metal_graph_compute: command buffer 0 failed with status 5
GGML_ASSERT: /private/var/folders/56/klht16nj4ks4hngpm9trsm540000gn/T/pip-install-jcn1s898/llama-cpp-python_58a63b51a93847c5ab0a87543a11841e/vendor/llama.cpp/ggml-metal.m:1185: false
zsh: abort python run_localGPT.py --device_type mps
/Users/user/miniforge3/envs/LocalGPT/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

I've followed all the amendment to make this run on M1 Mac but I have obviously missed something...?

Elearnn1 · 2023-09-20T12:14:16Z

Elearnn1
Sep 20, 2023
Author

added info:

(LocalGPT) user@e-MacBook-Air 2023-09-20 13:10:38,430 - 2023-09-20 13:10:38,430 - 2023-09-20 13:10:38,430 - 2023-09-20 13:10:38,729 - load INSTRUCTOR_Transformer
max_seq_length 512
2023-09-20 13:10:41,433 - 2023-09-20 13:10:41,484 - 2023-09-20 13:10:41,484 - 2023-09-20 13:10:41,484 - llama_model_loader: loaded llama_model_loader: - tensor 0: llama_model_loader: - tensor 1: llama_model_loader: - tensor 2: llama_model_loader: - tensor 3: llama_model_loader: - tensor 4: llama_model_loader: - tensor 5: llama_model_loader: - tensor 6: llama_model_loader: - tensor 7: llama_model_loader: - tensor 8: llama_model_loader: - tensor 9: llama_model_loader: - tensor 10: llama_model_loader: - tensor 11: llama_model_loader: - tensor 12: llama_model_loader: - tensor 13: llama_model_loader: - tensor 14: llama_model_loader: - tensor 15: llama_model_loader: - tensor 16: llama_model_loader: - tensor 17: llama_model_loader: - tensor 18: llama_model_loader: - tensor 19: llama_model_loader: - tensor 20: llama_model_loader: - tensor 21: llama_model_loader: - tensor 22: llama_model_loader: - tensor 23: llama_model_loader: - tensor 24: llama_model_loader: - tensor 25: llama_model_loader: - tensor 26: llama_model_loader: - tensor 27: llama_model_loader: - tensor 28: llama_model_loader: - tensor 29: llama_model_loader: - tensor 30: llama_model_loader: - tensor 31: llama_model_loader: - tensor 32: llama_model_loader: - tensor 33: llama_model_loader: - tensor 34: llama_model_loader: - tensor 35: llama_model_loader: - tensor 36: llama_model_loader: - tensor 37: llama_model_loader: - tensor 38: llama_model_loader: - tensor 39: llama_model_loader: - tensor 40: llama_model_loader: - tensor 41: llama_model_loader: - tensor 42: llama_model_loader: - tensor 43: llama_model_loader: - tensor 44: llama_model_loader: - tensor 45: llama_model_loader: - tensor 46: llama_model_loader: - tensor 47: llama_model_loader: - tensor 48: llama_model_loader: - tensor 49: llama_model_loader: - tensor 50: llama_model_loader: - tensor 51: llama_model_loader: - tensor 52: llama_model_loader: - tensor 53: llama_model_loader: - tensor 54: llama_model_loader: - tensor 55: llama_model_loader: - tensor 56: llama_model_loader: - tensor 57: llama_model_loader: - tensor 58: llama_model_loader: - tensor 59: llama_model_loader: - tensor 60: llama_model_loader: - tensor 61: llama_model_loader: - tensor 62: llama_model_loader: - tensor 63: llama_model_loader: - tensor 64: llama_model_loader: - tensor 65: llama_model_loader: - tensor 66: llama_model_loader: - tensor 67: llama_model_loader: - tensor 68: llama_model_loader: - tensor 69: llama_model_loader: - tensor 70: llama_model_loader: - tensor 71: llama_model_loader: - tensor 72: llama_model_loader: - tensor 73: llama_model_loader: - tensor 74: llama_model_loader: - tensor 75: llama_model_loader: - tensor 76: llama_model_loader: - tensor 77: llama_model_loader: - tensor 78: llama_model_loader: - tensor 79: llama_model_loader: - tensor 80: llama_model_loader: - tensor 81: llama_model_loader: - tensor 82: llama_model_loader: - tensor 83: llama_model_loader: - tensor 84: llama_model_loader: - tensor 85: llama_model_loader: - tensor 86: llama_model_loader: - tensor 87: llama_model_loader: - tensor 88: llama_model_loader: - tensor 89: llama_model_loader: - tensor 90: llama_model_loader: - tensor 91: llama_model_loader: - tensor 92: llama_model_loader: - tensor 93: llama_model_loader: - tensor 94: llama_model_loader: - tensor 95: llama_model_loader: - tensor 96: llama_model_loader: - tensor 97: llama_model_loader: - tensor 98: llama_model_loader: - tensor 99: llama_model_loader: - tensor 100: llama_model_loader: - tensor 101: llama_model_loader: - tensor 102: llama_model_loader: - tensor 103: llama_model_loader: - tensor 104: llama_model_loader: - tensor 105: llama_model_loader: - tensor 106: llama_model_loader: - tensor 107: llama_model_loader: - tensor 108: llama_model_loader: - tensor 109: llama_model_loader: - tensor 110: llama_model_loader: - tensor 111: llama_model_loader: - tensor 112: llama_model_loader: - tensor 113: llama_model_loader: - tensor 114: llama_model_loader: - tensor 115: llama_model_loader: - tensor 116: llama_model_loader: - tensor 117: llama_model_loader: - tensor 118: llama_model_loader: - tensor 119: llama_model_loader: - tensor 120: llama_model_loader: - tensor 121: llama_model_loader: - tensor 122: llama_model_loader: - tensor 123: llama_model_loader: - tensor 124: llama_model_loader: - tensor 125: llama_model_loader: - tensor 126: llama_model_loader: - tensor 127: llama_model_loader: - tensor 128: llama_model_loader: - tensor 129: llama_model_loader: - tensor 130: llama_model_loader: - tensor 131: llama_model_loader: - tensor 132: llama_model_loader: - tensor 133: llama_model_loader: - tensor 134: llama_model_loader: - tensor 135: llama_model_loader: - tensor 136: llama_model_loader: - tensor 137: llama_model_loader: - tensor 138: llama_model_loader: - tensor 139: llama_model_loader: - tensor 140: llama_model_loader: - tensor 141: llama_model_loader: - tensor 142: llama_model_loader: - tensor 143: llama_model_loader: - tensor 144: llama_model_loader: - tensor 145: llama_model_loader: - tensor 146: llama_model_loader: - tensor 147: llama_model_loader: - tensor 148: llama_model_loader: - tensor 149: llama_model_loader: - tensor 150: llama_model_loader: - tensor 151: llama_model_loader: - tensor 152: llama_model_loader: - tensor 153: llama_model_loader: - tensor 154: llama_model_loader: - tensor 155: llama_model_loader: - tensor 156: llama_model_loader: - tensor 157: llama_model_loader: - tensor 158: llama_model_loader: - tensor 159: llama_model_loader: - tensor 160: llama_model_loader: - tensor 161: llama_model_loader: - tensor 162: llama_model_loader: - tensor 163: llama_model_loader: - tensor 164: llama_model_loader: - tensor 165: llama_model_loader: - tensor 166: llama_model_loader: - tensor 167: llama_model_loader: - tensor 168: llama_model_loader: - tensor 169: llama_model_loader: - tensor 170: llama_model_loader: - tensor 171: llama_model_loader: - tensor 172: llama_model_loader: - tensor 173: llama_model_loader: - tensor 174: llama_model_loader: - tensor 175: llama_model_loader: - tensor 176: llama_model_loader: - tensor 177: llama_model_loader: - tensor 178: llama_model_loader: - tensor 179: llama_model_loader: - tensor 180: llama_model_loader: - tensor 181: llama_model_loader: - tensor 182: llama_model_loader: - tensor 183: llama_model_loader: - tensor 184: llama_model_loader: - tensor 185: llama_model_loader: - tensor 186: llama_model_loader: - tensor 187: llama_model_loader: - tensor 188: llama_model_loader: - tensor 189: llama_model_loader: - tensor 190: llama_model_loader: - tensor 191: llama_model_loader: - tensor 192: llama_model_loader: - tensor 193: llama_model_loader: - tensor 194: llama_model_loader: - tensor 195: llama_model_loader: - tensor 196: llama_model_loader: - tensor 197: llama_model_loader: - tensor 198: llama_model_loader: - tensor 199: llama_model_loader: - tensor 200: llama_model_loader: - tensor 201: llama_model_loader: - tensor 202: llama_model_loader: - tensor 203: llama_model_loader: - tensor 204: llama_model_loader: - tensor 205: llama_model_loader: - tensor 206: llama_model_loader: - tensor 207: llama_model_loader: - tensor 208: llama_model_loader: - tensor 209: llama_model_loader: - tensor 210: llama_model_loader: - tensor 211: llama_model_loader: - tensor 212: llama_model_loader: - tensor 213: llama_model_loader: - tensor 214: llama_model_loader: - tensor 215: llama_model_loader: - tensor 216: llama_model_loader: - tensor 217: llama_model_loader: - tensor 218: llama_model_loader: - tensor 219: llama_model_loader: - tensor 220: llama_model_loader: - tensor 221: llama_model_loader: - tensor 222: llama_model_loader: - tensor 223: llama_model_loader: - tensor 224: llama_model_loader: - tensor 225: llama_model_loader: - tensor 226: llama_model_loader: - tensor 227: llama_model_loader: - tensor 228: llama_model_loader: - tensor 229: llama_model_loader: - tensor 230: llama_model_loader: - tensor 231: llama_model_loader: - tensor 232: llama_model_loader: - tensor 233: llama_model_loader: - tensor 234: llama_model_loader: - tensor 235: llama_model_loader: - tensor 236: llama_model_loader: - tensor 237: llama_model_loader: - tensor 238: llama_model_loader: - tensor 239: llama_model_loader: - tensor 240: llama_model_loader: - tensor 241: llama_model_loader: - tensor 242: llama_model_loader: - tensor 243: llama_model_loader: - tensor 244: llama_model_loader: - tensor 245: llama_model_loader: - tensor 246: llama_model_loader: - tensor 247: llama_model_loader: - tensor 248: llama_model_loader: - tensor 249: llama_model_loader: - tensor 250: llama_model_loader: - tensor 251: llama_model_loader: - tensor 252: llama_model_loader: - tensor 253: llama_model_loader: - tensor 254: llama_model_loader: - tensor 255: llama_model_loader: - tensor 256: llama_model_loader: - tensor 257: llama_model_loader: - tensor 258: llama_model_loader: - tensor 259: llama_model_loader: - tensor 260: llama_model_loader: - tensor 261: llama_model_loader: - tensor 262: llama_model_loader: - tensor 263: llama_model_loader: - tensor 264: llama_model_loader: - tensor 265: llama_model_loader: - tensor 266: llama_model_loader: - tensor 267: llama_model_loader: - tensor 268: llama_model_loader: - tensor 269: llama_model_loader: - tensor 270: llama_model_loader: - tensor 271: llama_model_loader: - tensor 272: llama_model_loader: - tensor 273: llama_model_loader: - tensor 274: llama_model_loader: - tensor 275: llama_model_loader: - tensor 276: llama_model_loader: - tensor 277: llama_model_loader: - tensor 278: llama_model_loader: - tensor 279: llama_model_loader: - tensor 280: llama_model_loader: - tensor 281: llama_model_loader: - tensor 282: llama_model_loader: - tensor 283: llama_model_loader: - tensor 284: llama_model_loader: - tensor 285: llama_model_loader: - tensor 286: llama_model_loader: - tensor 287: llama_model_loader: - tensor 288: llama_model_loader: - tensor 289: llama_model_loader: - tensor 290: llama_model_loader: - kv 0: llama_model_loader: - kv 1: llama_model_loader: - kv 2: llama_model_loader: - kv 3: llama_model_loader: - kv 4: llama_model_loader: - kv 5: llama_model_loader: - kv 6: llama_model_loader: - kv 7: llama_model_loader: - kv 8: llama_model_loader: - kv 9: llama_model_loader: - kv 10: llama_model_loader: - kv 11: llama_model_loader: - kv 12: llama_model_loader: - kv 13: llama_model_loader: - kv 14: llama_model_loader: - kv 15: llama_model_loader: - kv 16: llama_model_loader: - kv 17: llama_model_loader: - kv 18: llama_model_loader: - type f32: llama_model_loader: - type q4_K: llama_model_loader: - type q6_K: llm_load_print_meta: format llm_load_print_meta: arch llm_load_print_meta: vocab type llm_load_print_meta: n_vocab llm_load_print_meta: n_merges llm_load_print_meta: n_ctx_train llm_load_print_meta: n_ctx llm_load_print_meta: n_embd llm_load_print_meta: n_head llm_load_print_meta: n_head_kv llm_load_print_meta: n_layer llm_load_print_meta: n_rot llm_load_print_meta: n_gqa llm_load_print_meta: f_norm_eps llm_load_print_meta: f_norm_rms_eps llm_load_print_meta: n_ff llm_load_print_meta: freq_base llm_load_print_meta: freq_scale llm_load_print_meta: model type llm_load_print_meta: model ftype llm_load_print_meta: model size llm_load_print_meta: general.name llm_load_print_meta: BOS llm_load_print_meta: EOS llm_load_print_meta: UNK llm_load_print_meta: LF token llm_load_tensors: ggml ctx size = llm_load_tensors: mem required .................................. llama_new_context_with_model: kv self size ggml_metal_init: allocating
ggml_metal_init: loading ggml_metal_init: loaded kernel_add ggml_metal_init: loaded kernel_add_row ggml_metal_init: loaded kernel_mul ggml_metal_init: loaded kernel_mul_row ggml_metal_init: loaded kernel_scale ggml_metal_init: loaded kernel_silu ggml_metal_init: loaded kernel_relu ggml_metal_init: loaded kernel_gelu ggml_metal_init: loaded kernel_soft_max ggml_metal_init: loaded kernel_diag_mask_inf ggml_metal_init: loaded kernel_get_rows_f16 ggml_metal_init: loaded kernel_get_rows_q4_0 ggml_metal_init: loaded kernel_get_rows_q4_1 ggml_metal_init: loaded kernel_get_rows_q8_0 ggml_metal_init: loaded kernel_get_rows_q2_K ggml_metal_init: loaded kernel_get_rows_q3_K ggml_metal_init: loaded kernel_get_rows_q4_K ggml_metal_init: loaded kernel_get_rows_q5_K ggml_metal_init: loaded kernel_get_rows_q6_K ggml_metal_init: loaded kernel_rms_norm ggml_metal_init: loaded kernel_norm ggml_metal_init: loaded kernel_mul_mat_f16_f32 ggml_metal_init: loaded kernel_mul_mat_q4_0_f32 ggml_metal_init: loaded kernel_mul_mat_q4_1_f32 ggml_metal_init: loaded kernel_mul_mat_q8_0_f32 ggml_metal_init: loaded kernel_mul_mat_q2_K_f32 ggml_metal_init: loaded kernel_mul_mat_q3_K_f32 ggml_metal_init: loaded kernel_mul_mat_q4_K_f32 ggml_metal_init: loaded kernel_mul_mat_q5_K_f32 ggml_metal_init: loaded kernel_mul_mat_q6_K_f32 ggml_metal_init: loaded kernel_mul_mm_f16_f32 ggml_metal_init: loaded kernel_mul_mm_q4_0_f32 ggml_metal_init: loaded kernel_mul_mm_q8_0_f32 ggml_metal_init: loaded kernel_mul_mm_q4_1_f32 ggml_metal_init: loaded kernel_mul_mm_q2_K_f32 ggml_metal_init: loaded kernel_mul_mm_q3_K_f32 ggml_metal_init: loaded kernel_mul_mm_q4_K_f32 ggml_metal_init: loaded kernel_mul_mm_q5_K_f32 ggml_metal_init: loaded kernel_mul_mm_q6_K_f32 ggml_metal_init: loaded kernel_rope ggml_metal_init: loaded kernel_alibi_f32 ggml_metal_init: loaded kernel_cpy_f32_f16 ggml_metal_init: loaded kernel_cpy_f32_f32 ggml_metal_init: loaded kernel_cpy_f16_f16 ggml_metal_init: recommendedMaxWorkingSetSize ggml_metal_init: hasUnifiedMemory ggml_metal_init: maxTransferRate llama_new_context_with_model: llama_new_context_with_model: ggml_metal_add_buffer: allocated 'data ggml_metal_add_buffer: allocated 'eval ggml_metal_add_buffer: allocated 'kv ggml_metal_add_buffer: allocated 'alloc AVX = 0 | AVX2 = 0 | AVX512 localGPT % python run_localGPT.py --device_type mps
INFO - run_localGPT.py:221 - Running on: mps
INFO - run_localGPT.py:222 - Display Source Documents set to: False
INFO - run_localGPT.py:223 - Use history set to: False
INFO - SentenceTransformer.py:66 - Load pretrained SentenceTransformer: hkunlp/instructor-large
INFO - posthog.py:16 - Anonymized telemetry enabled. See https://docs.trychroma.com/telemetry for more information.
INFO - run_localGPT.py:56 - Loading Model: TheBloke/Llama-2-7b-Chat-GGUF, on: mps
INFO - run_localGPT.py:57 - This action can take a few minutes!
INFO - load_models.py:38 - Using Llamacpp for GGUF/GGML quantized models
meta data with 19 key-value pairs and 291 tensors from ./models/models--TheBloke--Llama-2-7b-Chat-GGUF/snapshots/ad37d4910ba009a69bb41de44942056d635214ab/llama-2-7b-chat.Q4_K_M.gguf (version GGUF V2 (latest))
token_embd.weight q4_K [ 4096, 32000, 1, 1 ]
blk.0.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.0.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.0.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.0.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.0.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.0.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.0.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.0.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.0.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.1.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.1.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.1.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.1.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.1.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.1.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.1.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.1.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.1.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.10.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.10.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.10.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.10.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.10.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.10.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.10.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.10.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.10.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.11.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.11.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.11.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.11.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.11.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.11.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.11.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.11.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.11.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.12.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.12.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.12.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.12.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.12.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.12.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.12.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.12.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.12.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.13.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.13.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.13.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.13.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.13.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.13.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.13.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.13.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.13.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.14.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.14.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.14.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.14.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.14.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.14.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.14.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.14.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.14.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.15.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.15.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.15.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.15.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.15.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.15.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.15.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.15.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.15.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.16.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.16.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.16.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.16.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.16.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.16.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.16.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.16.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.16.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.17.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.17.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.17.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.17.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.17.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.17.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.17.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.17.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.17.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.18.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.18.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.18.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.18.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.18.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.18.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.18.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.18.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.18.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.19.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.19.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.19.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.19.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.19.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.19.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.19.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.19.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.19.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.2.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.2.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.2.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.2.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.2.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.2.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.2.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.2.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.2.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.20.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.20.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.20.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.20.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.20.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.20.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.20.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.20.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.20.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.21.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.21.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.21.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.21.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.21.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.21.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.21.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.21.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.21.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.22.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.22.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.22.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.22.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.22.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.22.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.22.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.22.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.22.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.23.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.23.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.23.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.23.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.23.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.23.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.23.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.23.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.23.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.3.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.3.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.3.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.3.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.3.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.3.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.3.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.3.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.3.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.4.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.4.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.4.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.4.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.4.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.4.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.4.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.4.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.4.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.5.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.5.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.5.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.5.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.5.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.5.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.5.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.5.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.5.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.6.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.6.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.6.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.6.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.6.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.6.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.6.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.6.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.6.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.7.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.7.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.7.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.7.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.7.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.7.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.7.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.7.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.7.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.8.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.8.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.8.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.8.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.8.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.8.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.8.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.8.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.8.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.9.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.9.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.9.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.9.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.9.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.9.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.9.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.9.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.9.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
output.weight q6_K [ 4096, 32000, 1, 1 ]
blk.24.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.24.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.24.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.24.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.24.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.24.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.24.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.24.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.24.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.25.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.25.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.25.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.25.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.25.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.25.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.25.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.25.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.25.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.26.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.26.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.26.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.26.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.26.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.26.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.26.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.26.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.26.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.27.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.27.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.27.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.27.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.27.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.27.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.27.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.27.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.27.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.28.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.28.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.28.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.28.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.28.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.28.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.28.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.28.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.28.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.29.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.29.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.29.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.29.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.29.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.29.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.29.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.29.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.29.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.30.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.30.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.30.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.30.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.30.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.30.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.30.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.30.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.30.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.31.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.31.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.31.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.31.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.31.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.31.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.31.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.31.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.31.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
output_norm.weight f32 [ 4096, 1, 1, 1 ]
general.architecture str
general.name str
llama.context_length u32
llama.embedding_length u32
llama.block_count u32
llama.feed_forward_length u32
llama.rope.dimension_count u32
llama.attention.head_count u32
llama.attention.head_count_kv u32
llama.attention.layer_norm_rms_epsilon f32
general.file_type u32
tokenizer.ggml.model str
tokenizer.ggml.tokens arr
tokenizer.ggml.scores arr
tokenizer.ggml.token_type arr
tokenizer.ggml.bos_token_id u32
tokenizer.ggml.eos_token_id u32
tokenizer.ggml.unknown_token_id u32
general.quantization_version u32
65 tensors
193 tensors
33 tensors
= GGUF V2 (latest)
= llama
= SPM
= 32000
= 0
= 4096
= 4096
= 4096
= 32
= 32
= 32
= 128
= 1
= 1.0e-05
= 1.0e-06
= 11008
= 10000.0
= 1
= 7B
= mostly Q4_K - Medium
= 6.74 B
= LLaMA v2
token = 1 ''
token = 2 ''
token = 0 ''
= 13 '<0x0A>'
0.09 MB
= 3891.34 MB (+ 2048.00 MB per state)
................................................................
= 2048.00 MB
'/Users/user/miniforge3/envs/LocalGPT/lib/python3.10/site-packages/llama_cpp/ggml-metal.metal'
0x17dc4c780 | th_max = 1024 | th_width = 32
0x17dc4e500 | th_max = 1024 | th_width = 32
0x17dc4dda0 | th_max = 1024 | th_width = 32
0x17dc4caf0 | th_max = 1024 | th_width = 32
0x17dc4f390 | th_max = 1024 | th_width = 32
0x17dc4fa00 | th_max = 1024 | th_width = 32
0x17dc4cd50 | th_max = 1024 | th_width = 32
0x17dc50700 | th_max = 1024 | th_width = 32
0x17dc50db0 | th_max = 1024 | th_width = 32
0x17dc511f0 | th_max = 1024 | th_width = 32
0x17dc51cb0 | th_max = 1024 | th_width = 32
0x17dc52cc0 | th_max = 1024 | th_width = 32
0x17dc533b0 | th_max = 1024 | th_width = 32
0x17dc53af0 | th_max = 1024 | th_width = 32
0x1052e6200 | th_max = 1024 | th_width = 32
0x1052e6460 | th_max = 1024 | th_width = 32
0x1052e6720 | th_max = 1024 | th_width = 32
0x1052e6980 | th_max = 1024 | th_width = 32
0x17dc53ee0 | th_max = 1024 | th_width = 32
0x17dc54990 | th_max = 1024 | th_width = 32
0x17dc550c0 | th_max = 1024 | th_width = 32
0x17dc55a40 | th_max = 1024 | th_width = 32
0x17dc56260 | th_max = 896 | th_width = 32
0x17dc56c20 | th_max = 896 | th_width = 32
0x17dc573d0 | th_max = 768 | th_width = 32
0x17dc57c80 | th_max = 640 | th_width = 32
0x17dc584c0 | th_max = 704 | th_width = 32
0x17dc58e40 | th_max = 576 | th_width = 32
0x17dc59930 | th_max = 576 | th_width = 32
0x17dc5a1a0 | th_max = 1024 | th_width = 32
0x17dc5aa10 | th_max = 768 | th_width = 32
0x17dc5b2b0 | th_max = 768 | th_width = 32
0x17dc5bae0 | th_max = 768 | th_width = 32
0x17dc5c360 | th_max = 768 | th_width = 32
0x17dc5cb90 | th_max = 768 | th_width = 32
0x17dc5d410 | th_max = 768 | th_width = 32
0x17dc5dc40 | th_max = 768 | th_width = 32
0x17dc5e4c0 | th_max = 704 | th_width = 32
0x17dc5ecf0 | th_max = 704 | th_width = 32
0x17dc5f2f0 | th_max = 1024 | th_width = 32
0x17dc5fc60 | th_max = 1024 | th_width = 32
0x17dc605d0 | th_max = 1024 | th_width = 32
0x17dc60f40 | th_max = 1024 | th_width = 32
0x17dc61830 | th_max = 1024 | th_width = 32
= 5461.34 MB
= true
= built-in GPU
compute buffer total size = 297.47 MB
max tensor size = 102.54 MB
' buffer, size = 3891.95 MB, ( 3892.39 / 5461.34)
' buffer, size = 1.48 MB, ( 3893.88 / 5461.34)
' buffer, size = 2050.00 MB, ( 5943.88 / 5461.34), warning: current allocated size is greater than the recommended max working set size
' buffer, size = 296.02 MB, ( 6239.89 / 5461.34), warning: current allocated size is greater than the recommended max working set size
= 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 |

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Running LocalGPT on M1 Mac - no support for int64 reduction ops #503

{{title}}

Replies: 1 comment

{{title}}

Select a reply

Running LocalGPT on M1 Mac - no support for int64 reduction ops #503

Elearnn1 Sep 20, 2023

Replies: 1 comment

Elearnn1 Sep 20, 2023 Author

Elearnn1
Sep 20, 2023

Elearnn1
Sep 20, 2023
Author