Try removing custom annotation

[cccclai]

Differential Revision: D78142146

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12386

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (4 Unrelated Failures)

As of commit 0e83560 with merge base fc435fa ():

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / linux / linux-job (gh) (trunk failure)
  devtools/inspector/tests/inspector_utils_test.py::TestInspectorUtils::test_equip_debug_handle_to_export_program_success
• pull / unittest / macos / macos-job (gh) (trunk failure)
  devtools/inspector/tests/inspector_utils_test.py::TestInspectorUtils::test_equip_debug_handle_to_export_program_success
• pull / unittest-editable / linux / linux-job (gh) (trunk failure)
  devtools/inspector/tests/inspector_utils_test.py::TestInspectorUtils::test_equip_debug_handle_to_export_program_success
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  devtools/inspector/tests/inspector_utils_test.py::TestInspectorUtils::test_equip_debug_handle_to_export_program_success

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D78142146

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[cccclai]

@shewu-quic for the graph discrepancy, do you mean changes like this? Is the quant attrs for kv cache stored somewhere else, or as part of the model?

[shewu-quic]

@shewu-quic for the graph discrepancy, do you mean changes like this? Is the quant attrs for kv cache stored somewhere else, or as part of the model?

They are as part of the model. If you would like to remove them, it is better to add a check the quant_attrs is the same both prefill model and kv model.

for node in llama_instance.llama_graph_module.graph.nodes:
                    if node.op == "output":
                        for output in node.args[0]:
                            kv_quant_attrs[output_indices] = output.args[1:]
                            output_indices += 1
                        break

[cccclai]

@shewu-quic for the graph discrepancy, do you mean changes like this? Is the quant attrs for kv cache stored somewhere else, or as part of the model?

They are as part of the model. If you would like to remove them, it is better to add a check the quant_attrs is the same both prefill model and kv model.

for node in llama_instance.llama_graph_module.graph.nodes:
                    if node.op == "output":
                        for output in node.args[0]:
                            kv_quant_attrs[output_indices] = output.args[1:]
                            output_indices += 1
                        break

I see, is it like compare llama_instance[0] and llama_instance[1] for the value above? The code snippets seems not comparing things

[shewu-quic]

I see, is it like compare llama_instance[0] and llama_instance[1] for the value above? The code snippets seems not comparing things

Yes, we didn't compare them since we forcedly set the same value.