add tool to synchronize add-on repository with latest addonTemplate

[abdel792]

Link to issue number:

Replaces #41.

Summary of the issue:

While upgrading an existing add-on using standard Git commands remains fully supported, some add-on developers prefer an automated and dedicated tool to streamline the process. Manual step-by-step migration of metadata (from buildVars.py to pyproject.toml) and file merges can require extra care and verification. Providing an automated script offers a convenient alternative for developers who wish to perform these updates quickly and with built-in AST-based metadata handling.

Description of developer facing changes:

• Introduces syncAddonWithTemplate.py at the repository root as an automated alternative to Git-based manual upgrades for add-on developers.
• Protects template-specific files, including .github/workflows/unitTests.yml and tests/, ensuring they are excluded from add-on synchronization.
• Excludes both syncAddonWithTemplate.py and the tests/ directory from static analysis in pyproject.toml (ruff and pyright).
• Adds a dedicated unit test suite for the synchronization logic (tests/unit/test_syncAddonWithTemplate.py).
• Updates documentation for updating existing add-ons and executing unit tests.

Description of development approach:

• Synchronization Engine Creation: Implemented syncAddonWithTemplate.py using an AST-aware approach to seamlessly migrate legacy buildVars.py metadata or modern AddonInfo setups into pyproject.toml. It supports timestamped backups, custom exclusions via .addonmergeignore, and includes .github/workflows/unitTests.yml in PROTECTED_ELEMENTS.
• Linter Exclusion Rules: Configured pyproject.toml to exclude syncAddonWithTemplate.py and tests/ from ruff and pyright checks.
• Verification Suite & Execution Order: Created a dedicated unit test suite (tests/unit/test_syncAddonWithTemplate.py) using unittest and tempfile, utilizing a load_tests hook to guarantee deterministic test execution order.
• Documentation Integration: Updated technical guides to detail CLI usage, available options, and unit testing commands.

Testing strategy:

Validated the synchronization script and unit test suite locally using uv:

1. Executed syncAddonWithTemplate.py across various CLI modes:
   • Standard Mode (inside add-on repo, -ad is optional):

uv run python syncAddonWithTemplate.py

• External Directory Mode:

uv run python /path/to/syncAddonWithTemplate.py -ad /path/to/my-nvda-addon

• Dry Run Simulation:

uv run python syncAddonWithTemplate.py --dry-run

2. Verified that backup folders (<addon>_bak_<timestamp>) were generated, .addonmergeignore rules were respected, and protected elements (including .github/workflows/unitTests.yml and tests/) were not pushed to target add-on repositories.
3. Executed the unit test suite via unittest:

uv run python -m unittest -v tests/unit/test_syncAddonWithTemplate.py

Confirmed all tests passed successfully.
4. Confirmed ruff and pyright ignore syncAddonWithTemplate.py and tests/.

Known issues with pull request:

None.

---

Code Review Checklist

• Testing: Tested manually across diverse execution paths and validated via local unit tests. Steps to reproduce are detailed in the testing strategy.
• Compatibility: Developer-facing infrastructure tool; introduces no breaking changes to existing repository behaviors.
• Documentation: Technical developer documentation has been explicitly updated.
• UX of all users considered: Not applicable (Developer-facing infrastructure tool only).
• Security precautions taken: Not applicable (Runs locally in isolated temporary directories during development).

[Copilot]

Pull request overview

Adds a developer-facing synchronization tool to help NVDA add-on authors update their repositories to the latest AddonTemplate structure, including AST-aware metadata handling and TOML merging, with accompanying unit tests and documentation updates.

Changes:

• Introduces syncAddonWithTemplate.py to sync template infrastructure while merging buildVars.py and pyproject.toml.
• Adds unit tests for key merge/formatting behavior and a small template sanity suite.
• Updates dependency groups/docs to support running the sync tool and the unit test suite.

Reviewed changes

Copilot reviewed 8 out of 10 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
syncAddonWithTemplate.py	New sync/merge engine (AST + tomlkit) for updating add-on repos from the template.
tests/unit/test_syncAddonWithTemplate.py	Unit tests for metadata extraction, buildVars merge, TOML indentation, and dependency merging.
tests/unit/template/sanity.py	Minimal sanity tests intended to validate CI test discovery/execution.
docs/managementFromGit/updatingExistingAddons.md	Expanded guidance for updating add-ons (automated tool + manual merge).
docs/unitTesting.md	Updated instructions for running the unit tests and describing the new suites.
pyproject.toml	Adds tomlkit, updates pyright, and excludes the new sync script/tests from ruff/pyright.
uv.lock	Locks tomlkit and bumps pyright to match pyproject.toml.
tests/__init__.py	Updates package docstring.
tests/unit/__init__.py	Updates package docstring.
.addonmergeignore	Present at repo root (empty in this PR context).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[abdel792]

Hi @seanbudd,
Thanks for your feedback! I have updated the branch accordingly:

• Docstrings adjustment: I moved the test infrastructure docstring from root unit tests to the template submodule and restored the root tests/__init__.py docstring to reflect all test types (system and unit).
• Sanity test removal: I removed the obsolete sanity.py test file.

[abdel792]

@Copilot,
Here are the updates I made regarding your remarks on normalization and .addonmergeignore:

• Package Name Normalization: I replaced underscores with hyphens in getBasePackageName and legacyToolingBases to consistently handle equivalent Python package name formats during dependency merges.
• Dynamic Bootstrapping & Self-Exclusion: Instead of adding .addonmergeignore to protectedElements, I updated syncAddonWithTemplate.setupAddonMergeIgnore to bootstrap .addonmergeignore from the template on the first sync if it is absent locally, ensuring it includes its own self-exclusion. I prefer this approach as it allows the file to self-reference and manage its own persistence dynamically.

[abdel792]

@Copilot,
Regarding your comment on the load_tests implementation in test_syncAddonWithTemplate.py:

• Robust Test Sorting: I updated load_tests to use an orderIndex lookup dictionary with orderIndex.get(a, defaultOrder) instead of methodOrder.index().
• Error Prevention: I added this to prevent potential ValueError exceptions if inherited or dynamic test methods are present during test execution.

[abdel792]

@Copilot,
Regarding your feedback on the documentation:

• Unit Test Path Fix: I fixed the incorrect path for test_syncAddonWithTemplate.py in docs/unitTesting.md.
• CLI Usage & Dependency Clarification: I clarified the tomlkit installation requirement and corrected the no-argument command example in docs/managementFromGit/updatingExistingAddons.md.

[abdel792]

Hi @seanbudd, @Copilot,

Regarding the test coverage for .addonmergeignore:

• Unit Tests: I added testSetupAddonMergeIgnore and testAddonMergeIgnore to test_syncAddonWithTemplate.py to verify both the creation/bootstrapping behavior when missing and the actual file exclusion during synchronization.
• Documentation: I documented both new test methods in docs/unitTesting.md.

[abdel792]

Hi @seanbudd,

When running pyright on version 1.1.407 via uv (e.g., uv run pyright addon), the underlying Python wrapper package (pyright-python) checks for upstream NPM release updates and systematically outputs a warning to stderr before executing the type checker:

WARNING: there is a new pyright version available (v1.1.407 -> v1.1.411).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`

I bumped the lock version to 1.1.411 in pyproject.toml to eliminate this noise during local check runs and developer workflows.

Let me know if you would still prefer me to revert it to keep the PR scope strictly minimal!

[abdel792]

Thanks for the review and all these helpful suggestions, @seanbudd!

I have addressed the quick fixes and documentation updates:

• Restored the exact casing for AddonTemplate in docs/managementFromGit/updatingExistingAddons.md.
• Removed the detailed test suite section from docs/unitTesting.md (relying on test docstrings instead).
• Fixed the missing newline at the end of test_syncAddonWithTemplate.py.

Regarding your suggestion to split syncAddonWithTemplate.py into submodules: I completely agree, it's a great idea that will make the code much cleaner and easier to maintain!

I'm currently refactoring the code into submodules, writing the requested additional test cases, and running all necessary tests. I will address all of your remaining feedback and suggestions throughout this process and post a full update over the weekend.

I'll leave the PR in draft mode until everything is ready for review.

[abdel792]

Sean, regarding your comment on legacyToolingBases:

I'm currently implementing a solution for this and will push it in my next commit over the weekend, as I need to finish up some work commitments first.

The previous legacyToolingBases set was indeed handling this, but as you pointed out, maintaining such a hardcoded list long-term is cumbersome.

Replacing it with a dynamic lookup alongside a minimal mapping dictionary for non-supported/replaced dependencies (like pre-commit) makes maintenance much simpler.

The updated logic now dynamically collects all tooling packages directly from the template's dependencies and dependency-groups in pyproject.toml.

Additionally, for deprecated tools that were replaced in the template (such as pre-commit being replaced by prek), I introduced a lightweight dictionary (REPLACED_PACKAGES = {"pre-commit": "prek"}).

This maps replaced tools to their modern template equivalents so they aren't accidentally preserved as custom dependencies.

Here is a snippet of how the dynamic collection works:

# Dynamically collect all template packages (dependencies + dependency-groups)
templateBases: set[str] = {
	getBasePackageName(d) for d in tplDeps if isinstance(d, str)
}
if "dependency-groups" in mergedData and isinstance(mergedData["dependency-groups"], MutableMapping):
	for grp in mergedData["dependency-groups"].values():
		if isinstance(grp, (list, MutableSequence)):
			for grpItem in grp:
				if isinstance(grpItem, str):
					templateBases.add(getBasePackageName(grpItem))

[abdel792]

Hi Sean,

Here is a summary of what I recently added:

• Module Modularization: Created the syncAddonTool package directory and split syncAddonWithTemplate.py into dedicated submodules (engine, buildVarsSync, pyproject) to improve readability and code structure.

• PyInstaller Executable Support: Added syncAddonTool.spec at the repository root to generate a single standalone executable using uv run --with pyinstaller pyinstaller syncAddonTool.spec.
  Why PyInstaller? Now that the sync tool is split into syncAddonWithTemplate.py and the syncAddonTool/ package directory, portability outside the repository becomes less straightforward. Users running the tool from external directories would otherwise need to manually copy and track the syncAddonTool/ folder alongside the script. Generating a single standalone binary completely restores full portability—allowing users and CI workflows to run synchronization anywhere without managing relative paths or dependencies.

• Test Fixtures: Moved embedded Python strings and legacy dictionary definitions out of test_syncAddonWithTemplate.py and into external file fixtures.

• Unit Tests Enhancement: Added dedicated unit tests to verify dependency version resolution, specifically covering scenarios where project dependency versions are higher than those in the template.

[abdel792]

Hi Sean,

Here is a summary of the changes made for the documentation update:

• Documentation Update: Updated docs/managementFromGit/updatingExistingAddon.md to reflect the new syncAddonTool package layout.
• Standalone Executable Instructions: Added clear guidelines explaining how to compile and run the standalone executable using PyInstaller without prior environment setup via uv run --with pyinstaller.

[abdel792]

Hi Sean,

Regarding your comment about legacyToolingBases and hardcoded lists:
The hardcoded legacyToolingBases set and legacy tool mapping dictionaries (including the pre-commit replacement logic) have been replaced.
The tool now reads directly from the [dependency-groups] section of pyproject.toml as the single source of truth. It dynamically inspects and extracts the tool base names from the template's dependency groups at runtime, avoiding any hardcoded lists and ensuring seamless maintainability as the template evolves.

[abdel792]

I added the syncAddonTool/ directory and the syncAddonTool.spec file to the Pyright and Ruff exclusions in pyproject.toml.

[abdel792]

In the last two commits, I made the following updates:

• First commit: Added the build/ and dist/ directories to .gitignore to prevent tracking PyInstaller build artifacts.
• Second commit: Fixed an issue causing extra consecutive empty lines to be added to TOML files after generation.

[abdel792]

Hi Sean,

Thanks for the feedback! I've noted all your points and will work on applying these updates tomorrow or the day after.

[abdel792]

Hi Sean,

I have updated the PR with the following cleanup and refactoring changes:

• Removed deprecated script: Deleted syncAddonWithTemplate.py and moved syncAddonTool.spec inside the syncAddonTool/ directory.
• Renamed unit test: Renamed test_syncAddonWithTemplate.py to test_syncAddonTool.py under tests/unit/template/.
• Updated linter configuration: Removed obsolete exclusions in pyproject.toml for Pyright and Ruff, as syncAddonTool.spec is now naturally handled by the existing syncAddonTool directory inclusion/exclusion rules.
• Documentation fix: Updated docs/unittesting.md to reflect the new test file name (test_syncAddonTool.py).

[abdel792]

Hi Sean,

I have updated docs/managementFromGit/updatingExistingAddons.md to document the syncAddonTool automated synchronization process.

While working on these changes, I also noticed three minor pre-existing typos/formatting issues in the master version of this file:

1. "submolder" instead of "subfolder" in item 3 under Pre-requisites.
2. Unmatched quote / trailing dash in backtick: if you haven't passed the "--squash- flag, under Before you begin.
3. Duplicate section: ## Adding the template repository appears both in the top pre-requisites section and under the manual update section.

I deliberately left these untouched to keep the diff strictly focused on introducing syncAddonTool.

However, if you'd like me to clean those up in this PR as well, I'd be glad to push a quick fix!

[abdel792]

Hi Sean,

Just pushed a quick update to docs/managementFromGit/updatingExistingAddons.md to fix the execution syntax in the examples (removing the .py extension when using -m).

As mentioned earlier, I haven't touched the three pre-existing typos/formatting issues yet—let me know if you'd like me to fix them in this PR or leave them for later.

[abdel792]

Hi Sean,

Just pushed a new update to docs/managementFromGit/updatingExistingAddons.md to add the -v / --verbose flag to the options table and clean up the debug output during dependency merging. Running syncAddonTool commands with -v now gives much cleaner, deterministic logs without duplicate decision entries.

As agreed, I haven't touched the three minor pre-existing typos/formatting issues.

[abdel792]

While reviewing a verbose debug report, I noticed that section key logging during the pyproject.toml merge was ambiguous for duplicate keys like include and exclude (it wasn't clear whether they belonged to Ruff, Pyright, or another section).

I just pushed a small refinement to pass the full TOML context path to the merge logger.

This makes the debug output much more explicit and easier to trace:

• Before: [DEBUG] Merging list [include]
• Now: [DEBUG] Merging list [tool.ruff.include] / [DEBUG] Merging list [tool.pyright.include]

[abdel792]

Hi @seanbudd,

I just merged master into this PR branch to ensure it remains fast-forwardable on top of master.