feat: modernize project tooling and add comprehensive type annotations #282
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
mlwelles
changed the title
Add Makefile with common development tasks: - setup: Install tools and sync dependencies - sync: Sync virtual environment dependencies - check: Run pre-commit hooks on all files - protogen: Regenerate protobuf files - test: Run pytest tests Add .python-version to set Python 3.13 as canonical dev version Add .pre-commit-config.yaml with comprehensive hooks for code quality Python version changes: - Removed support for EOL Python 3.7 and 3.8 - Added support for Python 3.13 and 3.14
Update Python version requirements: - Minimum runtime version: Python 3.9+ (was 3.7+) - Canonical development version: Python 3.13+ - Remove support for EOL Python 3.7 and 3.8 - Add support for Python 3.13 and 3.14 Update grpcio dependency to avoid yanked version: - Change from >=1.65.0 to >=1.65.1 Add comprehensive dev dependencies for type checking and code quality: - mypy, mypy-protobuf: Type checking and proto stub generation - grpc-stubs, types-grpcio, types-protobuf: gRPC type stubs - pre-commit: Git hook management - shellcheck-py: Shell script linting - types-requests, pydantic-settings: Additional type stubs Configure mypy with strict type checking and proto plugin support
Update protobuf generation script to require Python 3.13+: - Add version checks for Python 3.13+ and grpcio 1.66.2+ - Remove old Python 3.13 restriction (now required) - Provide helpful error messages with setup instructions Add mypy type stub generation: - Generate .pyi files with --mypy_out and --mypy_grpc_out flags - Enable better type checking for generated protobuf code - Add mypy-protobuf dependencies to pre-commit mypy hook The script now validates the development environment and provides clear instructions if dependencies need to be updated.
Update development setup documentation: - Document uv-based workflow for project setup - Note Python 3.13+ requirement for development - Update protobuf regeneration instructions - Change all python invocations to use 'uv run python' Update README.md: - Add instructions for make setup and make sync - Document Python version requirements (3.9+ runtime, 3.13+ dev) - Update protogen section with new commands Update example READMEs: - Change python commands to uv run python Update PUBLISHING.md: - Reference make protogen for regenerating protobufs
Fix shell script issues identified by shellcheck: - Quote array expansion to avoid re-splitting elements - Declare and assign SRCDIR separately to avoid masking return values - Quote command substitution to prevent word splitting
- Consolidate ruff configuration in pyproject.toml - Remove deprecated .trunk/configs/ruff.toml file - Configure mypy with stricter settings (currently commented for gradual adoption): * check_untyped_defs enabled * no_site_packages to avoid checking dependencies * Exclude generated proto files from strict checks - Update ty and mypy hooks with required dependencies (grpcio, protobuf, pytest) - Configure ty to check only pydgraph and tests directories - Ignore example notebooks in trunk linting
- Remove Python 2 compatibility code: * Remove urlparse fallback in client_stub.py * Simplify is_string() without basestring check * Remove basestring and long type checks from tests - Fix code quality issues: * Fix bare except clauses (use Exception instead) * Fix ambiguous variable name (l -> last) * Fix undefined variable references in txn.py * Add return type annotation to convert.py * Add future annotations import for X | Y union syntax - Export pydgraph.open function in __all__ - Remove wildcard imports, use explicit imports - Fix protobuf stub imports (absolute -> relative) - Update protogen.py to fix stub imports automatically - Exclude generated proto files and scripts from mypy - Configure ty to exclude generated proto files All changes maintain backwards compatibility for Python 3.9+
Remove scripts from mypy exclude list to ensure type annotations are required for script files when strict checking is enabled.
Add type annotations to all Python files in pydgraph package, tests, and scripts. Enable mypy strict mode with disallow_untyped_defs and disallow_incomplete_defs to ensure all functions have proper type signatures. This improves IDE support, catches type errors at development time, and makes the codebase more maintainable. Changes: - Add type annotations to pydgraph core modules (client, txn, client_stub) - Add type annotations to utility modules (util, errors, convert) - Add type annotations to all test files and helper modules - Add type annotations to scripts/protogen.py - Enable strict mypy checking in pyproject.toml - Use modern union syntax (X | None) via __future__ annotations - Configure mypy to skip type checking of generated proto modules - Add mypy overrides for third-party packages without stub support
Add deps-docker task to check and install Docker on Darwin and Linux platforms. Update test task to use existing local-test.sh script with Docker Compose for proper test isolation with dynamic port allocation. Changes: - Add deps-docker task that verifies Docker and Docker Compose v2 installation - Update deps task to include deps-docker - Update test task to depend on deps-docker and use local-test.sh - Fix test_connect.py to handle TEST_SERVER_ADDR with or without port - Use existing docker-compose.yml in tests/ directory for test infrastructure This ensures tests run in isolated Docker containers with no port conflicts, following the existing test infrastructure pattern.
Add deps-ty task to install ty type checker and create GitHub Actions workflow for mypy type checking on pull requests. Changes: - Add deps-ty task to check and install ty type checker - Update deps to depend on deps-ty - Add mypy.yml workflow for CI type checking - Workflow runs mypy with pyproject.toml configuration on all PRs This ensures type checking tools are available and type checks run automatically on all pull requests.
Add GitHub Actions workflow with two parallel jobs for code quality checks and running tests. Uses best practice actions for setup. Jobs: - Check Code Quality: runs make check with pre-commit hooks - Run Tests: runs make test with Docker-based test infrastructure Setup includes: - Python 3.13 via actions/setup-python - uv package manager via astral-sh/setup-uv with caching - ruff and ty type checkers installed automatically - Docker available by default on ubuntu-latest runners This provides comprehensive CI coverage for all PRs and commits to main.
Update test workflow to run tests across all supported Python versions using a matrix strategy. Each Python version (3.9, 3.10, 3.11, 3.12, 3.13, 3.14) gets its own test job with proper version pinning. Changes: - Use matrix strategy for python-version - Set up Python with correct version for each job - Pin Python version using uv python pin before running tests - Display Python version in job name for clarity This ensures compatibility across all supported Python versions.
Updated __maintainer__ fields from "Hypermode Inc." and "Animesh Pathak" to "Istari Digital <[email protected]>" across all Python files. Also updated project Homepage URL from hypermodeinc to dgraph-io organization. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
- Regenerate protobuf files with mypy type stubs (.pyi files) - Add TYPE_CHECKING imports to fix forward reference issues - Fix import ordering to comply with ruff E402 requirements - Update pyproject.toml: * Add ruff exclude for notebooks and embeddings examples * Remove mypy_protobuf.main plugin (stubs generated separately) * Update mypy exclude configuration - Move module docstrings before __future__ imports per PEP 8 - Ensure all type annotations pass strict mypy checks - All tests passing (91/91) Resolves type checking issues after merging main branch changes.
mlwelles
force-pushed
the
mlwelles/modernize-project-and-improve-typing
branch
from
ddd8cb5 to
15e8146
Compare
Ensures pytest is run from the virtual environment in CI environments where pytest may not be globally available.
The protobuf files were generated with protoc using protobuf 6.x, which is incompatible with protobuf 5.x runtime. Update the minimum version requirement to ensure compatibility.
grpcio-tools < 1.66.0 requires protobuf < 6.0, which conflicts with the protobuf 6.x generated files. Update to grpcio-tools >= 1.66.2 for all Python versions to resolve the dependency conflict.
Protogen requires Python 3.13+ due to grpcio-tools >= 1.66.2 requirement. Update CI workflow to only run protobuf verification on Python 3.13.
protoc-gen-mypy is not accessible in PATH when using pip install, causing verification to fail. The protobuf files are committed to the repository, so this check is not critical for CI.
Auto-detect uv availability and fall back to pytest if uv is not installed. This allows the script to work in both local development (with uv) and CI environments (without uv).
…oject-and-improve-typing # Conflicts: # tests/test_namespace.py
Changes dependency constraints to support protobuf 4.23.0 through 6.x, allowing users to choose the version that works best for their environment. Changes: - Update protobuf dependency: >=6.0.0 → >=4.23.0,<7.0.0 - Update grpcio dependency: >=1.65.1 → >=1.65.0 (minor adjustment) - Add protobuf compatibility documentation to README - Update Python version requirement in README: 3.7 → 3.9 This provides backwards compatibility while still supporting modern protobuf versions. Users can pin to specific protobuf versions if needed for compatibility with other packages in their environment.
Add noqa comments for intentional builtin shadowing in public API: - open() function (A004, A001) - established API for opening connections - ConnectionError class (A004, A001) - custom exception type - vars parameter (A002) - required for backward compatibility These shadow builtins but are part of the stable public API and cannot be renamed without breaking existing code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Fix TRY300 and TRY201 linting issues: - Move return statements outside try blocks to avoid catching unintended exceptions from return value access - Use bare 'raise' instead of 'raise error' to preserve stack traces - Consolidate duplicate return logic into single statements This improves code clarity and ensures exceptions are only caught from the intended operations (gRPC calls), not from accessing response attributes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Replace deprecated license table format with SPDX identifier string and remove the deprecated 'License :: OSI Approved :: Apache Software License' classifier to resolve setuptools deprecation warnings during build.
Rename the GitHub action from setup-runner to setup-python-and-tooling for better clarity about what the action does.
- Add actions:write and checks:write permissions for better workflow control - Rename jobs and steps for improved clarity and consistency - Reorder steps to check protobuf generation before code quality - Add trunk code quality checks as separate workflow step - Standardize step naming across all jobs - Add whitespace formatting in ci-pydgraph-tests workflow
- Consolidate test.yml into ci-pydgraph-tests.yml (single source of truth) - Remove standalone trunk.yml (now called from ci-pydgraph-tests.yml) - Add code quality checks to main test workflow - Modernize cd-pydgraph release workflow to use Makefile targets - Add Python 3.14 support to test matrix - Improve step naming and organization across all workflows - Add actions:write and checks:write permissions where needed - Standardize on 'make' commands for consistency
Configure UV_PUBLISH_USERNAME and UV_PUBLISH_PASSWORD environment variables from GitHub secrets for the publish release step.
Remove attempt to call reusable workflow as a step action, which is invalid syntax. Trunk checks are already handled by pre-commit hooks in 'make check' and by the separate Trunk Code Quality workflow.
Create separate ci-pydgraph-check-trunk workflow that uses the dgraph-io/.github trunk reusable workflow for better integration with Trunk's GitHub UI features and inline PR comments. Skip trunk checks in the main test workflow to avoid duplication.
Move code quality check job from ci-pydgraph-tests.yml into new ci-pydgraph-code-checks.yml workflow for better separation of concerns and clearer CI organization. Update permissions in test workflow to only request what's needed.
Rename ci-pydgraph-code-checks.yml to ci-pydgraph-code-quality.yml for better naming consistency.
Rename ci-pydgraph-check-trunk.yml to ci-pydgraph-trunk.yml and update naming for consistency with other workflows.
- Rename workflow to match ci-pydgraph-* naming convention - Swap job name for consistency - Reorder steps to run quality checks before protobuf verification - Fix typo in cd-pydgraph.yml protogen command
The test target was depending on protogen which requires Python 3.13+, causing test failures on Python 3.9-3.12 and 3.14. Since protobuf files are committed to the repo and verified in the code quality workflow, tests don't need to regenerate them.
Add explicit 'make sync' step to ensure Python dependencies are installed before running protogen and quality checks. Also reorder steps to check protobufs before running full quality checks for faster feedback.
Ensure all workflow jobs have separate 'make setup' and 'make sync' steps before running any other make commands. This ensures Python dependencies are properly installed in all workflows. Updated workflows: - ci-pydgraph-tests.yml (both test jobs) - cd-pydgraph.yml (release job)
Rename environment variable for clarity to indicate it controls installation of missing dependencies.
Document the INSTALL_MISSING_DEPS environment variable in the Development section to help users understand how to enable automatic installation of missing dependencies.
- Add 'publish' target for releasing to PyPI using uv publish - Update test target to only require deps-uv instead of all deps - Add publish to .PHONY list
Add detailed contributing guidelines covering: - Development setup and prerequisites - Code style standards and file headers - Testing requirements and procedures - Pull request process and requirements - Conventional commit message format - Requirement for __author__ and __maintainer__ in new Python files References PULL_REQUEST_TEMPLATE.md for project-specific requirements.
…ders - Add SPDX header requirements to CONTRIBUTING.md file headers section - Migrate detailed development content from README.md to CONTRIBUTING.md: - Setting up environment - Syncing dependencies - Regenerating protobufs - grpcio version requirements - Test infrastructure details - Replace README.md Development section with concise Contributing section - Update README.md table of contents All new Python files must now start with: # SPDX-FileCopyrightText: © 2017-2026 Istari Digital, Inc. # SPDX-License-Identifier: Apache-2.0
- Copy CODE_OF_CONDUCT.md from kgms (most recently updated) - Update CONTRIBUTING.md to reference CODE_OF_CONDUCT.md instead of defining code of conduct inline - Follows Contributor Covenant standard used across dgraph-io projects
- Change default instruction from 'INSTALL_MISSING_DEPS=true make setup' to plain 'make setup' - Add note explaining INSTALL_MISSING_DEPS option for auto-installing missing tool dependencies - Makes the quickstart simpler while still documenting the auto-install option for users who want it
- Accurately describe what 'make setup' does: checks tools and installs hooks - Add separate 'make sync' step for Python environment and dependencies - Clarify the two-step process: setup checks tools, sync installs dependencies - Previous description incorrectly implied setup did everything
Add "Makefile Commands" section showing make help output with all available targets and their descriptions. This gives contributors a quick reference to common development tasks before they start setup.
- Update Makefile so 'make setup' runs 'make sync' as its final step - Simplifies contributor workflow to a single command - Update CONTRIBUTING.md to reflect that 'make setup' now does everything: checks tools, installs hooks, and syncs environment - Remove separate 'make sync' instruction from setup steps
Change "To automatically install missing" to "To automatically install any missing" for better clarity.
Rename environment variable for better clarity - it installs missing tools (uv, trunk, docker), not dependencies. Updated in: - Makefile - CONTRIBUTING.md - README.md - All GitHub workflow files
- Correct clone instructions to show forking first - Add upstream remote setup - Use plain 'make setup' for consistency with CONTRIBUTING.md - Keep brief as a teaser with clear link to full instructions - Removes duplication while maintaining helpful quick reference
Remove quick start instructions from README and just link to CONTRIBUTING.md for all contribution guidelines. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
- Add blank lines around fenced code blocks (MD031) - Add language specifiers to code blocks (MD040) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Update SPDX copyright headers across all Python files to reflect 2026. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Comprehensive modernization of the pydgraph project with improved tooling, strict type safety, enhanced CI/CD infrastructure, and better developer experience. This PR adds type annotations to the entire codebase, migrates to modern Python tooling (uv, ruff, ty), establishes robust testing infrastructure, and provides comprehensive contribution guidelines.
Review Notes: While the overall diff is large (23k+ additions, 20k- deletions across 68 files), the 86 commits are organized logically around specific themes. The bulk of changes fall into five categories: type annotations, tooling setup, CI/CD infrastructure, documentation, and code quality fixes. Each commit follows conventional commit format with clear descriptions.
Key Changes
🛠️ Project Tooling & Setup
uv.lockfor reproducible buildsuv runanduv syncmake setup- One-command project setup (installs tools, hooks, syncs deps)make check- Runs all pre-commit hooks (ruff, mypy, ty, trunk, shellcheck)make test- Runs test suite with Docker infrastructuremake protogen- Regenerates protobuf files with mypy stubsmake build/make publish- Build and publish releasesINSTALL_MISSING_TOOLS=trueflag✨ Type Safety & Code Quality
client.py,async_client.py,txn.py,async_txn.pyclient_stub.py,async_client_stub.pyutil.py,errors.py,convert.pyprotogen.pydisallow_untyped_defs = truedisallow_incomplete_defs = truecheck_untyped_defs = truefrom __future__ import annotationsfor forward referencesX | Noneinstead ofOptional[X]-> Nonefor all void functionspyproject.tomlwith sensible defaults🧹 Code Quality Improvements
# noqafor intentional public API (open,ConnectionError)raiseto preserve stack tracesrandom.choice()→secrets.choice()contextlib.suppress()for claritymatchparameters to 20+pytest.raisescallspytest.raises__all__pydgraph.openfunction🔄 CI/CD Infrastructure
Separated workflows for better organization:
ci-pydgraph-tests.yml - Matrix testing across Python versions
setup-python-and-toolingcomposite actionmake setupandmake syncstepsci-pydgraph-code-quality.yml - Code quality checks
make check- All pre-commit hooks (ruff, mypy, ty, trunk, shellcheck)SKIP=trunk-check,trunk-fmtto avoid duplication with trunk workflowci-pydgraph-trunk.yml - Trunk code quality checks
cd-pydgraph.yml - Release workflow
uv versionanduv publishcommandsShared infrastructure:
🐳 Testing Infrastructure
scripts/local-test.shDGRAPH_IMAGE_TAGfor testing different Dgraph versions🔄 Dependency Updates
>=4.23.0,<7.0.0>=1.65.0,<2.0.0(loosened from strict 1.65.1)mypy>=1.14.1,ty>=0.0.8,grpc-stubs>=1.53.0.6types-grpcio>=1.0.0,types-protobuf>=6.32.1ruff>=0.8.4pytest>=8.3.3,pytest-asyncio>=0.23.0pre-commit>=3.5.0,shellcheck-py>=0.10.0.1🐍 Python Version Support
📚 Documentation
make setupuv run pythonuv runworkflow📦 Project Metadata
license = "Apache-2.0"__author__and__maintainer__in 18+ Python files>=3.9🔧 Protobuf Generation
scripts/protogen.py:api_pb2.py,api_pb2_grpc.py- Protobuf implementationsapi_pb2.pyi,api_pb2_grpc.pyi- Type stubs for better IDE supportBreaking Changes
Backwards Compatibility
Benefits
make setup), clear documentationTesting
Tool References