Scripts issue post v4.0.2 upgrade

[cruse1977]

Deployment Type

Self-hosted

NetBox Version

4.0.2

Python Version

3.12

Steps to Reproduce

This bug occurs with a 3.7 -> 4.x upgrade and cannot be resolved without removing affected scripts. It only occurs when the upgrade occurs with missing scripts.

Replicate empty 'file' environment whereas DB has been restored but files NOT:

• Create Script using git datastore
• Manually remove file from netbox scripts dir ; breaking things and replicating an unsynced environment

Go to Script:

• Expected: NetBox 3.7: 'error: missing file'.
• Observed: Missing file

Run Upgrade to 4.0.2 - upgrade succeeds

Go to scripts -

• expected: runnable (if data source has synced)
• Observed: unable to run scripts, run button missing, clicking edit yields to empty form

Sync datasource (UI, or manage.py syncdatasource --all) manually

• Expected: script recreated, script can run
• Observed: no change in behaviour as above - also no files created in scripts/ directory

REPO used: https://github.com/cruse1977/test-scripts

Expected Behavior

• Following a git sync scripts should become runnable and exist in the /script directory

Observed Behavior

• following a git sync (successful), no change in behaviour; no runs button, no files created in /scripts; editting the script gives an empty form with no pre-populated items (ie: the data source). Manually copying the affected script to /scripts also does not resolve this.

[jeffgdotorg]

@cruse1977 thanks for the report and the example script.

[llamafilm]

I ran into this problem because my CD dev pipeline tries to launch a new VM, restore the latest database backup locally, install the new Netbox, then run upgrade.sh. It's a catch-22 because you can't run syncdatasource before the venv is created; And if you create the venv first by running upgrade.sh then 0109_script_model.py does not properly update the scripts.

[cruse1977]

Did some digging on this - this look in part to be due to somewhere in the migration and creation of the ScriptModule objects; the data source being blanked but this is intermittent:

• NetBox 3.7 - Create git data source; sync; add script
• Remove script file from /scripts directory
• Run Upgrade

Upgrade 1: - migration creating empty data source ID

print(ScriptModule.objects.all()[0].dict)

{'_state': <django.db.models.base.ModelState object at 0xffffa3fc1250>, 'id': 1, 'data_source_id': None, 'data_file_id': None, 'data_path': '', 'auto_sync_enabled': False, 'data_synced': None, 'created': datetime.datetime(2024, 5, 22, 23, 50, 10, 653661, tzinfo=datetime.timezone.utc), 'last_updated': None, 'file_root': 'scripts', 'file_path': 'NewSiteScript.py'}

Upgrade 2: - migration correctly preserving data source id

{'_state': <django.db.models.base.ModelState object at 0xffffb4755510>, 'id': 2, 'data_source_id': 1, 'data_file_id': 1, 'data_path': 'NewSiteScript.py', 'auto_sync_enabled': True, 'data_synced': datetime.datetime(2024, 5, 23, 0, 18, 57, 940416, tzinfo=datetime.timezone.utc), 'created': datetime.datetime(2024, 5, 23, 0, 15, 51, 102798, tzinfo=datetime.timezone.utc), 'last_updated': None, 'file_root': 'scripts', 'file_path': 'NewSiteScript.py'}

When the data source is preserved, running an 'edit' then save, seems to sync the data source, updates the is_executable flag which allows the script to run - but its not available until this is done.

[cruse1977]

I ran into this problem because my CD dev pipeline tries to launch a new VM, restore the latest database backup locally, install the new Netbox, then run upgrade.sh. It's a catch-22 because you can't run syncdatasource before the venv is created; And if you create the venv first by running upgrade.sh then 0109_script_model.py does not properly update the scripts.

I think there's a few things going on here; one is the migration sometimes blanking the data source; the other is when the sync occurs within NetBox. See my other post on this.