Improve usage.json schema

[asmacdo]

Fixes #109

{
  "timestamp": "2024-06-22T12:05:35.337336-05:00",
  "num_samples": 1,
  "processes": {
    "1243562": {
      "pcpu": 0.0,
      "pmem": 0.0,
      "rss": 1760,
      "vsz": 221584,
      "timestamp": "2024-06-22T12:05:35.337336-05:00"
    }
  },
  "totals": {
    "pmem": 0.0,
    "pcpu": 0.0,
    "rss_kb": 1760,
    "vsz_kb": 221584
  },
  "averages": {
    "rss": 1760,
    "vsz": 221584,
    "pmem": 0.0,
    "pcpu": 0.0,
    "num_samples": 1
  }
}
{
  "timestamp": "2024-06-22T12:05:36.355308-05:00",
  "num_samples": 8,
  "processes": {
    "1243562": {
      "pcpu": 0.0,
      "pmem": 0.0,
      "rss": 1760,
      "vsz": 221584,
      "timestamp": "2024-06-22T12:05:36.355308-05:00"
    }
  },
  "totals": {
    "pmem": 0.0,
    "pcpu": 0.0,
    "rss_kb": 1760,
    "vsz_kb": 221584
  },
  "averages": {
    "rss": 1760.0,
    "vsz": 221584.0,
    "pmem": 0.0,
    "pcpu": 0.0,
    "num_samples": 8
  }
}

[asmacdo]

Marked as "draft" (and leaving the tests failing) until I'm done with #106 but review welcome

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.00%. Comparing base (25eb0f3) to head (80256d5).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #110      +/-   ##
==========================================
- Coverage   93.60%   93.00%   -0.61%     
==========================================
  Files           2        2              
  Lines         344      343       -1     
  Branches       57       56       -1     
==========================================
- Hits          322      319       -3     
- Misses         16       18       +2     
  Partials        6        6              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[asmacdo]

to be merged (and hopefully auto-released) after #113

[yarikoptic]

yes, this looks cleaner.

I guess unrelated -- what is "totals"? aren't they "max" through the time so we could see how much needed? Where is total wallclock run time (in seconds)? is there a way to get total cpu run time as well?

[asmacdo]

"max through time" is not recorded in usage.json. The usage file contains nothing about the full run, it is only (aggregated) samples.

The "totals" in this case are total of all processes. ie, if the inner command forks and is running multiple processes simultaneously, the totals will report on the command as a whole, rather than per process.

The total wallclock run time is recorded in the info.json under execution_summary which should not be confused with the "totals" of a single sample. Since we are recording wall clock time and average cpu utilization, do we need cpu time? If so, lets file a separate issue.

info.json

{
  "command": "sleep 5",
  "system": {
    "uid": "austin",
    "memory_total": 33336778752,
    "cpu_total": 20
  },
  "env": {},
  "gpu": null,
  "duct_version": "0.0.1",
  "execution_summary": {
    "exit_code": 0,
    "command": "sleep 5",
    "logs_prefix": "ZZZZ_",
    "wall_clock_time": "5.003 sec",
    "peak_rss": "1920 KiB",
    "average_rss": "1920.000 KiB",
    "peak_vsz": "221584 KiB",
    "average_vsz": "221584.000 KiB",
    "peak_pmem": "0.0%",
    "average_pmem": "0.000%",
    "peak_pcpu": "0.0%",
    "average_pcpu": "0.000%",
    "num_samples": 4,
    "num_reports": 3
  }
}