v2.2.0

The major update for this release is a complete overhaul of how resources are allocated and tracked in the pilot. Multiple gpus can now be allocated, time is a first-class resource, and cpu and memory measurements are a moving average.

Enhancements:

• #167: track resource usage all the time
• #186: pilot queueing - large slots
• #187: only contact the server through the pilot
• #188: exe_json.stillrunning error

Bug Fixes:

• #118: pilot gpu allocation
• #164: updated resource requirements
• #185: hostname lookup on submit
• #189: gpu detection - empty string

v2.1.1

Several improvements to the website, and a number of bugs fixed.

Not noted below is a general improvement in the order tasks are queued.

Enhancements:

• #138: UI Improvements
• #168: submit page layout
• #170: submission success action
• #172: submission macros

Bug Fixes:

• #151: cannot repeat submit after failure
• #152: submit number of jobs - moving right or left
• #166: Quotes not escaped in displayed JSON
• #169: edit dataset config

v2.1.0

Major change here is the removal of the internal RPC and consolidation into a single process. Futures on a threadpool executor handle any i/o waits, while everything else runs on the Tornado event loop.

IceProd is fully stable and has started running production.

Enhancements:

• #110: website - dataset task progress
• #124: remove internal RPC in the server
• #142: better getInterfaces
• #148: clean dagtemp
• #149: psutil oneshot context manager
• #153: jsonrpc spec and positional args
• #155: Put the settings table in a separate sqlite DB

Bug Fixes:

• #147: tasks_processing is a lie
• #154: nginx still running if startup error
• #156: master queueing times out
• #159: mysql connection lost
• #161: master gridftp proxy

Known Issues:

• #164: updated resource requirements
• #166: Quotes not escaped in displayed JSON

v2.0.6

Cleanups all around. This should be the last release in the 2.0 series.

With this release, IceProd is "stable" (sort-of, maybe). It can run a dataset all the way through without major errors, but it does it slowly.

Enhancements:

• #61: external monitoring support
• #67: upload/download
• #139: basic submit resource requirements
• #141: replace pycurl with requests

Bug fixes:

• #140: disk space violations
• #143: home page graphs take forever to load
• #145: handle integer resource amounts when updating requirements

Known issues:

• #147: tasks_processing is a lie

v2.0.5

The major change in this release is resource overusage in the pilot. Tasks can now use more resources than requested if the pilot has them available.

Pilots can also now run more than one task. This was enabled in v2.0.4, and successfully proven in this release.

As of this release, documentation is automatically built every commit:
https://wipacrepo.github.io/iceprod/

Enhancements:

• #79 - multi-job pilot

Bug fixes:

• #127 - rpc_new_task returing non-matching tasks
• #121 - multiprocess webserver
• #115 - tasks complete goes backwards

v2.0.4

It's mostly stable!

The major change in this release is pilot resource monitoring.

Enhancements:

• #69 - better way of selecting resources on submit
• #114 - pilot log separation
• #117 - pilot process labelling
• #81 - individual task resource usage
• #82 - live resource usage
• #108 - show whole logs

Bug fixes:

• #120 - config validation
• #119 - pilot processes taking 100% cpu time
• #72 - handle expiration of self-signed cert for webserver

Other:

• #94 - var/log/out and var/log/err
• #70 - remove site_id from db/settings table
• #112 - web authentication and mixed public/private pages
• #113 - web login
• #109 - master update individual failure

v2.0.3

Bug fixes:

• #103 - reset tasks when pilot dies
• #74 - dataset submit failure message
• #73 - clear error and output files between tasks on a pilot

Enhancements:

• #98 - master task reset
• #76 - write ahead logging for SQLite
• #68 - task reset from website

Other:

• #102 - sort parameters in submit json
• #63 - remove references to unittest2

v2.0.2

Multi-site seems stable, so let's make a release.

v2.0.1

• Make json 'human readable'
• fix setup.py script
• fix passing arguments to curl
• fix a bunch of tests

2.0 Milestone

First release of IceProd 2.