This change log spiritually adheres to these guidelines.
Categories for each release: Added, Changed, Deprecated, Removed, Fixed, Security
dx find jobs --json
, anddx describe job-xxxx
with--verbose
argument return new field: internetUsageIPs if the caller is in an organization that has enabled jobInternetUsageMonitoring license feature
- Initial support for Nextflow
- Do not check python3 syntax with python2 and vice versa in
dx build
dx build
properly verifies the applet's name given in theextra-args
parameter
dx extract_dataset
command- Optional pandas dependency for dxpy
dxpy.find_one_project
,dxpy.find_one_data_object
,dxpy.find_one_app
raiseDXError
ifzero_ok
argument is not abool
--head-job-on-demand
argument fordx run app(let)-xxxx
--head-job-on-demand
argument fordx-jobutil-new-job
--on-behalf-of <org>
argument fordx new user
- dx-toolkit never included in execDepends when building app(lets) with
dx build
--no-dx-toolkit-autodep
option for dx build
- Reduce the number of API calls for
dx run applet-xxxx
anddx run workflow-xxxx
dx upload f1 f2 --visibility hidden
now correctly marks both files as hiddendx upload
retry on all types of SSL errors
- Parsing ignoreReuse in
dx build
of workflow
- DXHTTPRequest to pass ssl_context
- '--rank' argument for
dx run
- Do not use job's workspace container ID in /applet-xxxx/run for detached jobs
dx describe
of executable with bundledDepends that is not an asset- Building globalworkflow from existing workflow with
dx build --from
- Improvements to symlink downloading reliability by solely using
aria2c
and enhancing options around its use (removeswget
option for downloading symlinked files, adds the ability to set max tries for aria2c, adds-c
flag for continuing downloads, removes the--check-certificate=false
option). dx build
comparison of workflow directory to workflow name- Set project argument for
dx run --detach
when executed from inside a job
- Removed
wget
option for downloading symlinked files - Bump allowed requests dxpy dependency version to 2.27.1
- New argument
symlink_max_tries
fordxpy.download_dxfile()
with default value of 15
- Do not list folder contents to speed up
dx cd
- API wrappers for
dbcluster
- Pin websocket-client to 0.54.0 to fix
dx watch
output to include job output - Do not install pyreadline on Windows with Python 3.10
- KeyError in
dx-app-wizard --json
- dxjava dependencies log4j2, jackson-databind
- Python 3.10 collections imports
- Recursive folder download
dx download -r
of folders with matching prefix
- Incorrect setting of the
folder
input option when building global workflows - Remove unused match_hostname urllib3 import
- Support for qualified workflow & applet IDs and paths when using
dx build --from
with an applet/workflow - Setting properties when building global workflows
- '--allow-ssh' parameter to
dx ssh
- '--no-firewall-update' parameter to
dx ssh
- Detect client IP for SSH access to job instead of
*
- Python 3.10 MutableMapping import
--no-temp-build-project
for single region app builds.--from
option todx build
for building a global workflow from a project-based workflow, including a workflow built using WDL
- Reduce file-xxxx/describe API load during
dx upload
dx get
uses a region compatible with user's billTo when downloading resources
dx run
warns users if priority is specified as low/normal when using '--watch/ssh/allow-ssh'
- Support for dxpy on macOS arm64
- Path input for
dx list database files
- Python 3 SSH Host key output in
dx describe job-xxxx
- dxpy dependencies cryptography, websocket-client, colorama, requests
- No significant changes
- Support FIPS enabled Python
dx archive
anddx unarchive
commands
dx upload
part retry where file would stay in an open statedx run <globalworkflow> --project/--destination/--folder
now submits analysis to given project or path
- '--cost-limit' arg for
dx run
- '--database-ui-view-only' flag for
dx new project
Total price
fordx describe
prints formatted currency based oncurrency
metadata
- No significant changes
DX_WATCH_PORT
env var for supportingdx watch
in the job execution environment
- No significant changes
low
option for--priority
argument fordx run
- Provide job container-id when downloading bundledDepends in job execution environment
- Upgrade to proot 5.2 from udocker2 fork for
dx-docker
dx-app-wizard
default to Ubuntu 20.04
- Search for reference genome project in region
- Connection leak with HttpClient in DXFile
--brief
flag to dx-clone-asset so that script results can be used downstream
- Bump jackson-databind from 2.9.10.5 to 2.9.10.7
- xattr import in
dx-upload-all-outputs
- Added '--keep-open' flag for
dx build
- Symlink download retries when error 22 is thrown
- '--detach' flag for
dx run
- Add xattr dependency to extras_require, only install if specified
- Unused python-magic, beatifulsoup4 python dependencies
- Building assets for Ubuntu 20.04
- Increase wget retries to 20 for symlink downloads
- gnureadline macos dependency to 8.0.0 for Python versions < 3.9
- Remote builders for 16.04 v1, 20.04
- Asset builder for 20.04
- '--verbose' flag for
dx-mount-all-inputs
- Provide project-id in batch tsv file
- Archival api wrappers
- Hide
notebook
andloupe-viewier
fromdx
help output
- Macos tarball build
- Ubuntu 20.04 build targets
- jackson-databind from 2.9.10.3 to 2.9.10.5
- API wrapper generation with Python 3
dx-clone-asset
when no project exists- DXJava DXDataObject.Rename()
- Python cryptography version >= 2.3
dx-clone-asset
with Python 3
- Ubuntu 14.04 builds
- Examples for
dx find
with negative timestamp dx build --from applet-xxx
for app- --brief option to dx build for apps and applets
- Error handling during syntax check for dx build
- Python 2.7 example applets to use Python 3
- Commit dxpy version during release
dx get
for applets withpython3
interpreterdx-upload-all-outputs ---xattr-properties
parsing
- Python 2.7 example applets to use Python 3
- Commit dxpy version during release
- No significant changes
dx-mount-all-inputs
for dxfuse- Sci Linux compilation option for upload agent
- Python 3 interpreter for
dx-app-wizard
- dxR build
dx upload
retry for "SSL EOF" error- Error out for dx run --clone with analysis and executable
- runSpec.version dxapp.json key for 16.04 applets
dx build_asset
support for runSpecVersion key in dxasset.json
- Python documentation build
- dxpy tests in Python 3
- Bump jackson-databind from 2.9.10 to 2.9.10.1
- Bump jackson-databind from 2.9.10 to 2.9.10.1
- Retry symlink file downloads with
aria2c
- Python3 issue in dx app builder test
- Remote app tarball builder for xenial
- Allow disabling system exit on log client
- database class in dx find data
- Python3 compatibility for exec environment scripts
- pip3 package manager for execDepends
- Precise debian package build target
- Run job as high priority if '--ssh' provided
- Project deletion warning if specifying file-id
- New instance types to
dx-app-wizard
- Upgrade jackson to 2.9.10
- Python 3 wrapper generation and tests
dx get
for database files- v2 instance types in
dx-app-wizard
- Generating ruby wrappers in python 3
- dx-app-wizard in python 3
- Documentation links to https://documentation.dnanexus.com
- Remove non-ascii char from readme
- '--xattr-properties' argument for dx-upload-all-outputs to include fs metadata as k,v properties
- xattr dependency for Linux builds
- Only require futures package for python 2.7
- Upgrade build dependencies for pip, setuptools, and wheel
- DXJava support for proxies
- Approved tools CLI for
dx update project
- Upgrade jackson-databind and jackson-core to version 2.9.8
- Provide project ID for dx make_download_url unless in job workspace
- Enabling argcomplete for
dx
installed with debian package in worker environment
dx upgrade
downloads the latest version from s3
- Reduce the number of API calls for
dx download
dx upload
error via proxy in Azure
- support for passing HTTPContext in
DXJava
to theexecute()
operation
--instance-count
todx run
so that Spark cluster size can be defined at app start
dx wait
behavior by adding exponential backoff and passing appropriate project- Decreased libcurl timeout in C++ bindings from infinity to default 10 min
- Default Ubuntu release to 16.04 in
dx-app-wizard
- Link handling to better support JBORs
- Handling file arrays in batch job runner
- no significant updates
- new
findDataObjects
inputs to DXJava - project name resolution to
--project
flag fordx run
- smart reuse and SAML identity provider
dx list database <entity>
for DNAnexus Apollo--ignore-reuse
and--ignore-reuse-stage
working fordx run <workflow>
- Upgrade
proot
to be compatible with kernel >= 4.8 - Skip symlinks test in isolated environment
- Uploading binary data, such at compressed files, works in python3.
- python3 parsing subcommand help output
- Binary mode for opening dx:files.
- A
--unicode
flag for command line tools that need to handle unicode text files. For example:dx cat --unicode file-xxxx
. This was added forcat
, anddownload
.
- 32-bit build target for Ubuntu 14.04
gtable
code
--phi
flag fordx new project
- Bug in downloading symlinks when using aria2c
- Max number of aria2c connections <= 16
- argcomplete eval in the worker when sourcing
environment
- Preserve
httpsApp
field in dxapp.json when callingdx get
- The
--except [array:file variable]
option fordx-download-all-inputs
- upload issue using api proxy in Python 3
--no-project
option indx-jobutil-parse-link
- A script for reconnecting to a jupyter notebook session
- support for dx building a global workflow with apps in multiple regions
- Mark Ubuntu 12.04 as deprecated in
Readme
- setting instance types on global workflow stages
- fix test code for Spark 2.4.0 upgrade
- symlink download with
aria2
No significant changes
- Update MANIFEST.in to include python
Readme
- Writing to stdout for py2 in
dx cat
- Python virtualenv path in traceability runner
- Failing
build app
when suggestion is not a dxlink - Handle keyboard interrupt and system exit gracefully
- Use absolute path to set argcomplete
- If a bash variable is long, print a warning, do not omit it
- Issue with backports deps on Windows
- Make
dx-toolkit
python 2 and 3 compatible - Update macOS gnureadline version
- Allow Windows to use certifi CA bundle
- Update bindings for Apollo API routes
- Update urllib import in
dx-docker
- Update requests in
make doc
target
- Test that attempts to upload to a closed file
- First draft of an environment file for fish shell
- If available, use
aria2
rather thanwget
as it's much faster
- Use of ubuntu 12.04 in
test_dxclient.py
- Old asset building script
- Rare subcommands (
compile
,compile_dxni
,sh
) - The
dx-run-app-locally
script
jackson-databind
andjackson-core
upgrade
- Update cran repository
- Add release/distro to example app
dxapp.json
files
- Download links for
docker2aci
- The test error: No root certificates specified for verification of other-side certificates
- A convenience login script for a cloud workstation
- Add
parents
param toclone()
- Allow batch IDs as output folders for batch run
- Setting a handler for a global workflow
- Redirecting proxy printout to stderr
cat
argument too long in a bash script- Ensure we can pop items from the main dictionary (python3)
- Warn user before
dx rm -r
on root of a project - Let
urllib3
set default CA bundle on Windows - Updgrade
pyopenssl
for test suite to 17.5.0 - Replace
ws4py
with websocket-client library injob_log_client
- Pass stages input from assetDepends to bundledDepends
- Traceability ID for locked workflow test (#406)
- Python 3 incompatibilites
- Python 3
dx upload
import dxpy
whensys.stdin
isNone
dxpy.describe()
used with a list of object IDs and an additionalfields
argument
- Sort inputs in
dx generate_batch_inputs
- 12.04 builds
- Windows install Python version upgrade to 2.7.15
- Windows installation
dx run --ignore-reuse
to disable job reuse for jobignoreReuse
key in dxapp.json
- dxWDL version 0.74
- Ubuntu release and asset version as parameters for
dx-docker create-asset
- Builds for Ubuntu 16.04
dx wait
where a file with object IDs is provided in pathdx compile
for debian install dxWDL.jar filepath
- Database wrappers
- Support dxni compilation
- requests >= 2.8.0
- psutil >= 3.3.0
- Python 3 incompatibilities
- Revert of finding project for data object
- support for setting and updating
details
on global workflows - decorators for traceability tests
dx watch
support for smart reuse- test for
dx watch
job log relay dx find jobs/executions
support for smart reuse- ability to provide a file which contains a list of jobs or data objects upon which to
dx wait
dxWDL
integration (dx compile
command)
python-dateutil
version upgrade to 2.5
- unclear behavior when
--clone
,--folder
, and--project
are simultaneously provided todx run
DXObject.set_ids()
with project set to None (it is now set to an arbitrary project associated with the file)- bash helpers tests
- Cosmetic update to
dx publish
dx publish
now sets thedefault
alias on the published version by default
- Support for updating a global workflow
- Wiki links to API documentation for API wrappers
- addTags/removeTags API wrappers for global workflow
- Better checking for inputs and/or inputSpec before batch running workflows
- A workflow must be closed before initializing a global workflow from it (test updates for API changes)
- Remove the "temporary" field from
dx describe workflow-xxxx
- "Deleted" field to
dx describe globalworkflow-xxxx
- a note to
dx describe
that the workflow is locked
- Print proper dx describe analysis of a global workflow
dx publish
command- inline cluster bootstrap script
- dx run for global workflows
- dx find apps tests
- resolve paths for global workflows
- add, remove, list users for global workflows
- add, remove, list developers for global workflows
- public project test
- API tests
- Support for dx find global workflows
- Initial support for dx build global workflow
- Publish method to global workflow bindings
- Support for dx get global workflow
dx download
for symlinks
- Support for
dx describe
for global workflows
- zsh support
- API wrappers for global workflow routes
- Basic Python bindings for global workflow
set_properties()
method to DXProject
- dx get applet without execDepends
dx-clone-asset
SSL error
- Clarify documentation of stage key for
dx run
- Asset builder support for Ubuntu 16.04
--singlethread
parameter fordx upload
dx-docker pull
retries more often
- c-ares package version for upload agent build
- Bug with Azure instance type selection in
dx-app-wizard
- Do not retry code
422
in dxpy
socketTimeout
andconnectionTimeout
parameters to JAVADXEnvironment
(thanks, @pkokoshnikov)- Generate batch inputs CLI
- Accept 302 as a success for
ua --test
- Batch runner CLI
- Updated c-ares and file packages (related to upload agent builds)
- API wrappers for
[applet,app,workflow]-xxxx/validateBatch
- Issue where dx-clone-asset doesn't create projects in different regions if they don't already exist
- App version and published/unpublished note to
dx run -h <app>
- Recursive search for asset in a path is disabled, and we strictly enforce only one matching asset name
- Use twine for pypy uploads
- Error that blocks org from being added as developers
dx-docker
: cleanup of the quote code (regression fix)dx-docker
: useshutil.move()
instead ofos.rename()
with the aci image- Accept 'http_proxy', 'HTTP_PROXY', 'https_proxy', 'HTTPS_PROXY' for proxy environmental variables
- Error out instead of warning when old pricing scheme is used in
dxapp.json
- Fix certain tests flakiness
- Warning when
runSpec.release
is not specified indxapp.json
with a note it will be required in the future - Numerous enhancements to Jupyter notebook support (see https://github.com/dnanexus/dx-toolkit/commit/7ecbcb6b75118c0acd27f7a7cfe37d0a19e6d6c3 for more information)
dx-app-wizard
anddx get
specify systemRequirements inregionalOptions
and not inrunSpec
ofdxapp.json
- multiple updates to jupyter notebook support
- setting
systemRequirements
inrunSpec
ofdxapp.json
dx-configset-to-fasta
script
dx-clone-asset
sets the default regions to be all regions that user has access to and skips redundant cloning to current regiondx build
now works seamlessly across apps and applets
- Update run input help and describe messages for locked workflows
- Warn when old, top-level pricing policy scheme is used in dxapp.json
dx-gtable-to-csv
anddx-gtable-to-tsv
scriptsdx-workflow-to-applet
scriptinclude_hidden_links
parameter from clone()
- dx-toolkit and apt conflict with argcomplete
- dx-clone-asset script
dx-docker
: ignore user argument if given- TLS documentation
- app building tests now include
runSpec.release
andrunSpec.distribution
in app specs dx-docker
: better handling of quotes
- New InvalidTLSProtocol Exception raised when connection fails due to wrong TLS protocol.
- Remove rstudio option for
dx notebook
- dx-toolkit and apt conflict with jq
- Azure instance types to the list of available types in
dx-app-wizard
- ua -- test now displays system messages coming from the apiserver.
- Update references to workflow
inputs
andoutputs
to keep them in sync with API changes
- No significant changes
- Priority arg to
build_asset
- Pass region-specific pricing policy in dxapp.json
- Execution of old workflows built without explicit IO
- CLI support for workflow lockdown
- Deprecated
dx-mount
- Initial 'dx notebook' commit
- Python bindings for workflow-level input and output
- Support for the 'downloadRestricted' flag
- Default to 14.04 release for all instance types in
dx-app-wizard
- No significant changes
- Point release to fix release version issues
- dx-jobutil-new-job now properly supports instance type
- Installation instructions now centralized on Github page
- Incrementally upgraded dependencies for Java bindings
- Helper script to check TLS 1.2 support
- A
region
parameter can now be passed todx find projects
,dx find data
,dx find org projects
- No significant changes
- A
region
parameter can now be passed todxpy.find_data_objects()
anddxpy.find_projects()
dx-docker
now no longer bind mounts$HOME
and/tmp
into the guest as this is consistent with Docker
- Python 3 compatibility with
dx-app-wizard
dx get
does not redundantly inline description and developerNotes in dxapp.json any more
- Client support for partial folder deletion
- Add methods
__next__()
andnext()
to DXFile to complete iteration interface (thanks to Eric Talevich!)
--bill-to
option is utilized when building multi-region apps withdx build
- Mac install no longer uses virtualenv. Instead, we ask users to install desired version of Python
- dx-docker bug where environment variables passed in through the CLI do not get set within container
dx build
creates a workflow on the platform based on the workflow's local source directory
- the deprecated
dx-build-app
anddx-build-applet
scripts (that were replaced withdx build
) were removed
- Bug introduced in release 204.0: including app resources fails
- Ability to specify additional resources for multi-region apps
dx ls -l
and friends now request only needed describe fields
- Ability to specify bundledDepends and assetDepends for multi-region apps
- Use DNAnexus DockerHub repo for dx-docker tests
- Issue where selecting from exactly 10 projects resulted in a stacktrace error
dx get workflow-xxxx
creates a local representation of a workflow
No significant changes.
- Python 3 compatibility with
repr
import
No significant changes.
- dx-docker test suite to toolkit
- Retry download of Docker image if there is transient network failure
- Allow image ID as a parameter for
dx-docker
; see wiki documentation for more details
- missing pyasn1 module for OSX list of install dependencies; gets rid of import warning messages
- Alternative export for
dx-docker
when docker image is improperly exported by docker engine
dx run -i=filename
now prompts user to select input file if duplicate filenames exist in projectdx-docker create-asset
now supports output pathdx download
failure when run within project that user has lost access todx build -f
now removes all applets with matching name/directory
dx-docker run
KeyError when docker image is built from container
- Recursive file upload on Windows for the Upload Agent
- Show download progress for calls to
dx download -r
- Issue where calls to
dxpy.download_all_inputs(parallel=True)
hang
- Upload agent now does not gzip compress .gz files on Ubuntu 14.04
- Minor log message fix if file is already uploaded
- Mark routes as retryable for those that support idempotent calls (e.g. creating a new file)
- High-level GTable bindings
No significant changes.
- Fix
dx get
untar issue with leading / - Missing
dx-verify-file
andjq
dependencies on Windows
- Support to build of upload agent on RHEL7
- Ability to build and get multi-region apps with custom system requirements
- Environment file so that
source environment
works properly on RHEL7 - Modified
dx-docker
so that--rm
passes through gracefully - Modified
dx-docker
so that theHOME
environment variable defaults to/root
- bug where
dx download
incorrectly interpreted the overwrite option during argument parsing
No significant changes.
- Workflow handler to the Java bindings
- Checksum verification for file downloads in Java
- Bug resulting in transient errors on large downloads
dx-app-wizard
now correctly specifiestimeoutPolicy
dx-docker
now handles default working directory and override properly
- On Mac OS dx-toolkit now supports TLS 1.2 by activating a virtualenv
dx-app-wizard
now defaults to Ubuntu 14.04 as opposed to 12.04- Cosmetic improvements to CLI
describe
andls
subcommands
- Perl and Ruby bindings are no longer supported and associated code has moved to the
dx-toolkit/contrib
directory