From 937eb2cae720656d741d5ab26a68d3ade348d046 Mon Sep 17 00:00:00 2001 From: "C. Benjamins" <75323339+benjamc@users.noreply.github.com> Date: Thu, 16 May 2024 15:11:43 +0200 Subject: [PATCH] Version 2.1.0 (#1108) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Hacking in the sum cost for MO * Fix mypi * Bugfix * Make mypi happy * Fix tests * Make format * Remove num_obj in MO optimizer * A lot of restructuring * Solve importings * Solve importings * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Updated changelog * Still fixing :) * fix docstrings * Change directory of tests * Fix ambiguous variable * Del newlines * fix mypy * mypy and doc fix * Removed int type checking * fix tests * Always return List of incs * fix path issue in example * maint boing facade * Better docstring * Return values if no bounds passed * Removed import * Return only one incumbent * Rename n_optimizers -> n_workers * Update example * Update docstring * Remove legacy function * Update to new signature * Update changelog.md * Format black * Update test * Please pre-commit * Merge development * Fix tests * Fix typos * Format black * Updated text * Increased version * Improved readme * SMAC logo 50% * SMAC logo 50% * Removed dev-2.0 * Added doc workflow for latest tag * Fixing docs latest tag * Fixing docs latest tag * First steps new intensifier * Make it work again * MOVE random weight initialization to smbo ask method * ADD enable parego in unittests * Test issues * FIX formatting * Link to old documentation * MOVE random weight initialization to smbo ask method * ADD enable parego in unittests * FIX formatting * Fix unit tests * Add annotation * corrected log prob for tophat prior * Removed configurations from callback * Fixed tests * Fix mypy * Save docs as latest version too * Enabled all tests again * Make version an env * Added last version to docs * Global variable fix * Added v before version number * New Intensifier Implementation (#888) - Completely reimplemented the intensifiers (including Successive Halving and Hyperband): All intensifiers support multi-fidelity, multi-objective and multi-threading by nature now. - Expected behaviour for ask-and-tell interface ensured (also for Successive Halving). - Continuing a run is now fully supported. - Added more examples. - Updated documentation based on new implementation. * First benchmark steps * Added a warning * Set xi to 0.01 * Incorporated walltime in trajectory * Working version of benchmark * Changed requirements * Changed xi back to 0.0 again * No duplicate budgets * Fixed bug which updated tracker wrongly * Bugfix: Delete correct bracket+stage * Fixed tests * Added costs to the trajectory * Changed initial design max ratio to 0.25 * Make format * Added more details to intensifier * Finished benchmark * Added benchmark source files * Fixed tests because of initial design changes * Improved runhistory API * Fixed typo (#894) * Merge doc fixes * Doc fixes (#893) Co-authored-by: timruhkopf Co-authored-by: Carolin Benjamins Co-authored-by: dengdifan Co-authored-by: Alexander Tornede Co-authored-by: Sarah Krebs * Make format * Satisfied mypy * Enabled more tests * Increased version * Mypy * Mypy * Mypy * Fixed critical bug when using highest budget for inc selection * Added SH+HB comparison * Stuff * Updated report * Fixed docstring * Fixed image path * Removed vscode folder * Bugfixes * Doc fixes * Updated changelog * Make isort * Fix typos * Removed new line * Added alpha2 to docs selection * update description * Fix 531 correctly create integers in initial design This PR fixes a bug in which integer hyperparameters would not be created correctly in the initial design. This went unnoticed, except in the case where it was part of a condition and the function deactivate_inactive_hyperparameters was called. This PR fixes this bug by mapping the random float that was sampled by, for example and LHD or Sobol sequence, to the float representation of an integer by calling _transform and _inverse_transform of the integer hyperparameter. * Add contact paragraph to README. * Increase days until stale issues are closed by stalebot (#904) * Edit changelog * Add issue number * Closes #907 * Clarify origin of configurations (#908) * Random forest speed up (#903) Co-authored-by: René Sass * Add: workflow to update pre-commit versions (#874) Co-authored-by: René Sass * Add Git-Flow Workflow Specification * [DOC] acquisition function maximizers * Add stale reminder email action * Increase version number of ConfigSpace * Add guideline for updating copyright notice years to CONTRIBUTING.md * Update requirement for pyrfr (#920) * Update pre-commit: isort,flake8 version numbers * Format black * Fix mypy * Ac bench (#929) * Add Branin AC Task * Add ac optimization type * Nicer plotting * Add ac mode * Update ac task * Make new smac env name adjustable * Increase python version for smac 1.4 * Update benchmark * Update plotting * Goodby comments * Fix path * Fix 1.4 wrapper * Update benchmark results --------- Co-authored-by: Carolin Benjamins * [DOC] added documentation on the incumbent update * Feature/add version to make publish (#931) * Add version, correct package name and use bash in makefile * Increase version * Increase version in makefile, add places with versions to make publish * Fix typo * Transform integer feature names to strings (#930) * Transform int features to str * Fix pre-commit * Update version to 2.0.0 * Update version to 2.0.0 * Update CHANGELOG.md * Add information on deepcave to the readme * Correct doc version * fix(runhistory): set id of the sampled config (#951) * fix: Set config_id of config in runhistory * ci: Set dependabot to target development branch (#963) * feat: Option preventing SMAC setting up logging (#947) * Version 2.0.0 (#898) * Update runhistory2epm.py * pre_commit * Updated changelog * Fixed comment * Updated changelog * Some small improvements * Removed trailing space * Updated changelog * Restricted pynisher * Updated changelog * Link JMLR paper * Update bibtex entry Before: arxiv paper, now journal article. * Fixed citation in docs * Fixed callback typos * BOinG for mixed hyperparameter search space * fix format * maint * maint documentation * maint pre-commit * fix workflow * maint * Add dependabot for workflow files * Bump actions/setup-python from 2 to 4 (#849) Bumps [actions/setup-python](https://github.com/actions/setup-python) from 2 to 4. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](https://github.com/actions/setup-python/compare/v2...v4) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump actions/checkout from 2 to 3 (#850) Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v2...v3) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * maint * Updated stale bot * Updated stale options again * Fixed typing * Increased version * rename check_point_in_ss to check_subspace_point * Make format * Uncomment test procedure * docstrings for boing facade * avoid typing namespace everywhere in epm_configuration_chooser * fix docstring in rh2epm_boing * Limit on dask * Trying to fix tests * Replaced typing in all files * docstring for boing subspace * Changed feature requests to feature * allow acq_optimizer_local_kwargs to be passed to boing subspace without acq_optimizer_local * fix pre-commit * Added any * Update facade * Fix typing * Match arguments from pSMAC with SMAC4AC signature * Add description of pSMAC to docstring * Add option to pass facade class to pSMAC * Add example for pSMAC * Removed docstring in readme's example * Corrected text * move construct_gp_kernels to smac.epm_gpytorch.boing_kernels * maint * fix wrongly imported func name in tests * Add reference for pSMAC #609 * Rename example file * Added roadmap to readme * Prettification * Add result merger * Reformat with black * Delete unnecessary import * Fix typing * Inject first trajectory entry from file from first rundir * Add test for ResultMerger * Fix docstring * Get runhistory and trajectory from pSMAC run Via the result merger. Searchers rundirs and merges found runhistories and calculates a new trajectory. * Get trajectory * Add regex to requirements * Fixed MO cache problem * Fix MO tests * Fixing MO in AbstractRacer * Accepting ints * Hacking in the sum cost for MO * Fix mypi * Bugfix * Make mypi happy * Fix tests * Make format * Remove num_obj in MO optimizer * A lot of restructuring * Solve importings * Solve importings * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Updated changelog * Still fixing :) * fix docstrings * Change directory of tests * Fix ambiguous variable * Del newlines * fix mypy * mypy and doc fix * Removed int type checking * fix tests * Always return List of incs * fix path issue in example * maint boing facade * Better docstring * Return values if no bounds passed * Removed import * Return only one incumbent * Rename n_optimizers -> n_workers * Update example * Update docstring * Remove legacy function * Update to new signature * Update changelog.md * Format black * Update test * Please pre-commit * Merge development * Fix tests * Fix typos * Format black * Updated text * Increased version * Improved readme * SMAC logo 50% * SMAC logo 50% * Removed dev-2.0 * Added doc workflow for latest tag * Fixing docs latest tag * Fixing docs latest tag * First steps new intensifier * Make it work again * MOVE random weight initialization to smbo ask method * ADD enable parego in unittests * Test issues * FIX formatting * Link to old documentation * MOVE random weight initialization to smbo ask method * ADD enable parego in unittests * FIX formatting * Fix unit tests * Add annotation * corrected log prob for tophat prior * Removed configurations from callback * Fixed tests * Fix mypy * Save docs as latest version too * Enabled all tests again * Make version an env * Added last version to docs * Global variable fix * Added v before version number * New Intensifier Implementation (#888) - Completely reimplemented the intensifiers (including Successive Halving and Hyperband): All intensifiers support multi-fidelity, multi-objective and multi-threading by nature now. - Expected behaviour for ask-and-tell interface ensured (also for Successive Halving). - Continuing a run is now fully supported. - Added more examples. - Updated documentation based on new implementation. * First benchmark steps * Added a warning * Set xi to 0.01 * Incorporated walltime in trajectory * Working version of benchmark * Changed requirements * Changed xi back to 0.0 again * No duplicate budgets * Fixed bug which updated tracker wrongly * Bugfix: Delete correct bracket+stage * Fixed tests * Added costs to the trajectory * Changed initial design max ratio to 0.25 * Make format * Added more details to intensifier * Finished benchmark * Added benchmark source files * Fixed tests because of initial design changes * Improved runhistory API * Fixed typo (#894) * Merge doc fixes * Doc fixes (#893) Co-authored-by: timruhkopf Co-authored-by: Carolin Benjamins Co-authored-by: dengdifan Co-authored-by: Alexander Tornede Co-authored-by: Sarah Krebs * Make format * Satisfied mypy * Enabled more tests * Increased version * Mypy * Mypy * Mypy * Fixed critical bug when using highest budget for inc selection * Added SH+HB comparison * Stuff * Updated report * Fixed docstring * Fixed image path * Removed vscode folder * Bugfixes * Doc fixes * Updated changelog * Make isort * Fix typos * Removed new line * Added alpha2 to docs selection * update description * Fix 531 correctly create integers in initial design This PR fixes a bug in which integer hyperparameters would not be created correctly in the initial design. This went unnoticed, except in the case where it was part of a condition and the function deactivate_inactive_hyperparameters was called. This PR fixes this bug by mapping the random float that was sampled by, for example and LHD or Sobol sequence, to the float representation of an integer by calling _transform and _inverse_transform of the integer hyperparameter. * Add contact paragraph to README. * Increase days until stale issues are closed by stalebot (#904) * Edit changelog * Add issue number * Closes #907 * Clarify origin of configurations (#908) * Random forest speed up (#903) Co-authored-by: René Sass * Add: workflow to update pre-commit versions (#874) Co-authored-by: René Sass * Add Git-Flow Workflow Specification * [DOC] acquisition function maximizers * Add stale reminder email action * Increase version number of ConfigSpace * Add guideline for updating copyright notice years to CONTRIBUTING.md * Update requirement for pyrfr (#920) * Update pre-commit: isort,flake8 version numbers * Format black * Fix mypy * Ac bench (#929) * Add Branin AC Task * Add ac optimization type * Nicer plotting * Add ac mode * Update ac task * Make new smac env name adjustable * Increase python version for smac 1.4 * Update benchmark * Update plotting * Goodby comments * Fix path * Fix 1.4 wrapper * Update benchmark results --------- Co-authored-by: Carolin Benjamins * [DOC] added documentation on the incumbent update * Feature/add version to make publish (#931) * Add version, correct package name and use bash in makefile * Increase version * Increase version in makefile, add places with versions to make publish * Fix typo * Transform integer feature names to strings (#930) * Transform int features to str * Fix pre-commit * Update version to 2.0.0 * Update version to 2.0.0 * Update CHANGELOG.md * Add information on deepcave to the readme * Correct doc version --------- Co-authored-by: dengdifan <33290713+dengdifan@users.noreply.github.com> Co-authored-by: dengdifan Co-authored-by: Matthias Feurer Co-authored-by: Carolin Benjamins Co-authored-by: eddiebergman Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Carolin Benjamins Co-authored-by: C. Benjamins <75323339+benjamc@users.noreply.github.com> Co-authored-by: Katharina Eggensperger Co-authored-by: Eric Kalosa-Kenyon Co-authored-by: Pascal <66594767+PascalIversen@users.noreply.github.com> Co-authored-by: helegraf Co-authored-by: timruhkopf Co-authored-by: Alexander Tornede Co-authored-by: Sarah Krebs Co-authored-by: Alexander Tornede Co-authored-by: Alexander Tornede <44395781+alexandertornede@users.noreply.github.com> Co-authored-by: Sarah Krebs <30651149+sarah-krebs@users.noreply.github.com> Co-authored-by: helegraf Co-authored-by: Jeroen Rook Co-authored-by: Tim Ruhkopf * feat: Option preventing SMAC setting up logging * fix: Run isort * Bump peter-evans/create-pull-request from 3 to 5 (#958) Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 3 to 5. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](https://github.com/peter-evans/create-pull-request/compare/v3...v5) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump actions/setup-python from 2 to 4 (#945) Bumps [actions/setup-python](https://github.com/actions/setup-python) from 2 to 4. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](https://github.com/actions/setup-python/compare/v2...v4) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: Option preventing SMAC setting up logging * fix: Run isort * fix: Add to changelog * feat: Option preventing SMAC setting up logging * fix: Run isort --------- Signed-off-by: dependabot[bot] Co-authored-by: René Sass Co-authored-by: dengdifan <33290713+dengdifan@users.noreply.github.com> Co-authored-by: dengdifan Co-authored-by: Matthias Feurer Co-authored-by: Carolin Benjamins Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Carolin Benjamins Co-authored-by: C. Benjamins <75323339+benjamc@users.noreply.github.com> Co-authored-by: Katharina Eggensperger Co-authored-by: Eric Kalosa-Kenyon Co-authored-by: Pascal <66594767+PascalIversen@users.noreply.github.com> Co-authored-by: helegraf Co-authored-by: timruhkopf Co-authored-by: Alexander Tornede Co-authored-by: Sarah Krebs Co-authored-by: Alexander Tornede Co-authored-by: Alexander Tornede <44395781+alexandertornede@users.noreply.github.com> Co-authored-by: Sarah Krebs <30651149+sarah-krebs@users.noreply.github.com> Co-authored-by: helegraf Co-authored-by: Jeroen Rook Co-authored-by: Tim Ruhkopf * Citation update (#961) * update CITATION.cff * update order in CITATION.cff * fix citation date * fix citation * maint * Adjust hyperband configuration distribution across brackets * Compute the actual differences between the isb keys. (#957) * Compute the actual differences between the isb keys. * Added change to CHANGELOG.md * Adjust schedule for stale reminder * fix(logging): Prevent automatic logging setup at init (#970) * fix(logging): Prevent automatic logging setup at init * fix(logging): Rely on `setup_logging` to handle arg * Fix validate method of smbo, update docstring. * Allow callbacks to be added to a specific index, make callback registering public * Update CHANGELOG.md * Fix broken references (#988) * Adapt developer install instructions to include pre-commit installation (#994) * [Feature] Initial Design now supports executing the default config provided by the user by means of the configspace he generated. (and is no longer ignored) * Pipe dask_client through facade for exec on hpc (#983) * Pipe dask_client through facade for exec on hpc * Handle pynisher's memory arg Before: Could only pass int via SMAC with the default unit B, but now tuple/list is possible * Update CHANGELOG.md * Make format * No printi --------- Co-authored-by: Carolin Benjamins * [Bug-fix] Pr was failing due to mutable additional configs default * [doc-fix] the count of initial design configs is computed slightly differently * Feature/metadata callback (#999) * Add example for using a callback to log run metadata to a file * Add issue number in changelog * Changelog formatting * Simplify metadata example * Simplify metadata example * Put callback in smac directory to allow importing it in custom repos * Move callback files to callback directory * Edit changelog * Correct metadata callback argument types * Additional metadata arguments in example * Set metadata default values in callback * Edit comment to fix PR * Documentation/dask client example (#1001) * Update requirements (dask_jobqueue) * Fix broken link * Ignore parallel example * Add parallelization example * Remove imports, add comment abt dask client * Add E's information * Update docs * Add comment about n_workers --------- Co-authored-by: Carolin Benjamins * Documentation/dask client example (#1001) * Update requirements (dask_jobqueue) * Fix broken link * Ignore parallel example * Add parallelization example * Remove imports, add comment abt dask client * Add E's information * Update docs * Add comment about n_workers --------- Co-authored-by: Carolin Benjamins * Feature/dask client (#1002) * Add info message that scenario.n_workers is ignored When passing a dask client, directly set n_workers in client. * Change info to warning --------- Co-authored-by: Carolin Benjamins * Update smac version (#1003) * Update smac version * Add hint on github release in make publish * Update copyright year * check if config in rh when storing state (#997) * check if config in rh when storing state * maint * remove unnecessary changes * Adapt developer install instructions to include pre-commit installation (#994) * Pipe dask_client through facade for exec on hpc (#983) * Pipe dask_client through facade for exec on hpc * Handle pynisher's memory arg Before: Could only pass int via SMAC with the default unit B, but now tuple/list is possible * Update CHANGELOG.md * Make format * No printi --------- Co-authored-by: Carolin Benjamins * update changelog * Increase tolerance for runhistory restoration test --------- Co-authored-by: Sarah Segel <30651149+sarah-segel@users.noreply.github.com> Co-authored-by: C. Benjamins <75323339+benjamc@users.noreply.github.com> Co-authored-by: Carolin Benjamins Co-authored-by: Helena Graf * Add a workaround to be able to pass a dataset via dask.scatter (#993) * [feat] Support dask.scatter * [fix] Add shared_data arg in abstract_facade * Address the comments by Eddie * [fix] Fix errors raised by Eddie * [format] Apply black locally * [fix] Get retries back to 16 as it was a mistake * [doc] Update CHANGELOG --------- Co-authored-by: helegraf Co-authored-by: C. Benjamins <75323339+benjamc@users.noreply.github.com> * Adding Binder links for example (#976) * binder * maint * fix dependencies * maint docs * Revert "maint docs" This reverts commit 4fa1a0fae8dfa0b646e43273d50f72ddd76bf191. * add requirement --------- Co-authored-by: Difan Deng * Remove commit message from dependabot PRs (#1009) * Bugfix/fix broken link (#1012) * Remove commit message from dependabot PRs (#1010) * rename check_point_in_ss to check_subspace_point * Make format * Uncomment test procedure * docstrings for boing facade * avoid typing namespace everywhere in epm_configuration_chooser * fix docstring in rh2epm_boing * Limit on dask * Trying to fix tests * Replaced typing in all files * docstring for boing subspace * Changed feature requests to feature * allow acq_optimizer_local_kwargs to be passed to boing subspace without acq_optimizer_local * fix pre-commit * Added any * Update facade * Fix typing * Match arguments from pSMAC with SMAC4AC signature * Add description of pSMAC to docstring * Add option to pass facade class to pSMAC * Add example for pSMAC * Removed docstring in readme's example * Corrected text * move construct_gp_kernels to smac.epm_gpytorch.boing_kernels * maint * fix wrongly imported func name in tests * Add reference for pSMAC #609 * Rename example file * Added roadmap to readme * Prettification * Add result merger * Reformat with black * Delete unnecessary import * Fix typing * Inject first trajectory entry from file from first rundir * Add test for ResultMerger * Fix docstring * Get runhistory and trajectory from pSMAC run Via the result merger. Searchers rundirs and merges found runhistories and calculates a new trajectory. * Get trajectory * Add regex to requirements * Fixed MO cache problem * Fix MO tests * Fixing MO in AbstractRacer * Accepting ints * Hacking in the sum cost for MO * Fix mypi * Bugfix * Make mypi happy * Fix tests * Make format * Remove num_obj in MO optimizer * A lot of restructuring * Solve importings * Solve importings * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Updated changelog * Still fixing :) * fix docstrings * Change directory of tests * Fix ambiguous variable * Del newlines * fix mypy * mypy and doc fix * Removed int type checking * fix tests * Always return List of incs * fix path issue in example * maint boing facade * Better docstring * Return values if no bounds passed * Removed import * Return only one incumbent * Rename n_optimizers -> n_workers * Update example * Update docstring * Remove legacy function * Update to new signature * Update changelog.md * Format black * Update test * Please pre-commit * Merge development * Fix tests * Fix typos * Format black * Updated text * Increased version * Improved readme * SMAC logo 50% * SMAC logo 50% * Removed dev-2.0 * Added doc workflow for latest tag * Fixing docs latest tag * Fixing docs latest tag * First steps new intensifier * Make it work again * MOVE random weight initialization to smbo ask method * ADD enable parego in unittests * Test issues * FIX formatting * Link to old documentation * MOVE random weight initialization to smbo ask method * ADD enable parego in unittests * FIX formatting * Fix unit tests * Add annotation * corrected log prob for tophat prior * Removed configurations from callback * Fixed tests * Fix mypy * Save docs as latest version too * Enabled all tests again * Make version an env * Added last version to docs * Global variable fix * Added v before version number * New Intensifier Implementation (#888) - Completely reimplemented the intensifiers (including Successive Halving and Hyperband): All intensifiers support multi-fidelity, multi-objective and multi-threading by nature now. - Expected behaviour for ask-and-tell interface ensured (also for Successive Halving). - Continuing a run is now fully supported. - Added more examples. - Updated documentation based on new implementation. * First benchmark steps * Added a warning * Set xi to 0.01 * Incorporated walltime in trajectory * Working version of benchmark * Changed requirements * Changed xi back to 0.0 again * No duplicate budgets * Fixed bug which updated tracker wrongly * Bugfix: Delete correct bracket+stage * Fixed tests * Added costs to the trajectory * Changed initial design max ratio to 0.25 * Make format * Added more details to intensifier * Finished benchmark * Added benchmark source files * Fixed tests because of initial design changes * Improved runhistory API * Fixed typo (#894) * Merge doc fixes * Doc fixes (#893) Co-authored-by: timruhkopf Co-authored-by: Carolin Benjamins Co-authored-by: dengdifan Co-authored-by: Alexander Tornede Co-authored-by: Sarah Krebs * Make format * Satisfied mypy * Enabled more tests * Increased version * Mypy * Mypy * Mypy * Fixed critical bug when using highest budget for inc selection * Added SH+HB comparison * Stuff * Updated report * Fixed docstring * Fixed image path * Removed vscode folder * Bugfixes * Doc fixes * Updated changelog * Make isort * Fix typos * Removed new line * Added alpha2 to docs selection * update description * Fix 531 correctly create integers in initial design This PR fixes a bug in which integer hyperparameters would not be created correctly in the initial design. This went unnoticed, except in the case where it was part of a condition and the function deactivate_inactive_hyperparameters was called. This PR fixes this bug by mapping the random float that was sampled by, for example and LHD or Sobol sequence, to the float representation of an integer by calling _transform and _inverse_transform of the integer hyperparameter. * Add contact paragraph to README. * Increase days until stale issues are closed by stalebot (#904) * Edit changelog * Add issue number * Closes #907 * Clarify origin of configurations (#908) * Random forest speed up (#903) Co-authored-by: René Sass * Add: workflow to update pre-commit versions (#874) Co-authored-by: René Sass * Add Git-Flow Workflow Specification * [DOC] acquisition function maximizers * Add stale reminder email action * Increase version number of ConfigSpace * Add guideline for updating copyright notice years to CONTRIBUTING.md * Update requirement for pyrfr (#920) * Update pre-commit: isort,flake8 version numbers * Format black * Fix mypy * Ac bench (#929) * Add Branin AC Task * Add ac optimization type * Nicer plotting * Add ac mode * Update ac task * Make new smac env name adjustable * Increase python version for smac 1.4 * Update benchmark * Update plotting * Goodby comments * Fix path * Fix 1.4 wrapper * Update benchmark results --------- Co-authored-by: Carolin Benjamins * [DOC] added documentation on the incumbent update * Feature/add version to make publish (#931) * Add version, correct package name and use bash in makefile * Increase version * Increase version in makefile, add places with versions to make publish * Fix typo * Transform integer feature names to strings (#930) * Transform int features to str * Fix pre-commit * Update version to 2.0.0 * Update version to 2.0.0 * Update CHANGELOG.md * Add information on deepcave to the readme * Correct doc version * fix(runhistory): set id of the sampled config (#951) * fix: Set config_id of config in runhistory * ci: Set dependabot to target development branch (#963) * feat: Option preventing SMAC setting up logging (#947) * Version 2.0.0 (#898) * Update runhistory2epm.py * pre_commit * Updated changelog * Fixed comment * Updated changelog * Some small improvements * Removed trailing space * Updated changelog * Restricted pynisher * Updated changelog * Link JMLR paper * Update bibtex entry Before: arxiv paper, now journal article. * Fixed citation in docs * Fixed callback typos * BOinG for mixed hyperparameter search space * fix format * maint * maint documentation * maint pre-commit * fix workflow * maint * Add dependabot for workflow files * Bump actions/setup-python from 2 to 4 (#849) Bumps [actions/setup-python](https://github.com/actions/setup-python) from 2 to 4. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](https://github.com/actions/setup-python/compare/v2...v4) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump actions/checkout from 2 to 3 (#850) Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v2...v3) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * maint * Updated stale bot * Updated stale options again * Fixed typing * Increased version * rename check_point_in_ss to check_subspace_point * Make format * Uncomment test procedure * docstrings for boing facade * avoid typing namespace everywhere in epm_configuration_chooser * fix docstring in rh2epm_boing * Limit on dask * Trying to fix tests * Replaced typing in all files * docstring for boing subspace * Changed feature requests to feature * allow acq_optimizer_local_kwargs to be passed to boing subspace without acq_optimizer_local * fix pre-commit * Added any * Update facade * Fix typing * Match arguments from pSMAC with SMAC4AC signature * Add description of pSMAC to docstring * Add option to pass facade class to pSMAC * Add example for pSMAC * Removed docstring in readme's example * Corrected text * move construct_gp_kernels to smac.epm_gpytorch.boing_kernels * maint * fix wrongly imported func name in tests * Add reference for pSMAC #609 * Rename example file * Added roadmap to readme * Prettification * Add result merger * Reformat with black * Delete unnecessary import * Fix typing * Inject first trajectory entry from file from first rundir * Add test for ResultMerger * Fix docstring * Get runhistory and trajectory from pSMAC run Via the result merger. Searchers rundirs and merges found runhistories and calculates a new trajectory. * Get trajectory * Add regex to requirements * Fixed MO cache problem * Fix MO tests * Fixing MO in AbstractRacer * Accepting ints * Hacking in the sum cost for MO * Fix mypi * Bugfix * Make mypi happy * Fix tests * Make format * Remove num_obj in MO optimizer * A lot of restructuring * Solve importings * Solve importings * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Fixing, fixing, fixing, ... * Updated changelog * Still fixing :) * fix docstrings * Change directory of tests * Fix ambiguous variable * Del newlines * fix mypy * mypy and doc fix * Removed int type checking * fix tests * Always return List of incs * fix path issue in example * maint boing facade * Better docstring * Return values if no bounds passed * Removed import * Return only one incumbent * Rename n_optimizers -> n_workers * Update example * Update docstring * Remove legacy function * Update to new signature * Update changelog.md * Format black * Update test * Please pre-commit * Merge development * Fix tests * Fix typos * Format black * Updated text * Increased version * Improved readme * SMAC logo 50% * SMAC logo 50% * Removed dev-2.0 * Added doc workflow for latest tag * Fixing docs latest tag * Fixing docs latest tag * First steps new intensifier * Make it work again * MOVE random weight initialization to smbo ask method * ADD enable parego in unittests * Test issues * FIX formatting * Link to old documentation * MOVE random weight initialization to smbo ask method * ADD enable parego in unittests * FIX formatting * Fix unit tests * Add annotation * corrected log prob for tophat prior * Removed configurations from callback * Fixed tests * Fix mypy * Save docs as latest version too * Enabled all tests again * Make version an env * Added last version to docs * Global variable fix * Added v before version number * New Intensifier Implementation (#888) - Completely reimplemented the intensifiers (including Successive Halving and Hyperband): All intensifiers support multi-fidelity, multi-objective and multi-threading by nature now. - Expected behaviour for ask-and-tell interface ensured (also for Successive Halving). - Continuing a run is now fully supported. - Added more examples. - Updated documentation based on new implementation. * First benchmark steps * Added a warning * Set xi to 0.01 * Incorporated walltime in trajectory * Working version of benchmark * Changed requirements * Changed xi back to 0.0 again * No duplicate budgets * Fixed bug which updated tracker wrongly * Bugfix: Delete correct bracket+stage * Fixed tests * Added costs to the trajectory * Changed initial design max ratio to 0.25 * Make format * Added more details to intensifier * Finished benchmark * Added benchmark source files * Fixed tests because of initial design changes * Improved runhistory API * Fixed typo (#894) * Merge doc fixes * Doc fixes (#893) Co-authored-by: timruhkopf Co-authored-by: Carolin Benjamins Co-authored-by: dengdifan Co-authored-by: Alexander Tornede Co-authored-by: Sarah Krebs * Make format * Satisfied mypy * Enabled more tests * Increased version * Mypy * Mypy * Mypy * Fixed critical bug when using highest budget for inc selection * Added SH+HB comparison * Stuff * Updated report * Fixed docstring * Fixed image path * Removed vscode folder * Bugfixes * Doc fixes * Updated changelog * Make isort * Fix typos * Removed new line * Added alpha2 to docs selection * update description * Fix 531 correctly create integers in initial design This PR fixes a bug in which integer hyperparameters would not be created correctly in the initial design. This went unnoticed, except in the case where it was part of a condition and the function deactivate_inactive_hyperparameters was called. This PR fixes this bug by mapping the random float that was sampled by, for example and LHD or Sobol sequence, to the float representation of an integer by calling _transform and _inverse_transform of the integer hyperparameter. * Add contact paragraph to README. * Increase days until stale issues are closed by stalebot (#904) * Edit changelog * Add issue number * Closes #907 * Clarify origin of configurations (#908) * Random forest speed up (#903) Co-authored-by: René Sass * Add: workflow to update pre-commit versions (#874) Co-authored-by: René Sass * Add Git-Flow Workflow Specification * [DOC] acquisition function maximizers * Add stale reminder email action * Increase version number of ConfigSpace * Add guideline for updating copyright notice years to CONTRIBUTING.md * Update requirement for pyrfr (#920) * Update pre-commit: isort,flake8 version numbers * Format black * Fix mypy * Ac bench (#929) * Add Branin AC Task * Add ac optimization type * Nicer plotting * Add ac mode * Update ac task * Make new smac env name adjustable * Increase python version for smac 1.4 * Update benchmark * Update plotting * Goodby comments * Fix path * Fix 1.4 wrapper * Update benchmark results --------- Co-authored-by: Carolin Benjamins * [DOC] added documentation on the incumbent update * Feature/add version to make publish (#931) * Add version, correct package name and use bash in makefile * Increase version * Increase version in makefile, add places with versions to make publish * Fix typo * Transform integer feature names to strings (#930) * Transform int features to str * Fix pre-commit * Update version to 2.0.0 * Update version to 2.0.0 * Update CHANGELOG.md * Add information on deepcave to the readme * Correct doc version --------- Co-authored-by: dengdifan <33290713+dengdifan@users.noreply.github.com> Co-authored-by: dengdifan Co-authored-by: Matthias Feurer Co-authored-by: Carolin Benjamins Co-authored-by: eddiebergman Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Carolin Benjamins Co-authored-by: C. Benjamins <75323339+benjamc@users.noreply.github.com> Co-authored-by: Katharina Eggensperger Co-authored-by: Eric Kalosa-Kenyon Co-authored-by: Pascal <66594767+PascalIversen@users.noreply.github.com> Co-authored-by: helegraf Co-authored-by: timruhkopf Co-authored-by: Alexander Tornede Co-authored-by: Sarah Krebs Co-authored-by: Alexander Tornede Co-authored-by: Alexander Tornede <44395781+alexandertornede@users.noreply.github.com> Co-authored-by: Sarah Krebs <30651149+sarah-krebs@users.noreply.github.com> Co-authored-by: helegraf Co-authored-by: Jeroen Rook Co-authored-by: Tim Ruhkopf * feat: Option preventing SMAC setting up logging * fix: Run isort * Bump peter-evans/create-pull-request from 3 to 5 (#958) Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 3 to 5. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](https://github.com/peter-evans/create-pull-request/compare/v3...v5) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump actions/setup-python from 2 to 4 (#945) Bumps [actions/setup-python](https://github.com/actions/setup-python) from 2 to 4. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](https://github.com/actions/setup-python/compare/v2...v4) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: Option preventing SMAC setting up logging * fix: Run isort * fix: Add to changelog * feat: Option preventing SMAC setting up logging * fix: Run isort --------- Signed-off-by: dependabot[bot] Co-authored-by: René Sass Co-authored-by: dengdifan <33290713+dengdifan@users.noreply.github.com> Co-authored-by: dengdifan Co-authored-by: Matthias Feurer Co-authored-by: Carolin Benjamins Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Carolin Benjamins Co-authored-by: C. Benjamins <75323339+benjamc@users.noreply.github.com> Co-authored-by: Katharina Eggensperger Co-authored-by: Eric Kalosa-Kenyon Co-authored-by: Pascal <66594767+PascalIversen@users.noreply.github.com> Co-authored-by: helegraf Co-authored-by: timruhkopf Co-authored-by: Alexander Tornede Co-authored-by: Sarah Krebs Co-authored-by: Alexander Tornede Co-authored-by: Alexander Tornede <44395781+alexandertornede@users.noreply.github.com> Co-authored-by: Sarah Krebs <30651149+sarah-krebs@users.noreply.github.com> Co-authored-by: helegraf Co-authored-by: Jeroen Rook Co-authored-by: Tim Ruhkopf * Citation update (#961) * update CITATION.cff * update order in CITATION.cff * fix citation date * fix citation * maint * Adjust hyperband configuration distribution across brackets * Compute the actual differences between the isb keys. (#957) * Compute the actual differences between the isb keys. * Added change to CHANGELOG.md * Adjust schedule for stale reminder * fix(logging): Prevent automatic logging setup at init (#970) * fix(logging): Prevent automatic logging setup at init * fix(logging): Rely on `setup_logging` to handle arg * Fix validate method of smbo, update docstring. * Allow callbacks to be added to a specific index, make callback registering public * Update CHANGELOG.md * Fix broken references (#988) * Adapt developer install instructions to include pre-commit installation (#994) * [Feature] Initial Design now supports executing the default config provided by the user by means of the configspace he generated. (and is no longer ignored) * Pipe dask_client through facade for exec on hpc (#983) * Pipe dask_client through facade for exec on hpc * Handle pynisher's memory arg Before: Could only pass int via SMAC with the default unit B, but now tuple/list is possible * Update CHANGELOG.md * Make format * No printi --------- Co-authored-by: Carolin Benjamins * [Bug-fix] Pr was failing due to mutable additional configs default * [doc-fix] the count of initial design configs is computed slightly differently * Feature/metadata callback (#999) * Add example for using a callback to log run metadata to a file * Add issue number in changelog * Changelog formatting * Simplify metadata example * Simplify metadata example * Put callback in smac directory to allow importing it in custom repos * Move callback files to callback directory * Edit changelog * Correct metadata callback argument types * Additional metadata arguments in example * Set metadata default values in callback * Edit comment to fix PR * Documentation/dask client example (#1001) * Update requirements (dask_jobqueue) * Fix broken link * Ignore parallel example * Add parallelization example * Remove imports, add comment abt dask client * Add E's information * Update docs * Add comment about n_workers --------- Co-authored-by: Carolin Benjamins * Documentation/dask client example (#1001) * Update requirements (dask_jobqueue) * Fix broken link * Ignore parallel example * Add parallelization example * Remove imports, add comment abt dask client * Add E's information * Update docs * Add comment about n_workers --------- Co-authored-by: Carolin Benjamins * Feature/dask client (#1002) * Add info message that scenario.n_workers is ignored When passing a dask client, directly set n_workers in client. * Change info to warning --------- Co-authored-by: Carolin Benjamins * Update smac version (#1003) * Update smac version * Add hint on github release in make publish * Update copyright year * check if config in rh when storing state (#997) * check if config in rh when storing state * maint * remove unnecessary changes * Adapt developer install instructions to include pre-commit installation (#994) * Pipe dask_client through facade for exec on hpc (#983) * Pipe dask_client through facade for exec on hpc * Handle pynisher's memory arg Before: Could only pass int via SMAC with the default unit B, but now tuple/list is possible * Update CHANGELOG.md * Make format * No printi --------- Co-authored-by: Carolin Benjamins * update changelog * Increase tolerance for runhistory restoration test --------- Co-authored-by: Sarah Segel <30651149+sarah-segel@users.noreply.github.com> Co-authored-by: C. Benjamins <75323339+benjamc@users.noreply.github.com> Co-authored-by: Carolin Benjamins Co-authored-by: Helena Graf * Add a workaround to be able to pass a dataset via dask.scatter (#993) * [feat] Support dask.scatter * [fix] Add shared_data arg in abstract_facade * Address the comments by Eddie * [fix] Fix errors raised by Eddie * [format] Apply black locally * [fix] Get retries back to 16 as it was a mistake * [doc] Update CHANGELOG --------- Co-authored-by: helegraf Co-authored-by: C. Benjamins <75323339+benjamc@users.noreply.github.com> * Adding Binder links for example (#976) * binder * maint * fix dependencies * maint docs * Revert "maint docs" This reverts commit 4fa1a0fae8dfa0b646e43273d50f72ddd76bf191. * add requirement --------- Co-authored-by: Difan Deng * Remove commit message from dependabot PRs (#1009) --------- Signed-off-by: dependabot[bot] Co-authored-by: dengdifan Co-authored-by: René Sass Co-authored-by: Carolin Benjamins Co-authored-by: Carolin Benjamins Co-authored-by: C. Benjamins <75323339+benjamc@users.noreply.github.com> Co-authored-by: Katharina Eggensperger Co-authored-by: Matthias Feurer Co-authored-by: Eric Kalosa-Kenyon Co-authored-by: Pascal <66594767+PascalIversen@users.noreply.github.com> Co-authored-by: helegraf Co-authored-by: timruhkopf Co-authored-by: Alexander Tornede Co-authored-by: Alexander Tornede Co-authored-by: Alexander Tornede <44395781+alexandertornede@users.noreply.github.com> Co-authored-by: Sarah Krebs <30651149+sarah-krebs@users.noreply.github.com> Co-authored-by: helegraf Co-authored-by: Jeroen Rook Co-authored-by: Eddie Bergman Co-authored-by: Tim Ruhkopf Co-authored-by: Difan Deng <33290713+dengdifan@users.noreply.github.com> Co-authored-by: Aron Bahram Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Shuhei Watanabe <47781922+nabenabe0928@users.noreply.github.com> Co-authored-by: Difan Deng * Fix broken link --------- Signed-off-by: dependabot[bot] Co-authored-by: dengdifan Co-authored-by: René Sass Co-authored-by: Carolin Benjamins Co-authored-by: Carolin Benjamins Co-authored-by: C. Benjamins <75323339+benjamc@users.noreply.github.com> Co-authored-by: Katharina Eggensperger Co-authored-by: Matthias Feurer Co-authored-by: Eric Kalosa-Kenyon Co-authored-by: Pascal <66594767+PascalIversen@users.noreply.github.com> Co-authored-by: helegraf Co-authored-by: timruhkopf Co-authored-by: Alexander Tornede Co-authored-by: Alexander Tornede Co-authored-by: Alexander Tornede <44395781+alexandertornede@users.noreply.github.com> Co-authored-by: Sarah Krebs <30651149+sarah-krebs@users.noreply.github.com> Co-authored-by: helegraf Co-authored-by: Jeroen Rook Co-authored-by: Eddie Bergman Co-authored-by: Tim Ruhkopf Co-authored-by: Difan Deng <33290713+dengdifan@users.noreply.github.com> Co-authored-by: Aron Bahram Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Shuhei Watanabe <47781922+nabenabe0928@users.noreply.github.com> Co-authored-by: Difan Deng * Drop torch requirements (#1015) * Remove torch requirements * Fix format --------- Co-authored-by: Carolin Benjamins * Remove leftover CLI entry points (#1014) * Fix incumbent selection in case of SH+MO * Update CHANGELOG.md * Feature/GitHub actions pr draft (#1008) Update the workflows only to trigger extensive tests for PRs that are not in the draft stage --------- Co-authored-by: timruhkopf Co-authored-by: Helena Graf * Update stale report to include issues with recent interactions (#1025) * Add recently updated issues to list of reminders. * Add recently updated issues to list of reminders. * Add recently updated issues to list of reminders. * Raise error if resource limitation and parallelization is requested (#1023) * Add warnings for parallelism doc * Check for parallelization + pynisher Does not work together, throw error. Does not work bc Dask pickles jobs and pynisher runs the target function in a process. * Fix whitespaces --------- Co-authored-by: Carolin Benjamins * Raise an error for invalid scatter data (#1018) * [enhance] Add an error for scatter data * Update ChangeLog --------- Co-authored-by: C. Benjamins <75323339+benjamc@users.noreply.github.com> * fix(configspace): shows all keys in print (#1043) * Fix callback order (#1040) Co-authored-by: Carolin Benjamins * fix: dtypes (#1044) * Fix(config_selector): Budget checking (#1039) * Fix: target runner with partial func (#1045) * Feature/improve documentation (#1028) * Experimental instruction for installing SMAC in windows via WSL * More detailed documentation regarding continuing runs * Fixes * Fixes * Fixes * Update docs/10_experimental.rst Co-authored-by: Difan Deng <33290713+dengdifan@users.noreply.github.com> * Fixes from PR * Fixes from PR * Fixes from PR * Fixes from PR * Fixes from PR --------- Co-authored-by: Difan Deng <33290713+dengdifan@users.noreply.github.com> * Handle configspace as dictionary in mlp example (#1057) * Handle configspace as dictionary in mlp example * Adapt sgd loss to newest scikit-learn version * Add change to changelog * Handle configspace as dictionary in parego example * Correct get usage * Raise version number * Add missing period in docs. * allow rf to impute OrdinalHyperparameter (#1065) * allow rf to impute OrdinalHyperparameter * Add missing period in docs. * Add issue number in changelog. --------- Co-authored-by: Sarah Krebs * fix dask_scheduler_file path (#1055) * fix dask_scheduler_file path * update change log --------- Co-authored-by: dengdifan * Fix config rejection for #1068 (#1069) * racing: make sure config only deleted from rejected configs if among incumbents upon update * test case for incumbent rejection * update code comments * added missing newline ahead of new test case * update change log --------- Co-authored-by: dengdifan Co-authored-by: Difan Deng <33290713+dengdifan@users.noreply.github.com> * fix(RandomForest): Always cast `seed` to `int` (#1084) * fix(RandomForest): Always cast `seed` to `int` In cases where seeds were generated from some numpy objects, sometimes you'd get back an `np.integer`, which causes the program to crash when communicating with `pyrfr` through `swig`. It seems that swig doesn't know that it's an _int-like_ and so we explicitly cast it to `int`. * Update CHANGELOG.md --------- Co-authored-by: C. Benjamins <75323339+benjamc@users.noreply.github.com> * Bump actions/checkout from 3 to 4 (#1072) * Bump actions/checkout from 3 to 4 Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v3...v4) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] * Update CHANGELOG.md --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: benjamc * chore: log warn on condition when custom dask client is provided (#1071) * Bump conda-incubator/setup-miniconda from 2 to 3 (#1087) Bumps [conda-incubator/setup-miniconda](https://github.com/conda-incubator/setup-miniconda) from 2 to 3. - [Release notes](https://github.com/conda-incubator/setup-miniconda/releases) - [Changelog](https://github.com/conda-incubator/setup-miniconda/blob/main/CHANGELOG.md) - [Commits](https://github.com/conda-incubator/setup-miniconda/compare/v2...v3) --- updated-dependencies: - dependency-name: conda-incubator/setup-miniconda dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Propagate the Scenario random seed to get_random_design (#1066) * Propagate the Scenario's random seed to get_random_design * add changelog line --------- Co-authored-by: Sarah Segel <30651149+sarah-segel@users.noreply.github.com> * [#1056] Add example on intensify for cross-validation. (#1061) * Add example on intensify for cross-validation. * Improve example * Address Difan's comment * Update CHANGELOG.md * Change stale-reminder to recent-reminder (#1096) * Change stale-reminder to recent-reminder (#1095) (#1099) * Dont use mutable lists as default args (#1097) * Don't use mutable default argument. If this isn't respected, then additional_configs can be built up overtime when multiple optimizers are created in a single process. This can occur when the use_default_config option is set True and/or when additional_configs are provided. When different ConfigSpaces are provided this can result in an error due to mismatched defaults, for instance. In other cases it can result in non-deterministic behavior due to the random ordering of the configs that get added (e.g., with different parallel pytest runs). - [ ] Cleanup other sources of this error as well elsewhere in the code. - [ ] Add pylint checks for this issue. * fixup some others mutable defaults * add pylint checks * fix more related pylint issues * add changelog line * Update pre-commit config * Update pre-commit config * Remove TODO formulation as an issue already exists for this --------- Co-authored-by: Brian Kroth * Bump actions/setup-python from 4 to 5 (#1089) Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4 to 5. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](https://github.com/actions/setup-python/compare/v4...v5) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump peter-evans/create-pull-request from 5 to 6 (#1093) Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 5 to 6. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](https://github.com/peter-evans/create-pull-request/compare/v5...v6) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * test: Fix upper bounds for `test_transformer()` and no random in `test_transformer_conditional()` (#1102) * test: Don't rely on randomization for the test * test: Include import * test: Fix boundaries for catgoricals in test * test: Missing import * doc: Make todo not more clear about when to upgrade * Change stale-reminder to recent-reminder (#1095) (#1107) * Fix typo in docstring for `MultiFideltyFacade.get_initial_design` (#1104) * Fix typo in docstring for `MultiFideltyFacade.get_initial_design` The docstring specifies 0.1 is used as a default value for max_ratio, however in code 0.25 is used. * Bug/acquisition maximizer sampling (#1106) * Adapt how n_points works in the config_selector.py Also remove LocalAndSortedPriorRandomSearch by integrating it into LocalAndSortedRandomSearch * Change retrain_after to 1 for BlackBOx Facade * Fix bugs in local_and_random_search.py * Edit CHANGELOG.md * fix acq maximizer docstring * call get config selector from super class * Update random and local search to new functionality * Update CHANGELOG.md * Fix sampling of initial points for local search when less previous configs than n_points * Re-format blackbox_facade.py --------- Co-authored-by: Helena Graf Co-authored-by: dengdifan * Update version number to 2.1.0 --------- Signed-off-by: dependabot[bot] Co-authored-by: René Sass Co-authored-by: dengdifan Co-authored-by: Carolin Benjamins Co-authored-by: Katharina Eggensperger Co-authored-by: Matthias Feurer Co-authored-by: Eric Kalosa-Kenyon Co-authored-by: Pascal <66594767+PascalIversen@users.noreply.github.com> Co-authored-by: helegraf Co-authored-by: timruhkopf Co-authored-by: Alexander Tornede Co-authored-by: Sarah Krebs Co-authored-by: Alexander Tornede Co-authored-by: Alexander Tornede <44395781+alexandertornede@users.noreply.github.com> Co-authored-by: Sarah Krebs <30651149+sarah-krebs@users.noreply.github.com> Co-authored-by: helegraf Co-authored-by: Jeroen Rook Co-authored-by: Eddie Bergman Co-authored-by: Tim Ruhkopf Co-authored-by: Difan Deng <33290713+dengdifan@users.noreply.github.com> Co-authored-by: Aron Bahram Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sarah Segel <30651149+sarah-segel@users.noreply.github.com> Co-authored-by: Shuhei Watanabe <47781922+nabenabe0928@users.noreply.github.com> Co-authored-by: Difan Deng Co-authored-by: Bastian Zimmermann <10774221+BastianZim@users.noreply.github.com> Co-authored-by: Weihuang Wen Co-authored-by: PhilippBordne <71140732+PhilippBordne@users.noreply.github.com> Co-authored-by: SimonThormeyer <49559340+SimonThormeyer@users.noreply.github.com> Co-authored-by: Brian Kroth Co-authored-by: Brian Kroth Co-authored-by: fleance --- .github/workflows/citation.yml | 2 +- .github/workflows/dist.yml | 4 +- .github/workflows/docs.yml | 4 +- .github/workflows/pre-commit-update.yml | 6 +- .github/workflows/pre-commit.yml | 5 +- .github/workflows/pytest.yml | 12 +- .pre-commit-config.yaml | 13 + CHANGELOG.md | 30 ++- CITATION.cff | 2 +- Makefile | 2 +- benchmark/src/wrappers/v20.py | 2 +- .../4_intensify_crossvalidation.py | 127 ++++++++++ pyproject.toml | 5 + setup.py | 1 + smac/__init__.py | 2 +- smac/acquisition/maximizer/__init__.py | 2 - .../abstract_acqusition_maximizer.py | 13 +- .../maximizer/differential_evolution.py | 1 + smac/acquisition/maximizer/helpers.py | 47 +--- .../maximizer/local_and_random_search.py | 231 +++++++----------- smac/acquisition/maximizer/local_search.py | 29 +-- smac/facade/abstract_facade.py | 7 +- smac/facade/algorithm_configuration_facade.py | 4 +- smac/facade/blackbox_facade.py | 17 +- .../hyperparameter_optimization_facade.py | 4 +- smac/facade/multi_fidelity_facade.py | 6 +- smac/facade/random_facade.py | 4 +- smac/intensifier/abstract_intensifier.py | 8 +- smac/main/config_selector.py | 7 +- .../random_forest/abstract_random_forest.py | 5 +- smac/model/random_forest/random_forest.py | 4 +- smac/runhistory/encoder/abstract_encoder.py | 16 +- smac/runhistory/runhistory.py | 4 +- smac/runner/abstract_runner.py | 4 +- smac/runner/dask_runner.py | 2 +- smac/runner/target_function_runner.py | 4 +- smac/runner/target_function_script_runner.py | 4 +- tests/test_acquisition/test_maximizers.py | 28 +-- .../test_abstract_intensifier.py | 42 ++++ tests/test_model/test_rf.py | 14 +- .../test_runhistory_encoder.py | 138 ++++++++--- 41 files changed, 552 insertions(+), 310 deletions(-) create mode 100644 examples/4_advanced_optimizer/4_intensify_crossvalidation.py diff --git a/.github/workflows/citation.yml b/.github/workflows/citation.yml index 9ca4ace4b9..91509b9008 100644 --- a/.github/workflows/citation.yml +++ b/.github/workflows/citation.yml @@ -11,7 +11,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Check out a copy of the repository - uses: actions/checkout@v3 + uses: actions/checkout@v4 - name: Check whether the citation metadata from CITATION.cff is valid uses: citation-file-format/cffconvert-github-action@2.0.0 diff --git a/.github/workflows/dist.yml b/.github/workflows/dist.yml index 0965ea2f1c..8e99080608 100644 --- a/.github/workflows/dist.yml +++ b/.github/workflows/dist.yml @@ -28,10 +28,10 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v3 + uses: actions/checkout@v4 - name: Setup Python - uses: actions/setup-python@v4 + uses: actions/setup-python@v5 with: python-version: "3.10" diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index d89791712f..5f03e854f6 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -32,10 +32,10 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout - uses: actions/checkout@v3 + uses: actions/checkout@v4 - name: Setup Python - uses: actions/setup-python@v4 + uses: actions/setup-python@v5 with: python-version: "3.10" diff --git a/.github/workflows/pre-commit-update.yml b/.github/workflows/pre-commit-update.yml index 1688f60d7e..b027befb20 100644 --- a/.github/workflows/pre-commit-update.yml +++ b/.github/workflows/pre-commit-update.yml @@ -11,9 +11,9 @@ jobs: auto-update: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 - - uses: actions/setup-python@v4 + - uses: actions/setup-python@v5 - uses: browniebroke/pre-commit-autoupdate-action@main @@ -21,7 +21,7 @@ jobs: run: | pre-commit run --all-files - - uses: peter-evans/create-pull-request@v5 + - uses: peter-evans/create-pull-request@v6 with: token: ${{ secrets.GITHUB_TOKEN }} branch: update/pre-commit-hooks diff --git a/.github/workflows/pre-commit.yml b/.github/workflows/pre-commit.yml index e9ba1a01bf..31d9577628 100644 --- a/.github/workflows/pre-commit.yml +++ b/.github/workflows/pre-commit.yml @@ -27,12 +27,12 @@ jobs: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: recursive - name: Setup Python 3.10 - uses: actions/setup-python@v4 + uses: actions/setup-python@v5 with: python-version: "3.10" @@ -40,6 +40,7 @@ jobs: run: | pip install pre-commit pre-commit install + pip install pylint - name: Run pre-commit run: | diff --git a/.github/workflows/pytest.yml b/.github/workflows/pytest.yml index 37e15a4729..c94e246bbf 100644 --- a/.github/workflows/pytest.yml +++ b/.github/workflows/pytest.yml @@ -54,10 +54,10 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v3 + uses: actions/checkout@v4 - name: Setup Python - uses: actions/setup-python@v4 + uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} @@ -108,10 +108,10 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v3 + uses: actions/checkout@v4 - name: Conda install - uses: conda-incubator/setup-miniconda@v2 + uses: conda-incubator/setup-miniconda@v3 with: auto-update-conda: true python-version: ${{ matrix.python-version }} @@ -148,10 +148,10 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v3 + uses: actions/checkout@v4 - name: Setup Python - uses: actions/setup-python@v4 + uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index dda7e5cae1..6b2d3934ff 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -43,3 +43,16 @@ repos: name: flake8 smac files: smac exclude: "scripts|tests" + + - repo: local + hooks: + - id: pylint + name: pylint + entry: pylint + language: system + types: [python] + args: + [ + "-rn", # Only display messages + "-sn", # Don't display the score + ] diff --git a/CHANGELOG.md b/CHANGELOG.md index dbcac80e4c..f6768b6898 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,9 +1,37 @@ +# 2.1.0 + +## Improvements +- Change the surrogate model to be retrained after every iteration by default in the case of blackbox optimization + (#1106). +- Integrate `LocalAndSortedPriorRandomSearch` functionality into `LocalAndSortedRandomSearch` (#1106). +- Change the way the `LocalAndSortedRandomSearch` works such that the incumbent always is a starting point and that + random configurations are sampled as the basis of the local search, not in addition (#1106). + +## Bugfixes +- Fix path for dask scheduler file (#1055). +- Add OrdinalHyperparameter for random forest imputer (#1065). +- Don't use mutable default argument (#1067). +- Propagate the Scenario random seed to `get_random_design` (#1066). +- Configurations that fail to become incumbents will be added to the rejected lists (#1069). +- SMAC RandomForest doesn't crash when `np.integer` used, i.e. as generated from a `np.random.RandomState` (#1084). +- Fix the handling of n_points/ challengers in the acquisition maximizers, such that this number now functions as the + number of points that are sampled from the acquisition function to find the next challengers. Now also doesn't + restrict the config selector to n_retrain many points for finding the max, and instead uses the defaults that are + defined via facades/ scenarios (#1106). + +## Misc +- ci: Update action version (#1072). + +## Minor +- When a custom dask client is provided, emit the warning that the `n_workers` parameter is ignored only if it deviates from its default value, `1` ([#1071](https://github.com/automl/SMAC3/pull/1071)). + # 2.0.2 ## Improvements -- Add an error when we get an empty dict data_to_scatter so that we can avoid an internal error caused in Dask precautiously +- Add an error when we get an empty dict data_to_scatter so that we can avoid an internal error caused in Dask precautiously. - Add experimental instruction for installing SMAC in Windows via a WSL. - More detailed documentation regarding continuing runs. +- Add a new example that demonstrates the use of intensification to speed up cross-validation for machine learning. ## Bugfixes - Fix bug in the incumbent selection in the case that multi-fidelity is combined with multi-objective (#1019). diff --git a/CITATION.cff b/CITATION.cff index a984b24c58..ece8325c64 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -9,7 +9,7 @@ date-released: "2016-08-17" url: "https://automl.github.io/SMAC3/master/index.html" repository-code: "https://github.com/automl/SMAC3" -version: "2.0.2" +version: "2.1.0" type: "software" keywords: diff --git a/Makefile b/Makefile index ba7a4268b8..195773082e 100644 --- a/Makefile +++ b/Makefile @@ -5,7 +5,7 @@ SHELL := /bin/bash NAME := SMAC3 PACKAGE_NAME := smac -VERSION := 2.0.2 +VERSION := 2.1.0 DIR := "${CURDIR}" SOURCE_DIR := ${PACKAGE_NAME} diff --git a/benchmark/src/wrappers/v20.py b/benchmark/src/wrappers/v20.py index 9f2be56dcb..3ee376a376 100644 --- a/benchmark/src/wrappers/v20.py +++ b/benchmark/src/wrappers/v20.py @@ -6,7 +6,7 @@ class Version20(Wrapper): - supported_versions: list[str] = ["2.0.2"] + supported_versions: list[str] = ["2.1.0"] def __init__(self, task: Task, seed: int) -> None: super().__init__(task, seed) diff --git a/examples/4_advanced_optimizer/4_intensify_crossvalidation.py b/examples/4_advanced_optimizer/4_intensify_crossvalidation.py new file mode 100644 index 0000000000..d215dd8ec6 --- /dev/null +++ b/examples/4_advanced_optimizer/4_intensify_crossvalidation.py @@ -0,0 +1,127 @@ +""" +Speeding up Cross-Validation with Intensification +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +An example of optimizing a simple support vector machine on the digits dataset. In contrast to the +[simple example](examples/1_basics/2_svm_cv.py), in which all cross-validation folds are executed +at once, we use the intensification mechanism described in the original +[SMAC paper](https://link.springer.com/chapter/10.1007/978-3-642-25566-3_40) as also demonstrated +by [Auto-WEKA](https://dl.acm.org/doi/10.1145/2487575.2487629). This mechanism allows us to +terminate the evaluation of a configuration if after a certain number of folds, the configuration +is found to be worse than the incumbent configuration. This is especially useful if the evaluation +of a configuration is expensive, e.g., if we have to train a neural network or if we have to +evaluate the configuration on a large dataset. +""" +__copyright__ = "Copyright 2023, AutoML.org Freiburg-Hannover" +__license__ = "3-clause BSD" + +N_FOLDS = 10 # Global variable that determines the number of folds + +from ConfigSpace import Configuration, ConfigurationSpace, Float +from sklearn import datasets, svm +from sklearn.model_selection import StratifiedKFold + +from smac import HyperparameterOptimizationFacade, Scenario +from smac.intensifier import Intensifier + +# We load the digits dataset, a small-scale 10-class digit recognition dataset +X, y = datasets.load_digits(return_X_y=True) + + +class SVM: + @property + def configspace(self) -> ConfigurationSpace: + # Build Configuration Space which defines all parameters and their ranges + cs = ConfigurationSpace(seed=0) + + # First we create our hyperparameters + C = Float("C", (2 ** - 5, 2 ** 15), default=1.0, log=True) + gamma = Float("gamma", (2 ** -15, 2 ** 3), default=1.0, log=True) + + # Add hyperparameters to our configspace + cs.add_hyperparameters([C, gamma]) + + return cs + + def train(self, config: Configuration, instance: str, seed: int = 0) -> float: + """Creates a SVM based on a configuration and evaluate on the given fold of the digits dataset + + Parameters + ---------- + config: Configuration + The configuration to train the SVM. + instance: str + The name of the instance this configuration should be evaluated on. This is always of type + string by definition. In our case we cast to int, but this could also be the filename of a + problem instance to be loaded. + seed: int + The seed used for this call. + """ + instance = int(instance) + classifier = svm.SVC(**config, random_state=seed) + splitter = StratifiedKFold(n_splits=N_FOLDS, shuffle=True, random_state=seed) + for k, (train_idx, test_idx) in enumerate(splitter.split(X=X, y=y)): + if k != instance: + continue + else: + train_X = X[train_idx] + train_y = y[train_idx] + test_X = X[test_idx] + test_y = y[test_idx] + classifier.fit(train_X, train_y) + cost = 1 - classifier.score(test_X, test_y) + + return cost + + +if __name__ == "__main__": + classifier = SVM() + + # Next, we create an object, holding general information about the run + scenario = Scenario( + classifier.configspace, + n_trials=50, # We want to run max 50 trials (combination of config and instances in the case of + # deterministic=True. In the case of deterministic=False, this would be the + # combination of instances, seeds and configs). The number of distinct configurations + # evaluated by SMAC will be lower than this number because some of the configurations + # will be executed on more than one instance (CV fold). + instances=[f"{i}" for i in range(N_FOLDS)], # Specify all instances by their name (as a string) + instance_features={f"{i}": [i] for i in range(N_FOLDS)}, # breaks SMAC + deterministic=True # To simplify the problem we make SMAC believe that we have a deterministic + # optimization problem. + + ) + + # We want to run the facade's default initial design, but we want to change the number + # of initial configs to 5. + initial_design = HyperparameterOptimizationFacade.get_initial_design(scenario, n_configs=5) + + # Now we use SMAC to find the best hyperparameters + smac = HyperparameterOptimizationFacade( + scenario, + classifier.train, + initial_design=initial_design, + overwrite=True, # If the run exists, we overwrite it; alternatively, we can continue from last state + # The next line defines the intensifier, i.e., the module that governs the selection of + # instance-seed pairs. Since we set deterministic to True above, it only governs the instance in + # this example. Technically, it is not necessary to create the intensifier as a user, but it is + # necessary to do so because we change the argument max_config_calls (the number of instance-seed pairs + # per configuration to try) to the number of cross-validation folds, while the default would be 3. + intensifier=Intensifier(scenario=scenario, max_config_calls=N_FOLDS, seed=0) + ) + + incumbent = smac.optimize() + + # Get cost of default configuration + default_cost = smac.validate(classifier.configspace.get_default_configuration()) + print(f"Default cost: {default_cost}") + + # Let's calculate the cost of the incumbent + incumbent_cost = smac.validate(incumbent) + print(f"Incumbent cost: {incumbent_cost}") + + # Let's see how many configurations we have evaluated. If this number is higher than 5, we have looked + # at more configurations than would have been possible with regular cross-validation, where the number + # of configurations would be determined by the number of trials divided by the number of folds (50 / 10). + runhistory = smac.runhistory + print(f"Number of evaluated configurations: {len(runhistory.config_ids)}") \ No newline at end of file diff --git a/pyproject.toml b/pyproject.toml index c0e1ae4ed9..77783d8a77 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -65,6 +65,11 @@ add-ignore = [ # http://www.pydocstyle.org/en/stable/error_codes.html "D415", # First line should end with a period, question mark, or exclamation point ] +[tool.pylint."messages control"] +# FIXME: This is to do a staged introduction of pylint checks for just a single class of problems initially (#1067). +disable = ["all"] +enable = ["dangerous-default-value"] + [tool.mypy] python_version = "3.9" show_error_codes = true diff --git a/setup.py b/setup.py index f194d36771..27ec7e6954 100644 --- a/setup.py +++ b/setup.py @@ -38,6 +38,7 @@ def read_file(filepath: str) -> str: "pydocstyle", "flake8", "pre-commit", + "pylint", ], } diff --git a/smac/__init__.py b/smac/__init__.py index ab27812974..3241da0628 100644 --- a/smac/__init__.py +++ b/smac/__init__.py @@ -19,7 +19,7 @@ Copyright {datetime.date.today().strftime('%Y')}, Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhkopf, René Sass and Frank Hutter""" -version = "2.0.2" +version = "2.1.0" try: diff --git a/smac/acquisition/maximizer/__init__.py b/smac/acquisition/maximizer/__init__.py index 6fea59b459..5e3756190e 100644 --- a/smac/acquisition/maximizer/__init__.py +++ b/smac/acquisition/maximizer/__init__.py @@ -3,7 +3,6 @@ ) from smac.acquisition.maximizer.differential_evolution import DifferentialEvolution from smac.acquisition.maximizer.local_and_random_search import ( - LocalAndSortedPriorRandomSearch, LocalAndSortedRandomSearch, ) from smac.acquisition.maximizer.local_search import LocalSearch @@ -13,7 +12,6 @@ "AbstractAcquisitionMaximizer", "DifferentialEvolution", "LocalAndSortedRandomSearch", - "LocalAndSortedPriorRandomSearch", "LocalSearch", "RandomSearch", ] diff --git a/smac/acquisition/maximizer/abstract_acqusition_maximizer.py b/smac/acquisition/maximizer/abstract_acqusition_maximizer.py index 5a14a88141..e148cb6ede 100644 --- a/smac/acquisition/maximizer/abstract_acqusition_maximizer.py +++ b/smac/acquisition/maximizer/abstract_acqusition_maximizer.py @@ -27,12 +27,9 @@ class AbstractAcquisitionMaximizer: Parameters ---------- - configspace : ConfigurationSpace - acquisition_function : AbstractAcquisitionFunction - challengers : int, defaults to 5000 - Number of configurations to sample from the configuration space to get - the acquisition function value for, thus challenging the current - incumbent and becoming a candidate for the next function evaluation. + configspace : ConfigurationSpace acquisition_function : AbstractAcquisitionFunction + challengers : int, defaults to 5000 Number of configurations sampled during the optimization process, + details depend on the used maximizer. Also, the number of configurations that is returned by calling `maximize`. seed : int, defaults to 0 """ @@ -85,8 +82,8 @@ def maximize( previous_configs: list[Configuration] Previous evaluated configurations. n_points: int, defaults to None - Number of points to be sampled. If `n_points` is not specified, - `self._challengers` is used. + Number of points to be sampled & number of configurations to be returned. If `n_points` is not specified, + `self._challengers` is used. Semantics depend on concrete implementation. random_design: AbstractRandomDesign, defaults to None Part of the returned ChallengerList such that we can interleave random configurations by a scheme defined by the random design. The method `random_design.next_iteration()` diff --git a/smac/acquisition/maximizer/differential_evolution.py b/smac/acquisition/maximizer/differential_evolution.py index d1201c19c6..0f2ce15e60 100644 --- a/smac/acquisition/maximizer/differential_evolution.py +++ b/smac/acquisition/maximizer/differential_evolution.py @@ -30,6 +30,7 @@ def _maximize( previous_configs: list[Configuration], n_points: int, ) -> list[tuple[float, Configuration]]: + # n_points is not used here, but is required by the interface configs: list[tuple[float, Configuration]] = [] diff --git a/smac/acquisition/maximizer/helpers.py b/smac/acquisition/maximizer/helpers.py index d7319f20fb..db06e2aa46 100644 --- a/smac/acquisition/maximizer/helpers.py +++ b/smac/acquisition/maximizer/helpers.py @@ -4,8 +4,8 @@ from ConfigSpace import Configuration, ConfigurationSpace +from smac.random_design import ProbabilityRandomDesign from smac.random_design.abstract_random_design import AbstractRandomDesign -from smac.random_design.modulus_design import ModulusRandomDesign class ChallengerList(Iterator): @@ -20,7 +20,7 @@ class ChallengerList(Iterator): ---------- configspace : ConfigurationSpace challenger_callback : Callable - Callback function which returns a list of challengers (without interleaved random configurations, must a be a + Callback function which returns a list of challengers (without interleaved random configurations), must a be a python closure. random_design : AbstractRandomDesign | None, defaults to ModulusRandomDesign(modulus=2.0) Which random design should be used. @@ -30,7 +30,7 @@ def __init__( self, configspace: ConfigurationSpace, challenger_callback: Callable, - random_design: AbstractRandomDesign | None = ModulusRandomDesign(modulus=2.0), + random_design: AbstractRandomDesign | None = ProbabilityRandomDesign(seed=0, probability=0.08447232371720552), ): self._challengers_callback = challenger_callback self._challengers: list[Configuration] | None = None @@ -72,44 +72,3 @@ def __len__(self) -> int: self._challengers = self._challengers_callback() return len(self._challengers) - self._index - - -''' -class FixedSet(AbstractAcquisitionMaximizer): - def __init__( - self, - configurations: list[Configuration], - acquisition_function: AbstractAcquisitionFunction, - configspace: ConfigurationSpace, - challengers: int = 5000, - seed: int = 0, - ): - """Maximize the acquisition function over a finite list of configurations. - - Parameters - ---------- - configurations : list[~smac._configspace.Configuration] - Candidate configurations - acquisition_function : ~smac.acquisition.AbstractAcquisitionFunction - - configspace : ~smac._configspace.ConfigurationSpace - - rng : np.random.RandomState or int, optional - """ - super().__init__( - acquisition_function=acquisition_function, configspace=configspace, challengers=challengers, seed=seed - ) - self.configurations = configurations - - def _maximize( - self, - runhistory: RunHistory, - stats: Stats, - n_points: int, - ) -> list[tuple[float, Configuration]]: - configurations = copy.deepcopy(self.configurations) - for config in configurations: - config.origin = "Fixed Set" - - return self._sort_by_acquisition_value(configurations) -''' diff --git a/smac/acquisition/maximizer/local_and_random_search.py b/smac/acquisition/maximizer/local_and_random_search.py index bb6daaef51..d81af2f699 100644 --- a/smac/acquisition/maximizer/local_and_random_search.py +++ b/smac/acquisition/maximizer/local_and_random_search.py @@ -21,7 +21,7 @@ class LocalAndSortedRandomSearch(AbstractAcquisitionMaximizer): """Implement SMAC's default acquisition function optimization. - This optimizer performs local search from the previous best points according, to the acquisition + This optimizer performs local search from the previous best points according to the acquisition function, uses the acquisition function to sort randomly sampled configurations. Random configurations are interleaved by the main SMAC code. @@ -31,6 +31,10 @@ class LocalAndSortedRandomSearch(AbstractAcquisitionMaximizer): Parameters ---------- configspace : ConfigurationSpace + uniform_configspace : ConfigurationSpace + A version of the user-defined ConfigurationSpace where all parameters are uniform (or have their weights removed + in the case of a categorical hyperparameter). Can optionally be given and sampling ratios be defined via the + `prior_sampling_fraction` parameter. acquisition_function : AbstractAcquisitionFunction | None, defaults to None challengers : int, defaults to 5000 Number of challengers. @@ -40,6 +44,9 @@ class LocalAndSortedRandomSearch(AbstractAcquisitionMaximizer): [LocalSearch] number of steps during a plateau walk before local search terminates. local_search_iterations: int, defauts to 10 [Local Search] number of local search iterations. + prior_sampling_fraction: float, defaults to 0.5 + The ratio of random samples that are taken from the user-defined ConfigurationSpace, as opposed to the uniform + version (needs `uniform_configspace`to be defined). seed : int, defaults to 0 """ @@ -52,6 +59,8 @@ def __init__( n_steps_plateau_walk: int = 10, local_search_iterations: int = 10, seed: int = 0, + uniform_configspace: ConfigurationSpace | None = None, + prior_sampling_fraction: float | None = None, ) -> None: super().__init__( configspace, @@ -60,11 +69,28 @@ def __init__( seed=seed, ) - self._random_search = RandomSearch( - configspace=configspace, - acquisition_function=acquisition_function, - seed=seed, - ) + if uniform_configspace is not None and prior_sampling_fraction is None: + prior_sampling_fraction = 0.5 + if uniform_configspace is None and prior_sampling_fraction is not None: + raise ValueError("If `prior_sampling_fraction` is given, `uniform_configspace` must be defined.") + if uniform_configspace is not None and prior_sampling_fraction is not None: + self._prior_random_search = RandomSearch( + acquisition_function=acquisition_function, + configspace=configspace, + seed=seed, + ) + + self._uniform_random_search = RandomSearch( + acquisition_function=acquisition_function, + configspace=uniform_configspace, + seed=seed, + ) + else: + self._random_search = RandomSearch( + configspace=configspace, + acquisition_function=acquisition_function, + seed=seed, + ) self._local_search = LocalSearch( configspace=configspace, @@ -75,6 +101,8 @@ def __init__( ) self._local_search_iterations = local_search_iterations + self._prior_sampling_fraction = prior_sampling_fraction + self._uniform_configspace = uniform_configspace @property def acquisition_function(self) -> AbstractAcquisitionFunction | None: # noqa: D102 @@ -84,18 +112,31 @@ def acquisition_function(self) -> AbstractAcquisitionFunction | None: # noqa: D @acquisition_function.setter def acquisition_function(self, acquisition_function: AbstractAcquisitionFunction) -> None: self._acquisition_function = acquisition_function - self._random_search._acquisition_function = acquisition_function + if self._uniform_configspace is not None: + self._prior_random_search._acquisition_function = acquisition_function + self._uniform_random_search._acquisition_function = acquisition_function + else: + self._random_search._acquisition_function = acquisition_function self._local_search._acquisition_function = acquisition_function @property def meta(self) -> dict[str, Any]: # noqa: D102 meta = super().meta - meta.update( - { - "random_search": self._random_search.meta, - "local_search": self._local_search.meta, - } - ) + if self._uniform_configspace is None: + meta.update( + { + "random_search": self._random_search.meta, + "local_search": self._local_search.meta, + } + ) + else: + meta.update( + { + "prior_random_search": self._prior_random_search.meta, + "uniform_random_search": self._uniform_random_search.meta, + "local_search": self._local_search.meta, + } + ) return meta @@ -105,147 +146,45 @@ def _maximize( n_points: int, ) -> list[tuple[float, Configuration]]: - # Get configurations sorted by EI - next_configs_by_random_search_sorted = self._random_search._maximize( - previous_configs=previous_configs, - n_points=n_points, - _sorted=True, - ) - + if self._uniform_configspace is not None and self._prior_sampling_fraction is not None: + # Get configurations sorted by acquisition function value + next_configs_by_prior_random_search_sorted = self._prior_random_search._maximize( + previous_configs, + round(n_points * self._prior_sampling_fraction), + _sorted=True, + ) + + # Get configurations sorted by acquisition function value + next_configs_by_uniform_random_search_sorted = self._uniform_random_search._maximize( + previous_configs, + round(n_points * (1 - self._prior_sampling_fraction)), + _sorted=True, + ) + next_configs_by_random_search_sorted = ( + next_configs_by_uniform_random_search_sorted + next_configs_by_prior_random_search_sorted + ) + next_configs_by_random_search_sorted.sort(reverse=True, key=lambda x: x[0]) + else: + # Get configurations sorted by acquisition function value + next_configs_by_random_search_sorted = self._random_search._maximize( + previous_configs=previous_configs, + n_points=n_points, + _sorted=True, + ) + + # Choose the best self._local_search_iterations random configs to start the local search, and choose only + # incumbent from previous configs + random_starting_points = next_configs_by_random_search_sorted[: self._local_search_iterations] next_configs_by_local_search = self._local_search._maximize( previous_configs=previous_configs, n_points=self._local_search_iterations, - additional_start_points=next_configs_by_random_search_sorted, + additional_start_points=random_starting_points, ) - # Having the configurations from random search, sorted by their - # acquisition function value is important for the first few iterations - # of SMAC. As long as the random forest predicts constant value, we - # want to use only random configurations. Having them at the begging of - # the list ensures this (even after adding the configurations by local - # search, and then sorting them) - next_configs_by_acq_value = next_configs_by_random_search_sorted + next_configs_by_local_search + next_configs_by_acq_value = next_configs_by_local_search next_configs_by_acq_value.sort(reverse=True, key=lambda x: x[0]) first_five = [f"{_[0]} ({_[1].origin})" for _ in next_configs_by_acq_value[:5]] logger.debug(f"First 5 acquisition function values of selected configurations:\n{', '.join(first_five)}") return next_configs_by_acq_value - - -class LocalAndSortedPriorRandomSearch(AbstractAcquisitionMaximizer): - """Implements SMAC's default acquisition function optimization. - - This optimizer performs local search from the previous best points according to the acquisition function, uses the - acquisition function to sort randomly sampled configurations. Random configurations are interleaved by the main SMAC - code. The random configurations are retrieved from two different ConfigurationSpaces - one which uses priors - (e.g. NormalFloatHP) and is defined by the user, and one that is a uniform version of the same space, i.e. with the - priors removed. - - Parameters - ---------- - configspace : ConfigurationSpace - The original ConfigurationSpace specified by the user. - uniform_configspace : ConfigurationSpace - A version of the user-defined ConfigurationSpace where all parameters are uniform (or have their weights removed - in the case of a categorical hyperparameter). - acquisition_function : AbstractAcquisitionFunction | None, defaults to None - challengers : int, defaults to 5000 - Number of challengers. - max_steps: int, defaults to None - [LocalSearch] Maximum number of steps that the local search will perform. - n_steps_plateau_walk: int, defaults to 10 - [LocalSearch] number of steps during a plateau walk before local search terminates. - local_search_iterations: int, defaults to 10 - [Local Search] number of local search iterations. - prior_sampling_fraction: float, defaults to 0.5 - The ratio of random samples that are taken from the user-defined ConfigurationSpace, as opposed to the uniform - version. - seed : int, defaults to 0 - """ - - def __init__( - self, - configspace: ConfigurationSpace, - uniform_configspace: ConfigurationSpace, - acquisition_function: AbstractAcquisitionFunction | None = None, - challengers: int = 5000, - max_steps: int | None = None, - n_steps_plateau_walk: int = 10, - local_search_iterations: int = 10, - prior_sampling_fraction: float = 0.5, - seed: int = 0, - ) -> None: - super().__init__( - acquisition_function, - configspace, - challengers=challengers, - seed=seed, - ) - - self._prior_random_search = RandomSearch( - acquisition_function=acquisition_function, - configspace=configspace, - seed=seed, - ) - - self._uniform_random_search = RandomSearch( - acquisition_function=acquisition_function, - configspace=uniform_configspace, - seed=seed, - ) - - self._local_search = LocalSearch( - acquisition_function=acquisition_function, - configspace=configspace, - max_steps=max_steps, - n_steps_plateau_walk=n_steps_plateau_walk, - seed=seed, - ) - - self._local_search_iterations = local_search_iterations - self._prior_sampling_fraction = prior_sampling_fraction - - def _maximize( - self, - previous_configs: list[Configuration], - n_points: int, - ) -> list[tuple[float, Configuration]]: - - # Get configurations sorted by EI - next_configs_by_prior_random_search_sorted = self._prior_random_search._maximize( - previous_configs, - round(n_points * self._prior_sampling_fraction), - _sorted=True, - ) - - # Get configurations sorted by EI - next_configs_by_uniform_random_search_sorted = self._uniform_random_search._maximize( - previous_configs, - round(n_points * (1 - self._prior_sampling_fraction)), - _sorted=True, - ) - next_configs_by_random_search_sorted = [] - next_configs_by_random_search_sorted.extend(next_configs_by_prior_random_search_sorted) - next_configs_by_random_search_sorted.extend(next_configs_by_uniform_random_search_sorted) - - next_configs_by_local_search = self._local_search._maximize( - previous_configs, - self._local_search_iterations, - additional_start_points=next_configs_by_random_search_sorted, - ) - - # Having the configurations from random search, sorted by their - # acquisition function value is important for the first few iterations - # of SMAC. As long as the random forest predicts constant value, we - # want to use only random configurations. Having them at the begging of - # the list ensures this (even after adding the configurations by local - # search, and then sorting them) - next_configs_by_acq_value = next_configs_by_random_search_sorted + next_configs_by_local_search - next_configs_by_acq_value.sort(reverse=True, key=lambda x: x[0]) - logger.debug( - "First 5 acq func (origin) values of selected configurations: %s", - str([[_[0], _[1].origin] for _ in next_configs_by_acq_value[:5]]), - ) - - return next_configs_by_acq_value diff --git a/smac/acquisition/maximizer/local_search.py b/smac/acquisition/maximizer/local_search.py index c6f545f9a6..3ef1ae96e7 100644 --- a/smac/acquisition/maximizer/local_search.py +++ b/smac/acquisition/maximizer/local_search.py @@ -90,9 +90,9 @@ def _maximize( n_points: int, additional_start_points: list[tuple[float, Configuration]] | None = None, ) -> list[tuple[float, Configuration]]: - """Start a local search from the given startpoints. Iteratively collect neighbours + """Start a local search from the given start points. Iteratively collect neighbours using Configspace.utils.get_one_exchange_neighbourhood and evaluate them. - If the new config is better than the current best, the local search is coninued from the + If the new config is better than the current best, the local search is continued from the new config. Quit if either the max number of steps is reached or @@ -149,16 +149,22 @@ def _get_initial_points( list[Configuration] A list of initial points/configurations. """ - if len(previous_configs) == 0: - init_points = self._configspace.sample_configuration(size=n_points) - else: + sampled_points = [] + init_points = [] + n_init_points = n_points + if len(previous_configs) < n_points: + sampled_points = self._configspace.sample_configuration(size=n_points - len(previous_configs)) + n_init_points = len(previous_configs) + if not isinstance(sampled_points, list): + sampled_points = [sampled_points] + if len(previous_configs) > 0: init_points = self._get_init_points_from_previous_configs( previous_configs, - n_points, + n_init_points, additional_start_points, ) - return init_points + return sampled_points + init_points def _get_init_points_from_previous_configs( self, @@ -187,7 +193,7 @@ def _get_init_points_from_previous_configs( previous_configs: list[Configuration] Previous configuration (e.g., from the runhistory). n_points: int - Number of initial points to be generated. + Number of initial points to be generated; selected from previous configs (+ random configs if necessary). additional_start_points: list[tuple[float, Configuration]] | None Additional starting points. @@ -198,10 +204,6 @@ def _get_init_points_from_previous_configs( """ assert self._acquisition_function is not None - # configurations with the highest previous EI - configs_previous_runs_sorted = self._sort_by_acquisition_value(previous_configs) - configs_previous_runs_sorted = [conf[1] for conf in configs_previous_runs_sorted[:n_points]] - # configurations with the lowest predictive cost, check for None to make unit tests work if self._acquisition_function.model is not None: conf_array = convert_configurations_to_array(previous_configs) @@ -228,14 +230,13 @@ def _get_init_points_from_previous_configs( previous_configs_sorted_by_cost = [] if additional_start_points is not None: - additional_start_points = [asp[1] for asp in additional_start_points[:n_points]] + additional_start_points = [asp[1] for asp in additional_start_points] else: additional_start_points = [] init_points = [] init_points_as_set: set[Configuration] = set() for cand in itertools.chain( - configs_previous_runs_sorted, previous_configs_sorted_by_cost, additional_start_points, ): diff --git a/smac/facade/abstract_facade.py b/smac/facade/abstract_facade.py index a5db5ab521..6ca49b057c 100644 --- a/smac/facade/abstract_facade.py +++ b/smac/facade/abstract_facade.py @@ -117,12 +117,15 @@ def __init__( runhistory_encoder: AbstractRunHistoryEncoder | None = None, config_selector: ConfigSelector | None = None, logging_level: int | Path | Literal[False] | None = None, - callbacks: list[Callback] = [], + callbacks: list[Callback] = None, overwrite: bool = False, dask_client: Client | None = None, ): setup_logging(logging_level) + if callbacks is None: + callbacks = [] + if model is None: model = self.get_model(scenario) @@ -190,7 +193,7 @@ def __init__( # In case of multiple jobs, we need to wrap the runner again using DaskParallelRunner if (n_workers := scenario.n_workers) > 1 or dask_client is not None: - if dask_client is not None: + if dask_client is not None and n_workers > 1: logger.warning( "Provided `dask_client`. Ignore `scenario.n_workers`, directly set `n_workers` in `dask_client`." ) diff --git a/smac/facade/algorithm_configuration_facade.py b/smac/facade/algorithm_configuration_facade.py index a82e2f92ca..58229f4b72 100644 --- a/smac/facade/algorithm_configuration_facade.py +++ b/smac/facade/algorithm_configuration_facade.py @@ -125,7 +125,7 @@ def get_intensifier( def get_initial_design( # type: ignore scenario: Scenario, *, - additional_configs: list[Configuration] = [], + additional_configs: list[Configuration] = None, ) -> DefaultInitialDesign: """Returns an initial design, which returns the default configuration. @@ -134,6 +134,8 @@ def get_initial_design( # type: ignore additional_configs: list[Configuration], defaults to [] Adds additional configurations to the initial design. """ + if additional_configs is None: + additional_configs = [] return DefaultInitialDesign( scenario=scenario, additional_configs=additional_configs, diff --git a/smac/facade/blackbox_facade.py b/smac/facade/blackbox_facade.py index 9e2d4b2d1d..a5e1a18ab5 100644 --- a/smac/facade/blackbox_facade.py +++ b/smac/facade/blackbox_facade.py @@ -11,6 +11,7 @@ from smac.facade.abstract_facade import AbstractFacade from smac.initial_design.sobol_design import SobolInitialDesign from smac.intensifier.intensifier import Intensifier +from smac.main.config_selector import ConfigSelector from smac.model.gaussian_process.abstract_gaussian_process import ( AbstractGaussianProcess, ) @@ -240,7 +241,7 @@ def get_initial_design( # type: ignore n_configs: int | None = None, n_configs_per_hyperparamter: int = 8, max_ratio: float = 0.25, - additional_configs: list[Configuration] = [], + additional_configs: list[Configuration] = None, ) -> SobolInitialDesign: """Returns a Sobol design instance. @@ -259,6 +260,8 @@ def get_initial_design( # type: ignore additional_configs: list[Configuration], defaults to [] Adds additional configurations to the initial design. """ + if additional_configs is None: + additional_configs = [] return SobolInitialDesign( scenario=scenario, n_configs=n_configs, @@ -309,3 +312,15 @@ def get_runhistory_encoder( ) -> RunHistoryEncoder: """Returns the default runhistory encoder.""" return RunHistoryEncoder(scenario) + + @staticmethod + def get_config_selector( + scenario: Scenario, + *, + retrain_after: int = 1, + retries: int = 16, + ) -> ConfigSelector: + """Returns the default configuration selector.""" + return super(BlackBoxFacade, BlackBoxFacade).get_config_selector( + scenario, retrain_after=retrain_after, retries=retries + ) diff --git a/smac/facade/hyperparameter_optimization_facade.py b/smac/facade/hyperparameter_optimization_facade.py index 70a85517cd..3ed09cfdec 100644 --- a/smac/facade/hyperparameter_optimization_facade.py +++ b/smac/facade/hyperparameter_optimization_facade.py @@ -138,7 +138,7 @@ def get_initial_design( # type: ignore n_configs: int | None = None, n_configs_per_hyperparamter: int = 10, max_ratio: float = 0.25, - additional_configs: list[Configuration] = [], + additional_configs: list[Configuration] | None = None, ) -> SobolInitialDesign: """Returns a Sobol design instance. @@ -178,7 +178,7 @@ def get_random_design( # type: ignore probability : float, defaults to 0.2 Probability that a configuration will be drawn at random. """ - return ProbabilityRandomDesign(probability=probability) + return ProbabilityRandomDesign(probability=probability, seed=scenario.seed) @staticmethod def get_multi_objective_algorithm( # type: ignore diff --git a/smac/facade/multi_fidelity_facade.py b/smac/facade/multi_fidelity_facade.py index fdd52d8a4c..717162c09e 100644 --- a/smac/facade/multi_fidelity_facade.py +++ b/smac/facade/multi_fidelity_facade.py @@ -67,7 +67,7 @@ def get_initial_design( # type: ignore n_configs: int | None = None, n_configs_per_hyperparamter: int = 10, max_ratio: float = 0.25, - additional_configs: list[Configuration] = [], + additional_configs: list[Configuration] = None, ) -> RandomInitialDesign: """Returns a random initial design. @@ -80,12 +80,14 @@ def get_initial_design( # type: ignore Number of initial configurations per hyperparameter. For example, if my configuration space covers five hyperparameters and ``n_configs_per_hyperparameter`` is set to 10, then 50 initial configurations will be samples. - max_ratio: float, defaults to 0.1 + max_ratio: float, defaults to 0.25 Use at most ``scenario.n_trials`` * ``max_ratio`` number of configurations in the initial design. Additional configurations are not affected by this parameter. additional_configs: list[Configuration], defaults to [] Adds additional configurations to the initial design. """ + if additional_configs is None: + additional_configs = [] return RandomInitialDesign( scenario=scenario, n_configs=n_configs, diff --git a/smac/facade/random_facade.py b/smac/facade/random_facade.py index d5e269534c..78038ff2e3 100644 --- a/smac/facade/random_facade.py +++ b/smac/facade/random_facade.py @@ -92,7 +92,7 @@ def get_intensifier( def get_initial_design( scenario: Scenario, *, - additional_configs: list[Configuration] = [], + additional_configs: list[Configuration] = None, ) -> DefaultInitialDesign: """Returns an initial design, which returns the default configuration. @@ -101,6 +101,8 @@ def get_initial_design( additional_configs: list[Configuration], defaults to [] Adds additional configurations to the initial design. """ + if additional_configs is None: + additional_configs = [] return DefaultInitialDesign( scenario=scenario, additional_configs=additional_configs, diff --git a/smac/intensifier/abstract_intensifier.py b/smac/intensifier/abstract_intensifier.py index b944867273..cb537e9ccb 100644 --- a/smac/intensifier/abstract_intensifier.py +++ b/smac/intensifier/abstract_intensifier.py @@ -571,8 +571,12 @@ def update_incumbents(self, config: Configuration) -> None: if len(previous_incumbents) == len(new_incumbents): if previous_incumbents == new_incumbents: - # No changes in the incumbents - self._remove_rejected_config(config_id) + # No changes in the incumbents, we need this clause because we can't use set difference then + if config_id in new_incumbent_ids: + self._remove_rejected_config(config_id) + else: + # config worse than incumbents and thus rejected + self._add_rejected_config(config_id) return else: # In this case, we have to determine which config replaced which incumbent and reject it diff --git a/smac/main/config_selector.py b/smac/main/config_selector.py index 7b2053eee2..4e3574d589 100644 --- a/smac/main/config_selector.py +++ b/smac/main/config_selector.py @@ -91,7 +91,7 @@ def _set_components( acquisition_maximizer: AbstractAcquisitionMaximizer, acquisition_function: AbstractAcquisitionFunction, random_design: AbstractRandomDesign, - callbacks: list[Callback] = [], + callbacks: list[Callback] = None, ) -> None: self._runhistory = runhistory self._runhistory_encoder = runhistory_encoder @@ -99,7 +99,7 @@ def _set_components( self._acquisition_maximizer = acquisition_maximizer self._acquisition_function = acquisition_function self._random_design = random_design - self._callbacks = callbacks + self._callbacks = callbacks if callbacks is not None else [] self._initial_design_configs = initial_design.select_configurations() if len(self._initial_design_configs) == 0: @@ -213,7 +213,6 @@ def __iter__(self) -> Iterator[Configuration]: # Now we maximize the acquisition function challengers = self._acquisition_maximizer.maximize( previous_configs, - n_points=self._retrain_after, random_design=self._random_design, ) @@ -282,7 +281,7 @@ def _collect_data(self) -> tuple[np.ndarray, np.ndarray, np.ndarray]: if X.shape[0] >= self._min_trials: self._considered_budgets = [b] - # TODO: Add running configs + # Possible add running configs? configs_array = self._runhistory_encoder.get_configurations(budget_subset=self._considered_budgets) return X, Y, configs_array diff --git a/smac/model/random_forest/abstract_random_forest.py b/smac/model/random_forest/abstract_random_forest.py index d4f1f7ce30..e407331be5 100644 --- a/smac/model/random_forest/abstract_random_forest.py +++ b/smac/model/random_forest/abstract_random_forest.py @@ -6,6 +6,7 @@ from ConfigSpace import ( CategoricalHyperparameter, Constant, + OrdinalHyperparameter, UniformFloatHyperparameter, UniformIntegerHyperparameter, ) @@ -36,12 +37,14 @@ def _impute_inactive(self, X: np.ndarray) -> np.ndarray: self._conditional[idx] = True if isinstance(hp, CategoricalHyperparameter): self._impute_values[idx] = len(hp.choices) + elif isinstance(hp, OrdinalHyperparameter): + self._impute_values[idx] = len(hp.sequence) elif isinstance(hp, (UniformFloatHyperparameter, UniformIntegerHyperparameter)): self._impute_values[idx] = -1 elif isinstance(hp, Constant): self._impute_values[idx] = 1 else: - raise ValueError + raise ValueError(f"Unsupported hyperparameter type: {type(hp)}") if self._conditional[idx] is True: nonfinite_mask = ~np.isfinite(X[:, idx]) diff --git a/smac/model/random_forest/random_forest.py b/smac/model/random_forest/random_forest.py index 6d7840d2ac..634e0af06f 100644 --- a/smac/model/random_forest/random_forest.py +++ b/smac/model/random_forest/random_forest.py @@ -87,7 +87,9 @@ def __init__( self._rf_opts.compute_law_of_total_variance = False self._rf: BinaryForest | None = None self._log_y = log_y - self._rng = regression.default_random_engine(seed) + + # Case to `int` incase we get an `np.integer` type + self._rng = regression.default_random_engine(int(seed)) self._n_trees = n_trees self._n_points_per_tree = n_points_per_tree diff --git a/smac/runhistory/encoder/abstract_encoder.py b/smac/runhistory/encoder/abstract_encoder.py index 42454771ed..9a891229a4 100644 --- a/smac/runhistory/encoder/abstract_encoder.py +++ b/smac/runhistory/encoder/abstract_encoder.py @@ -42,17 +42,17 @@ class AbstractRunHistoryEncoder: def __init__( self, scenario: Scenario, - considered_states: list[StatusType] = [ - StatusType.SUCCESS, - StatusType.CRASHED, - StatusType.MEMORYOUT, - ], - lower_budget_states: list[StatusType] = [], + considered_states: list[StatusType] = None, + lower_budget_states: list[StatusType] = None, scale_percentage: int = 5, seed: int | None = None, ) -> None: if considered_states is None: - raise TypeError("No success states are given.") + considered_states = [ + StatusType.SUCCESS, + StatusType.CRASHED, + StatusType.MEMORYOUT, + ] if seed is None: seed = scenario.seed @@ -62,7 +62,7 @@ def __init__( self._scale_percentage = scale_percentage self._n_objectives = scenario.count_objectives() self._algorithm_walltime_limit = scenario.trial_walltime_limit - self._lower_budget_states = lower_budget_states + self._lower_budget_states = lower_budget_states if lower_budget_states is not None else [] self._considered_states = considered_states self._instances = scenario.instances diff --git a/smac/runhistory/runhistory.py b/smac/runhistory/runhistory.py index aaf88889c8..c713841074 100644 --- a/smac/runhistory/runhistory.py +++ b/smac/runhistory/runhistory.py @@ -178,7 +178,7 @@ def add( budget: float | None = None, starttime: float = 0.0, endtime: float = 0.0, - additional_info: dict[str, Any] = {}, + additional_info: dict[str, Any] = None, force_update: bool = False, ) -> None: """Adds a new trial to the RunHistory. @@ -205,6 +205,8 @@ def add( raise TypeError("Configuration must not be None.") elif not isinstance(config, Configuration): raise TypeError("Configuration is not of type Configuration, but %s." % type(config)) + if additional_info is None: + additional_info = {} # Squeeze is important to reduce arrays with one element # to scalars. diff --git a/smac/runner/abstract_runner.py b/smac/runner/abstract_runner.py index 2631bd966d..4e15cf4b5f 100644 --- a/smac/runner/abstract_runner.py +++ b/smac/runner/abstract_runner.py @@ -50,8 +50,10 @@ class AbstractRunner(ABC): def __init__( self, scenario: Scenario, - required_arguments: list[str] = [], + required_arguments: list[str] = None, ): + if required_arguments is None: + required_arguments = [] self._scenario = scenario self._required_arguments = required_arguments diff --git a/smac/runner/dask_runner.py b/smac/runner/dask_runner.py index b9aade4015..d4bb528bdc 100644 --- a/smac/runner/dask_runner.py +++ b/smac/runner/dask_runner.py @@ -91,7 +91,7 @@ def __init__( ) if self._scenario.output_directory is not None: - self._scheduler_file = self._scenario.output_directory / ".dask_scheduler_file" + self._scheduler_file = Path(self._scenario.output_directory, ".dask_scheduler_file") self._client.write_scheduler_file(scheduler_file=str(self._scheduler_file)) else: # We just use their set up diff --git a/smac/runner/target_function_runner.py b/smac/runner/target_function_runner.py index 5431f5ed47..8a14c2fc9e 100644 --- a/smac/runner/target_function_runner.py +++ b/smac/runner/target_function_runner.py @@ -45,8 +45,10 @@ def __init__( self, scenario: Scenario, target_function: Callable, - required_arguments: list[str] = [], + required_arguments: list[str] = None, ): + if required_arguments is None: + required_arguments = [] super().__init__(scenario=scenario, required_arguments=required_arguments) self._target_function = target_function diff --git a/smac/runner/target_function_script_runner.py b/smac/runner/target_function_script_runner.py index 17feffc983..5ab558906d 100644 --- a/smac/runner/target_function_script_runner.py +++ b/smac/runner/target_function_script_runner.py @@ -51,8 +51,10 @@ def __init__( self, target_function: str, scenario: Scenario, - required_arguments: list[str] = [], + required_arguments: list[str] = None, ): + if required_arguments is None: + required_arguments = [] super().__init__(scenario=scenario, required_arguments=required_arguments) self._target_function = target_function diff --git a/tests/test_acquisition/test_maximizers.py b/tests/test_acquisition/test_maximizers.py index 4cd4110f34..6d373c237c 100644 --- a/tests/test_acquisition/test_maximizers.py +++ b/tests/test_acquisition/test_maximizers.py @@ -30,7 +30,6 @@ from smac.acquisition.function import EI from smac.acquisition.maximizer import ( DifferentialEvolution, - LocalAndSortedPriorRandomSearch, LocalAndSortedRandomSearch, LocalSearch, RandomSearch, @@ -278,7 +277,7 @@ def __call__(self, X): random_configs = configspace.sample_configuration(size=100) points = ls._get_initial_points(random_configs, n_points=5, additional_start_points=None) - assert len(points) == 10 + assert len(points) == 5 # -------------------------------------------------------------- @@ -342,7 +341,6 @@ def test_local_and_random_search(configspace, acquisition_function): assert v_old >= v v_old = v - assert "Acquisition Function Maximizer: Random Search (sorted)" in config_origins assert "Acquisition Function Maximizer: Local Search" in config_origins @@ -390,26 +388,26 @@ def __call__(self, arrays): budget_kwargs = {"max_steps": 2, "n_steps_plateau_walk": 2, "local_search_iterations": 2} - prs_0 = LocalAndSortedPriorRandomSearch( - configspace_prior, - configspace_rosenbrock, - AcquisitionFunction(), + prs_0 = LocalAndSortedRandomSearch( + configspace=configspace_prior, + uniform_configspace=configspace_rosenbrock, + acquisition_function=AcquisitionFunction(), prior_sampling_fraction=0, **budget_kwargs, ) - prs_05 = LocalAndSortedPriorRandomSearch( - configspace_prior, - configspace_rosenbrock, - AcquisitionFunction(), + prs_05 = LocalAndSortedRandomSearch( + configspace=configspace_prior, + uniform_configspace=configspace_rosenbrock, + acquisition_function=AcquisitionFunction(), prior_sampling_fraction=0.9, **budget_kwargs, ) - prs_1 = LocalAndSortedPriorRandomSearch( - configspace_prior, - configspace_rosenbrock, - AcquisitionFunction(), + prs_1 = LocalAndSortedRandomSearch( + configspace=configspace_prior, + uniform_configspace=configspace_rosenbrock, + acquisition_function=AcquisitionFunction(), prior_sampling_fraction=1, **budget_kwargs, ) diff --git a/tests/test_intensifier/test_abstract_intensifier.py b/tests/test_intensifier/test_abstract_intensifier.py index ce980c49b9..b8dc91a1cb 100644 --- a/tests/test_intensifier/test_abstract_intensifier.py +++ b/tests/test_intensifier/test_abstract_intensifier.py @@ -109,6 +109,48 @@ def test_incumbent_selection_multi_objective(make_scenario, configspace_small, m assert intensifier.get_incumbents() == [config] +def test_config_rejection_single_objective(configspace_small, make_scenario): + """ Tests whether configs are rejected properly if they are worse than the incumbent. """ + scenario = make_scenario(configspace_small, use_instances=False) + runhistory = RunHistory() + intensifier = Intensifier(scenario=scenario) + intensifier.runhistory = runhistory + + configs = configspace_small.sample_configuration(3) + + runhistory.add(config=configs[0], + cost=5, + time=0.0, + seed=0, + status=StatusType.SUCCESS, + force_update=True) + intensifier.update_incumbents(configs[0]) + + assert intensifier._rejected_config_ids == [] + + # add config that yielded better results, updating incumbent and sending prior incumbent to rejected + runhistory.add(config=configs[1], + cost=1, + time=0.0, + seed=0, + status=StatusType.SUCCESS, + force_update=True) + intensifier.update_incumbents(config=configs[1]) + + assert intensifier._rejected_config_ids == [1] + + # add config that is no better should thus go to rejected + runhistory.add(config=configs[2], + cost=1, + time=0.0, + seed=0, + status=StatusType.SUCCESS, + force_update=True) + intensifier.update_incumbents(config=configs[2]) + + assert intensifier._rejected_config_ids == [1, 3] + + def test_incumbent_differences(make_scenario, configspace_small): pass diff --git a/tests/test_model/test_rf.py b/tests/test_model/test_rf.py index a30995b237..e59b2ded11 100644 --- a/tests/test_model/test_rf.py +++ b/tests/test_model/test_rf.py @@ -257,24 +257,36 @@ def test_with_ordinal(): def test_impute_inactive_hyperparameters(): cs = ConfigurationSpace(seed=0) - a = cs.add_hyperparameter(CategoricalHyperparameter("a", [0, 1])) + a = cs.add_hyperparameter(CategoricalHyperparameter("a", [0, 1, 2])) b = cs.add_hyperparameter(CategoricalHyperparameter("b", [0, 1])) c = cs.add_hyperparameter(UniformFloatHyperparameter("c", 0, 1)) + d = cs.add_hyperparameter(OrdinalHyperparameter("d", [0, 1, 2])) cs.add_condition(EqualsCondition(b, a, 1)) cs.add_condition(EqualsCondition(c, a, 0)) + cs.add_condition(EqualsCondition(d, a, 2)) configs = cs.sample_configuration(size=100) config_array = convert_configurations_to_array(configs) for line in config_array: if line[0] == 0: assert np.isnan(line[1]) + assert np.isnan(line[3]) elif line[0] == 1: assert np.isnan(line[2]) + assert np.isnan(line[3]) + elif line[0] == 2: + assert np.isnan(line[1]) + assert np.isnan(line[2]) model = RandomForest(configspace=cs) config_array = model._impute_inactive(config_array) for line in config_array: if line[0] == 0: assert line[1] == 2 + assert line[3] == 3 elif line[0] == 1: assert line[2] == -1 + assert line[3] == 3 + elif line[0] == 2: + assert line[1] == 2 + assert line[2] == -1 diff --git a/tests/test_runhistory/test_runhistory_encoder.py b/tests/test_runhistory/test_runhistory_encoder.py index 7b30bb76a5..e3a8f57303 100644 --- a/tests/test_runhistory/test_runhistory_encoder.py +++ b/tests/test_runhistory/test_runhistory_encoder.py @@ -13,6 +13,9 @@ from smac.runhistory.encoder.encoder import RunHistoryEncoder from smac.runner.abstract_runner import StatusType +from ConfigSpace import Configuration +from ConfigSpace.hyperparameters import CategoricalHyperparameter + @pytest.fixture def configs(configspace_small): @@ -39,70 +42,127 @@ def test_transform(runhistory, make_scenario, configspace_small, configs): ) # Normal encoder - encoder = RunHistoryEncoder(scenario=scenario, considered_states=[StatusType.SUCCESS]) + encoder = RunHistoryEncoder( + scenario=scenario, considered_states=[StatusType.SUCCESS] + ) encoder.runhistory = runhistory + + # TODO: Please replace with the more general solution once ConfigSpace 1.0 + # upper = np.array([hp.upper_vectorized for hp in space.values()]) + # lower = np.array([hp.lower_vectorized for hp in space.values()]) + # - + # Categoricals are upperbounded by their size, rest of hyperparameters are + # upperbounded by 1. + upper_bounds = { + hp.name: (hp.get_size() - 1) + if isinstance(hp, CategoricalHyperparameter) + else 1.0 + for hp in configspace_small.get_hyperparameters() + } + # Need to ensure they match the order in the Configuration vectorized form + sorted_by_indices = sorted( + upper_bounds.items(), + key=lambda x: configspace_small._hyperparameter_idx[x[0]], + ) + upper = np.array([upper_bound for _, upper_bound in sorted_by_indices]) + lower = 0.0 + X1, Y1 = encoder.transform() assert Y1.tolist() == [[1.0], [5.0]] - assert ((X1 <= 1.0) & (X1 >= 0.0)).all() + assert ((X1 <= upper) & (X1 >= lower)).all() # Log encoder - encoder = RunHistoryLogEncoder(scenario=scenario, considered_states=[StatusType.SUCCESS]) + encoder = RunHistoryLogEncoder( + scenario=scenario, considered_states=[StatusType.SUCCESS] + ) encoder.runhistory = runhistory X, Y = encoder.transform() assert Y.tolist() != Y1.tolist() - assert ((X <= 1.0) & (X >= 0.0)).all() + assert ((X <= upper) & (X >= lower)).all() - encoder = RunHistoryLogScaledEncoder(scenario=scenario, considered_states=[StatusType.SUCCESS]) + encoder = RunHistoryLogScaledEncoder( + scenario=scenario, considered_states=[StatusType.SUCCESS] + ) encoder.runhistory = runhistory X, Y = encoder.transform() assert Y.tolist() != Y1.tolist() - assert ((X <= 1.0) & (X >= 0.0)).all() + assert ((X <= upper) & (X >= lower)).all() - encoder = RunHistoryScaledEncoder(scenario=scenario, considered_states=[StatusType.SUCCESS]) + encoder = RunHistoryScaledEncoder( + scenario=scenario, considered_states=[StatusType.SUCCESS] + ) encoder.runhistory = runhistory X, Y = encoder.transform() assert Y.tolist() != Y1.tolist() - assert ((X <= 1.0) & (X >= 0.0)).all() + assert ((X <= upper) & (X >= lower)).all() - encoder = RunHistoryInverseScaledEncoder(scenario=scenario, considered_states=[StatusType.SUCCESS]) + encoder = RunHistoryInverseScaledEncoder( + scenario=scenario, considered_states=[StatusType.SUCCESS] + ) encoder.runhistory = runhistory X, Y = encoder.transform() assert Y.tolist() != Y1.tolist() - assert ((X <= 1.0) & (X >= 0.0)).all() + assert ((X <= upper) & (X >= lower)).all() - encoder = RunHistorySqrtScaledEncoder(scenario=scenario, considered_states=[StatusType.SUCCESS]) + encoder = RunHistorySqrtScaledEncoder( + scenario=scenario, considered_states=[StatusType.SUCCESS] + ) encoder.runhistory = runhistory X, Y = encoder.transform() assert Y.tolist() != Y1.tolist() - assert ((X <= 1.0) & (X >= 0.0)).all() + assert ((X <= upper) & (X >= lower)).all() - encoder = RunHistoryEIPSEncoder(scenario=scenario, considered_states=[StatusType.SUCCESS]) + encoder = RunHistoryEIPSEncoder( + scenario=scenario, considered_states=[StatusType.SUCCESS] + ) encoder.runhistory = runhistory X, Y = encoder.transform() assert Y.tolist() != Y1.tolist() - assert ((X <= 1.0) & (X >= 0.0)).all() + assert ((X <= upper) & (X >= lower)).all() -def test_transform_conditionals(runhistory, make_scenario, configspace_large, configs): - configs = configspace_large.sample_configuration(20) +def test_transform_conditionals(runhistory, make_scenario, configspace_large): scenario = make_scenario(configspace_large) + config_1 = Configuration( + configspace_large, + values={ + "activation": "tanh", + "n_layer": 5, + "n_neurons": 27, + "solver": "lbfgs", + }, + ) + config_2 = Configuration( + configspace_large, + values={ + "activation": "tanh", + "batch_size": 47, + "learning_rate": "adaptive", + "learning_rate_init": 0.6673206111956781, + "n_layer": 3, + "n_neurons": 88, + "solver": "sgd", + }, + ) runhistory.add( - config=configs[0], + config=config_1, cost=1, time=1, status=StatusType.SUCCESS, ) runhistory.add( - config=configs[2], + config=config_2, cost=5, time=4, status=StatusType.SUCCESS, ) - encoder = RunHistoryEncoder(scenario=scenario, considered_states=[StatusType.SUCCESS]) + encoder = RunHistoryEncoder( + scenario=scenario, considered_states=[StatusType.SUCCESS] + ) encoder.runhistory = runhistory X, Y = encoder.transform() @@ -124,7 +184,9 @@ def test_multi_objective(runhistory, make_scenario, configspace_small, configs): # Multi objective algorithm must be set with pytest.raises(AssertionError): - encoder = RunHistoryEncoder(scenario=scenario, considered_states=[StatusType.SUCCESS]) + encoder = RunHistoryEncoder( + scenario=scenario, considered_states=[StatusType.SUCCESS] + ) encoder.runhistory = runhistory _, Y = encoder.transform() @@ -180,7 +242,9 @@ def test_ignore(runhistory, make_scenario, configspace_small, configs): ) # Normal encoder - encoder = RunHistoryEncoder(scenario=scenario, considered_states=[StatusType.SUCCESS]) + encoder = RunHistoryEncoder( + scenario=scenario, considered_states=[StatusType.SUCCESS] + ) encoder.runhistory = runhistory X1, Y1 = encoder.transform() @@ -219,10 +283,14 @@ def test_budgets(runhistory, make_scenario, configspace_small, configs): budget=2, ) - runhistory.add(config=configs[1], cost=5, time=4, status=StatusType.SUCCESS, budget=2) + runhistory.add( + config=configs[1], cost=5, time=4, status=StatusType.SUCCESS, budget=2 + ) # Normal encoder - encoder = RunHistoryEncoder(scenario=scenario, considered_states=[StatusType.SUCCESS]) + encoder = RunHistoryEncoder( + scenario=scenario, considered_states=[StatusType.SUCCESS] + ) encoder.runhistory = runhistory X, Y = encoder.transform(budget_subset=[2]) assert Y.tolist() == [[99999999]] @@ -251,10 +319,14 @@ def test_budgets(runhistory, make_scenario, configspace_small, configs): budget=2, ) - runhistory.add(config=configs[1], cost=5, time=4, status=StatusType.SUCCESS, budget=2) + runhistory.add( + config=configs[1], cost=5, time=4, status=StatusType.SUCCESS, budget=2 + ) # Normal encoder - encoder = RunHistoryEncoder(scenario=scenario, considered_states=[StatusType.SUCCESS]) + encoder = RunHistoryEncoder( + scenario=scenario, considered_states=[StatusType.SUCCESS] + ) encoder.runhistory = runhistory X, Y = encoder.transform(budget_subset=[2]) assert Y.tolist() == [[99999999]] @@ -266,12 +338,20 @@ def test_budgets(runhistory, make_scenario, configspace_small, configs): def test_lower_budget_states(runhistory, make_scenario, configspace_small, configs): """Tests lower budgets based on budget subset and considered states.""" scenario = make_scenario(configspace_small) - encoder = RunHistoryEncoder(scenario=scenario, considered_states=[StatusType.SUCCESS]) + encoder = RunHistoryEncoder( + scenario=scenario, considered_states=[StatusType.SUCCESS] + ) encoder.runhistory = runhistory - runhistory.add(config=configs[0], cost=1, time=1, status=StatusType.SUCCESS, budget=3) - runhistory.add(config=configs[0], cost=2, time=2, status=StatusType.SUCCESS, budget=4) - runhistory.add(config=configs[0], cost=3, time=4, status=StatusType.TIMEOUT, budget=5) + runhistory.add( + config=configs[0], cost=1, time=1, status=StatusType.SUCCESS, budget=3 + ) + runhistory.add( + config=configs[0], cost=2, time=2, status=StatusType.SUCCESS, budget=4 + ) + runhistory.add( + config=configs[0], cost=3, time=4, status=StatusType.TIMEOUT, budget=5 + ) # We request a higher budget but can't find it, so we expect an empty list X, Y = encoder.transform(budget_subset=[500])