[fix] Add first draft of the PR for issue#349 #365

nabenabe0928 · 2021-12-22T02:52:07Z

Types of changes

Breaking change (fix or feature that would cause existing functionality to not work as expected)
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)

Note that a Pull Request should only contain one of refactoring, new features or documentation changes.
Please separate these changes and send us individual PRs for each.
For more information on how to create a good pull request, please refer to The anatomy of a perfect pull request.

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.

Have you checked to ensure there aren't other open Pull Requests for the same update/change?
Have you added an explanation of what your changes do and why you'd like us to include them?
Have you written new tests for your core changes, as applicable?
Have you successfully ran tests with your changes locally?

Description

See issue#349.
Note that although this PR has so many changes, most changes are gathered in three files:

tae.py
evaluator.py
abstract_evaluator.py

The changes in other files are mostly for tests or deletions because I integrated those features into the aforementioned three files.

…oml#334) * [feat] Support statistics print by adding results manager object * [refactor] Make SearchResults extract run_history at __init__ Since the search results should not be kept in eternally, I made this class to take run_history in __init__ so that we can implicitly call extraction inside. From this change, the call of extraction from outside is not recommended. However, you can still call it from outside and to prevent mixup of the environment, self.clear() will be called. * [fix] Separate those changes into PR#336 * [fix] Fix so that test_loss includes all the metrics * [enhance] Strengthen the test for sprint and SearchResults * [fix] Fix an issue in documentation * [enhance] Increase the coverage * [refactor] Separate the test for results_manager to organize the structure * [test] Add the test for get_incumbent_Result * [test] Remove the previous test_get_incumbent and see the coverage * [fix] [test] Fix reversion of metric and strengthen the test cases * [fix] Fix flake8 issues and increase coverage * [fix] Address Ravin's comments * [enhance] Increase the coverage * [fix] Fix a flake8 issu

* [doc] Add workflow of the AutoPytorch * [doc] Address Ravin's comment

* [feat] Add an object that realizes the perf over time viz * [fix] Modify TODOs and add comments to avoid complications * [refactor] [feat] Format visualizer API and integrate this feature into BaseTask * [refactor] Separate a shared raise error process as a function * [refactor] Gather params in Dataclass to look smarter * [refactor] Merge extraction from history to the result manager Since this feature was added in a previous PR, we now rely on this feature to extract the history. To handle the order by the start time issue, I added the sort by endtime feature. * [feat] Merge the viz in the latest version * [fix] Fix nan --> worst val so that we can always handle by number * [fix] Fix mypy issues * [test] Add test for get_start_time * [test] Add test for order by end time * [test] Add tests for ensemble results * [test] Add tests for merging ensemble results and run history * [test] Add the tests in the case of ensemble_results is None * [fix] Alternate datetime to timestamp in tests to pass universally Since the mapping of timestamp to datetime variates on machine, the tests failed in the previous version. In this version, we changed the datetime in the tests to the fixed timestamp so that the tests will pass universally. * [fix] Fix status_msg --> status_type because it does not need to be str * [fix] Change the name for the homogeniety * [fix] Fix based on the file name change * [test] Add tests for set_plot_args * [test] Add tests for plot_perf_over_time in BaseTask * [refactor] Replace redundant lines by pytest parametrization * [test] Add tests for _get_perf_and_time * [fix] Remove viz attribute based on Ravin's comment * [fix] Fix doc-string based on Ravin's comments * [refactor] Hide color label settings extraction in dataclass Since this process makes the method in BaseTask redundant and this was pointed out by Ravin, I made this process a method of dataclass so that we can easily fetch this information. Note that since the color and label information always depend on the optimization results, we always need to pass metric results to ensure we only get related keys. * [test] Add tests for color label dicts extraction * [test] Add tests for checking if plt.show is called or not * [refactor] Address Ravin's comments and add TODO for the refactoring * [refactor] Change KeyError in EnsembleResults to empty Since it is not convenient to not be able to instantiate EnsembleResults in the case when we do not have any histories, I changed the functionality so that we can still instantiate even when the results are empty. In this case, we have empty arrays and it also matches the developers intuition. * [refactor] Prohibit external updates to make objects more robust * [fix] Remove a member variable _opt_scores since it is confusing Since opt_scores are taken from cost in run_history and metric_dict takes from additional_info, it was confusing for me where I should refer to what. By removing this, we can always refer to additional_info when fetching information and metrics are always available as a raw value. Although I changed a lot, the functionality did not change and it is easier to add any other functionalities now. * [example] Add an example how to plot performance over time * [fix] Fix unexpected train loss when using cross validation * [fix] Remove __main__ from example based on the Ravin's comment * [fix] Move results_xxx to utils from API * [enhance] Change example for the plot over time to save fig Since the plt.show() does not work on some environments, I changed the example so that everyone can run at least this example.

* cleanup of simple_imputer * Fixed doc and typo * Fixed docs * Made changes, added test * Fixed init statement * Fixed docs * Flake'd

…#351) * [feat] Add the option to save a figure in plot setting params Since non-GUI based environments would like to avoid the usage of show method in the matplotlib, I added the option to savefig and thus users can complete the operations inside AutoPytorch. * [doc] Add a comment for non-GUI based computer in plot_perf_over_time method * [test] Add a test to check the priority of show and savefig Since plt.savefig and plt.show do not work at the same time due to the matplotlib design, we need to check whether show will not be called when a figname is specified. We can actually raise an error, but plot will be basically called in the end of an optimization, so I wanted to avoid raising an error and just sticked to a check by tests.

* update workflow files * Remove double quotes * Exclude python 3.10 * Fix mypy compliance check * Added PEP 561 compliance * Add py.typed to MANIFEST for dist * Update .github/workflows/dist.yml Co-authored-by: Ravin Kohli <[email protected]> Co-authored-by: Ravin Kohli <[email protected]>

* Add fit pipeline with tests * Add documentation for get dataset * update documentation * fix tests * remove permutation importance from visualisation example * change disable_file_output * add * fix flake * fix test and examples * change type of disable_file_output * Address comments from eddie * fix docstring in api * fix tests for base api * fix tests for base api * fix tests after rebase * reduce dataset size in example * remove optional from doc string * Handle unsuccessful fitting of pipeline better * fix flake in tests * change to default configuration for documentation * add warning for no ensemble created when y_optimization in disable_file_output * reduce budget for single configuration * address comments from eddie * address comments from shuhei * Add autoPyTorchEnum * fix flake in tests * address comments from shuhei * Apply suggestions from code review Co-authored-by: nabenabe0928 <[email protected]> * fix flake * use **dataset_kwargs * fix flake * change to enforce keyword args Co-authored-by: nabenabe0928 <[email protected]>

* Add workflow for publishing docker image to github packages and dockerhub * add docker installation to docs * add workflow dispatch

codecov · 2021-12-22T03:16:41Z

Codecov Report

Merging #365 (180ff33) into development (a679b09) will increase coverage by 1.85%.
The diff coverage is 99.33%.

@@               Coverage Diff               @@
##           development     #365      +/-   ##
===============================================
+ Coverage        83.44%   85.30%   +1.85%     
===============================================
  Files              163      163              
  Lines             9634     9567      -67     
  Branches          1689     1665      -24     
===============================================
+ Hits              8039     8161     +122     
+ Misses            1114      953     -161     
+ Partials           481      453      -28

Impacted Files	Coverage Δ
autoPyTorch/api/tabular_classification.py	`90.90% <ø> (ø)`
autoPyTorch/api/tabular_regression.py	`100.00% <ø> (ø)`
autoPyTorch/pipeline/base_pipeline.py	`71.13% <ø> (-0.95%)`	⬇️
autoPyTorch/datasets/resampling_strategy.py	`91.48% <80.00%> (-0.65%)`	⬇️
autoPyTorch/api/base_task.py	`84.06% <90.90%> (-0.34%)`	⬇️
autoPyTorch/evaluation/evaluator.py	`99.01% <99.01%> (ø)`
autoPyTorch/evaluation/abstract_evaluator.py	`99.14% <99.49%> (+24.49%)`	⬆️
...utoPyTorch/evaluation/pipeline_class_collection.py	`100.00% <100.00%> (ø)`
autoPyTorch/evaluation/tae.py	`95.69% <100.00%> (+25.05%)`	⬆️
autoPyTorch/evaluation/utils.py	`85.55% <100.00%> (+11.94%)`	⬆️
... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a679b09...180ff33. Read the comment docs.

ravinkohli · 2022-01-12T14:05:48Z

autoPyTorch/evaluation/tae.py

+ return cost, status, info, additional_run_info
+
+
+class TargetAlgorithmQuery(AbstractTAFunc):


I think ExecuteTAFuncWithQueue was more appropriate as a name. Also, it keeps AutoPyTorch in line with SMAC and auto-sklearn. Let's keep it that way.

Since this is not urgent, I will just reply to this message and another for now.

Although I agree that TargetAlgorithmQuery is not compatible with auto-sklearn, this name is still compatible with SMAC and the name itself is more appropriate because of the following problems in the previous name.

Apology
I checked official terminology and it seems Entry is the official name but not Query, so TargetAlgorithmQueueEntry or TAFunc4QueueEntry is more precise.
End of Apology

The problems of ExecuteTaFuncWithQueue are that:

the name starts from a verb, which is confusing for a class name except the class is callable,

the name itself is not correct because we run TA func (not execute as the method name is run()) when we query the entry from Queue, but we never execute TA func with Queue,

instances of this class are stored in Queue as a query, so it will not be executed when it is instantiated

Somehow it says TaFunc, but not TAFunc

It is impossible to merge BaseTask with the one in ASK, so I did not feel the compatibility with ASK in TAE is necessary, but do you think we need it?

ravinkohli · 2022-01-12T14:09:47Z

autoPyTorch/evaluation/tae.py

@@ -110,90 +255,66 @@ def __init__(
 stats: Optional[Stats] = None,
 run_obj: str = 'quality',
 par_factor: int = 1,
- output_y_hat_optimization: bool = True,
+ save_y_opt: bool = True,


actually, I think save_y_ensemble_optimization or even output_y_ensemble_optimization as this flag also acts as a way to disable ensemble construction. As if it is False we cant build an ensemble. We can also add this to the description of the variable. Actually, it has got nothing to do with saving y_opt, these true_targets_ensemble are only used for ensemble construction

ravinkohli · 2022-01-12T14:12:41Z

autoPyTorch/evaluation/tae.py

+ backend=backend,
+ seed=seed,
+ metric=metric,
+ save_y_opt=save_y_opt,


same here. please change the name.

ravinkohli · 2022-01-12T14:27:23Z

autoPyTorch/evaluation/tae.py

+ search_space_updates = self.fixed_pipeline_params.search_space_updates
+ self.logger.debug(f"Search space updates for {num_run}: {search_space_updates}")


I dont think we need to specially create a variable just to log out the search_space_updates. I added it because I wanted to see if they are being passed correctly when I wrote this functionality.

Especially when they will be printed for each run, when in fact they are the same for all num_runs.

Suggested change

search_space_updates = self.fixed_pipeline_params.search_space_updates

self.logger.debug(f"Search space updates for {num_run}: {search_space_updates}")

ravinkohli · 2022-01-12T14:40:15Z

autoPyTorch/evaluation/tae.py

 try:
- obj = pynisher.enforce_limits(**pynisher_arguments)(self.ta)
- obj(**obj_kwargs)
+ obj(queue=queue, evaluator_params=params, fixed_pipeline_params=self.fixed_pipeline_params)


Can we name this pynisher_function_wrapper_obj? I think it will make it easier to distinguish between the exit_status of pynisher_function_wrapper_obj and the Status coming from fitting the pipeline. As we have now encapsulated the code to process the results, I think having more meaningful names here will help us and others to understand whats's going on.

ravinkohli · 2022-01-12T14:41:05Z

autoPyTorch/evaluation/tae.py

+
+
+def _process_exceptions(
+ obj: PynisherFunctionWrapperType,


changing obj to pynisher_function_wrapper_obj should also be done here.

ravinkohli · 2022-01-12T14:52:07Z

autoPyTorch/evaluation/tae.py

+ budget: float,
+ worst_possible_result: float
+) -> ProcessedResultsType:
+ if obj.exit_status is TAEAbortException:


I think this way of implementing the logic is a bit confusing. Could you rather maybe handle the exception first and then handle creating the additional_run_info based on that? Mainly I am struggling to find the use of is_anything_exception. Moreover, I also think that you are missing some info for example, info_for_empty in case of a MEMOUT previously also contained the memory limit as well. I'd suggest you to add both the memory_limit and func_eval_time which is available in run_info.cutoff.

ravinkohli · 2022-01-12T14:53:08Z

autoPyTorch/api/base_task.py

-from autoPyTorch.evaluation.tae import ExecuteTaFuncWithQueue, get_cost_of_crash
+from autoPyTorch.evaluation.abstract_evaluator import fit_pipeline
+from autoPyTorch.evaluation.pipeline_class_collection import get_default_pipeline_config
+from autoPyTorch.evaluation.tae import TargetAlgorithmQuery


Also change back to ExecuteTaFuncWithQueue.

ravinkohli · 2022-01-12T14:53:16Z

autoPyTorch/api/base_task.py

@@ -669,22 +670,23 @@ def _do_dummy_prediction(self) -> None:
 # already be generated here!
 stats = Stats(scenario_mock)
 stats.start_timing()
- ta = ExecuteTaFuncWithQueue(
+ taq = TargetAlgorithmQuery(


here as well

ravinkohli · 2022-01-12T14:54:12Z

autoPyTorch/api/base_task.py

 stats=stats,
 memory_limit=memory_limit,
 disable_file_output=self._disable_file_output,
 all_supported_metrics=self._all_supported_metrics
 )

- status, _, _, additional_info = ta.run(num_run, cutoff=self._time_for_task)
+ status, _, _, additional_info = taq.run(num_run, cutoff=self._time_for_task)


if you dont like ta you can maybe use tae which stands for target algorithm execution.

ravinkohli · 2022-01-12T14:54:55Z

autoPyTorch/api/base_task.py

 do not save the predictions for the optimization set,
 which would later on be used to build an ensemble. Note that SMAC
 optimizes a metric evaluated on the optimization set.
- + `pipeline`:
+ + `model`:


I prefer the name pipeline.

ravinkohli · 2022-01-12T14:55:12Z

autoPyTorch/api/base_task.py

 do not save any individual pipeline files
- + `pipelines`:
+ + `cv_model`:


you can use cv_pipeline instead.

ravinkohli · 2022-01-12T14:55:41Z

autoPyTorch/api/base_task.py

@@ -1060,7 +1062,7 @@ def _search(

 # Here the budget is set to max because the SMAC intensifier can be:
 # Hyperband: in this case the budget is determined on the fly and overwritten
- # by the ExecuteTaFuncWithQueue
+ # by the TargetAlgorithmQuery


change here as well.

ravinkohli · 2022-01-12T14:57:48Z

autoPyTorch/api/base_task.py

@@ -1344,7 +1346,7 @@ def refit(
 dataset_properties=dataset_properties,
 dataset=dataset,
 split_id=split_id)
- fit_and_suppress_warnings(self._logger, model, X, y=None)
+ fit_pipeline(self._logger, model, X, y=None)


I think the previous name emphasised the fact that we are suppressing warnings, otherwise, we could have just used model.fit(X, y). So could you change this name back to what it was? Or I dont mind fit_pipeline_supress_warnings

This is another question that is mentioned in the other comment.

This is maybe a stupid question, but why do we need to emphasize suppress warnings?
I understand it if we squeeze errors during the fit, but warnings will not be critical for running a script and we write them in logging anyways.
So it is more like fit_pipeline_with_logging_warnings or superficially, this method looks identically to just fit_pipeline, doesn't it?

ravinkohli · 2022-01-12T15:01:18Z

autoPyTorch/evaluation/abstract_evaluator.py


+class FixedPipelineParams(NamedTuple):


What do you mean by "fixed" pipeline params?

ravinkohli · 2022-01-12T15:15:16Z

autoPyTorch/evaluation/abstract_evaluator.py

 access the train and test datasets
- queue (Queue):
- Each worker available will instantiate an evaluator, and after completion,
- it will append the result to a multiprocessing queue
 metric (autoPyTorchMetric):


Suggested change

metric (autoPyTorchMetric):

optimize_metric (autoPyTorchMetric):

* check if N==0, and handle this case * change position of comment * Address comments from shuhei

* add test evaluator * add no resampling and other changes for test evaluator * finalise changes for test_evaluator, TODO: tests * add tests for new functionality * fix flake and mypy * add documentation for the evaluator * add NoResampling to fit_pipeline * raise error when trying to construct ensemble with noresampling * fix tests * reduce fit_pipeline accuracy check * Apply suggestions from code review Co-authored-by: nabenabe0928 <[email protected]> * address comments from shuhei * fix bug in base data loader * fix bug in data loader for val set * fix bugs introduced in suggestions * fix flake * fix bug in test preprocessing * fix bug in test data loader * merge tests for evaluators and change listcomp in get_best_epoch * rename resampling strategies * add test for get dataset Co-authored-by: nabenabe0928 <[email protected]>

* [fix] Fix the no-training-issue when using simple intensifier * [test] Add a test for the modification * [fix] Modify the default budget so that the budget is compatible Since the previous version does not consider the provided budget_type when determining the default budget, I modified this part so that the default budget does not mix up the default budget for epochs and runtime. Note that since the default pipeline config defines epochs as the default budget, I also followed this rule when taking the default value. * [fix] Fix a mypy error * [fix] Change the total runtime for single config in the example Since the training sometimes does not finish in time, I increased the total runtime for the training so that we can accomodate the training in the given amount of time. * [fix] [refactor] Fix the SMAC requirement and refactor some conditions

…oml#371)

* add variance thresholding * fix flake and mypy * Apply suggestions from code review Co-authored-by: nabenabe0928 <[email protected]> Co-authored-by: nabenabe0928 <[email protected]>

* Add new scalers * fix flake and mypy * Apply suggestions from code review Co-authored-by: nabenabe0928 <[email protected]> * add robust scaler * fix documentation * remove power transformer from feature preprocessing * fix tests * check for default in include and exclude * Apply suggestions from code review Co-authored-by: nabenabe0928 <[email protected]> Co-authored-by: nabenabe0928 <[email protected]>

* remove categorical strategy from simple imputer * fix tests * address comments from eddie * fix flake and mypy error * fix test cases for imputation

* [fix] Add check dataset in transform as well for test dataset, which does not require fit * [test] Migrate tests from the francisco's PR without modifications * [fix] Modify so that tests pass * [test] Increase the coverage

* Fix: keyword arguments to submit * Fix: Missing param for implementing AbstractTA * Fix: Typing of multi_objectives * Add: mutli_objectives to each ExecuteTaFucnWithQueue

* remove datamanager instances from evaluation and smbo * fix flake * Apply suggestions from code review Co-authored-by: nabenabe0928 <[email protected]> * fix flake Co-authored-by: nabenabe0928 <[email protected]>

* [fix] Fix the task inference issue mentioned in automl#352 Since sklearn task inference regards targets with integers as a classification task, I modified target_validator so that we always cast targets for regression to float. This workaround is mentioned in the reference below: scikit-learn/scikit-learn#8952 * [fix] [test] Add a small number to label for regression and add tests Since target labels are required to be float and sklearn requires numbers after a decimal point, I added a workaround to add the almost possible minimum fraction to array so that we can avoid a mis-inference of task type from sklearn. Plus, I added tests to check if we get the expected results for extreme cases. * [fix] [test] Adapt the modification of targets to scipy.sparse.xxx_matrix * [fix] Address Ravin's comments and loosen the small number choice

* Initial implementation without tests * add tests and make necessary changes * improve documentation * fix tests * Apply suggestions from code review Co-authored-by: nabenabe0928 <[email protected]> * undo change in as it causes tests to fail * change name from InputValidator to input_validator * extract statements to methods * refactor code * check if mapping is the same as expected * update precision reduction for dataframes and tests * fix flake Co-authored-by: nabenabe0928 <[email protected]>

* [refactor] Refactor __init__ of abstract evaluator * [refactor] Collect shared variables in NamedTuples * [fix] Copy the budget passed to the evaluator params * [refactor] Add cross validation result manager for separate management * [refactor] Separate pipeline classes from abstract evaluator * [refactor] Increase the safety level of pipeline config * [test] Add tests for the changes * [test] Modify queue.empty in a safer way [fix] Find the error in test_tabular_xxx Since pipeline is updated after the evaluations and the previous code updated self.pipeline in the predict method, dummy class only needs to override this method. However, the new code does it separately, so I override get_pipeline method so that we can reproduce the same results. [fix] Fix the shape issue in regression and add bug comment in a test [fix] Fix the ground truth of test_cv Since we changed the weighting strategy for the cross validation in the validation phase so that we weight performance from each model proportionally to the size of each VALIDATION split. I needed to change the answer. Note that the previous was weighting the performance proportionally to the TRAINING splits for both training and validation phases. [fix] Change qsize --> Empty since qsize might not be reliable [refactor] Add cost for crash in autoPyTorchMetrics [fix] Fix the issue when taking num_classes from regression task [fix] Deactivate the save of cv model in the case of holdout

…ter)

[test] Add the tests for the instantiation of abstract evaluator 1 -- 3 [test] Add the tests for util 1 -- 2 [test] Add the tests for train_evaluator 1 -- 2 [refactor] [test] Clean up the pipeline classes and add tests for it 1 -- 2 [test] Add the tests for tae 1 -- 4 [fix] Fix an error due to the change in extract learning curve [experimental] Increase the coverage [test] Add tests for pipeline repr Since the modifications in tests removed the coverage on pipeline repr, I added tests to increase those parts. Basically, the decrease in the coverage happened due to the usage of dummy pipelines.

…in_evaluator Since test_evaluator can be merged, I merged it. * [rebase] Rebase and merge the changes in non-test files without issues * [refactor] Merge test- and train-evaluator * [fix] Fix the import error due to the change xxx_evaluator --> evaluator * [test] Fix errors in tests * [fix] Fix the handling of test pred in no resampling * [refactor] Move save_y_opt=False for no resampling deepter for simplicity * [test] Increase the budget size for no resample tests * [test] [fix] Rebase, modify tests, and increase the coverage

nabenabe0928 and others added 10 commits December 21, 2021 16:16

[doc] Add the workflow of the Auto-Pytorch (automl#285)

4d28006

* [doc] Add workflow of the AutoPytorch * [doc] Address Ravin's comment

Update README.md with link for master branch

54ee98e

Cleanup of simple_imputer (automl#346)

0ae9cbf

* cleanup of simple_imputer * Fixed doc and typo * Fixed docs * Made changes, added test * Fixed init statement * Fixed docs * Flake'd

[ADD] Docker publish workflow (automl#357)

e3aeb55

* Add workflow for publishing docker image to github packages and dockerhub * add docker installation to docs * add workflow dispatch

fix error after merge

f612f46

nabenabe0928 force-pushed the 349-fix-additional-info-for-cv branch from 4b322b5 to ffb4dae Compare January 10, 2022 06:55

ravinkohli reviewed Jan 12, 2022

View reviewed changes

ravinkohli and others added 2 commits January 24, 2022 13:14

Fix 361 (automl#367)

c0fb82e

* check if N==0, and handle this case * change position of comment * Address comments from shuhei

nabenabe0928 force-pushed the 349-fix-additional-info-for-cv branch 4 times, most recently from c41c87f to 4d4e306 Compare January 30, 2022 02:30

nabenabe0928 and others added 6 commits January 31, 2022 23:23

[fix] Change int to np.int32 for the ndarray dtype specification (aut…

bd4fabf

…oml#371)

[ADD] variance thresholding (automl#373)

466bc18

* add variance thresholding * fix flake and mypy * Apply suggestions from code review Co-authored-by: nabenabe0928 <[email protected]> Co-authored-by: nabenabe0928 <[email protected]>

[FIX] Remove redundant categorical imputation (automl#375)

ba9c86a

* remove categorical strategy from simple imputer * fix tests * address comments from eddie * fix flake and mypy error * fix test cases for imputation

[feat] Add coalescer (automl#376)

bf264d6

* [fix] Add check dataset in transform as well for test dataset, which does not require fit * [test] Migrate tests from the francisco's PR without modifications * [fix] Modify so that tests pass * [test] Increase the coverage

Fix: keyword arguments to submit (automl#384)

b5c1757

* Fix: keyword arguments to submit * Fix: Missing param for implementing AbstractTA * Fix: Typing of multi_objectives * Add: mutli_objectives to each ExecuteTaFucnWithQueue

nabenabe0928 force-pushed the 349-fix-additional-info-for-cv branch 2 times, most recently from 6b577f6 to c13e13a Compare February 21, 2022 20:47

ravinkohli and others added 2 commits February 23, 2022 18:03

[FIX] Datamanager in memory (automl#382)

4a0c773

* remove datamanager instances from evaluation and smbo * fix flake * Apply suggestions from code review Co-authored-by: nabenabe0928 <[email protected]> * fix flake Co-authored-by: nabenabe0928 <[email protected]>

nabenabe0928 force-pushed the 349-fix-additional-info-for-cv branch 2 times, most recently from e48fd14 to 8d9c132 Compare February 23, 2022 22:33

nabenabe0928 and others added 6 commits February 25, 2022 23:22

[fix] Update the SMAC version (automl#388)

dafd480

[temporal] Change tests to pass temporally (might need to get back la…

98bd007

…ter)

nabenabe0928 force-pushed the 349-fix-additional-info-for-cv branch from 8d9c132 to 180ff33 Compare February 25, 2022 15:08

ravinkohli force-pushed the development branch from 7f641b5 to 0e574af Compare July 18, 2022 18:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix] Add first draft of the PR for issue#349 #365

[fix] Add first draft of the PR for issue#349 #365

nabenabe0928 commented Dec 22, 2021 •

edited

codecov bot commented Dec 22, 2021 •

edited

ravinkohli Jan 12, 2022

nabenabe0928 Jan 28, 2022 •

edited

ravinkohli Jan 12, 2022

ravinkohli Jan 12, 2022

ravinkohli Jan 12, 2022 •

edited

ravinkohli Jan 12, 2022

ravinkohli Jan 12, 2022

ravinkohli Jan 12, 2022

ravinkohli Jan 12, 2022

ravinkohli Jan 12, 2022

ravinkohli Jan 12, 2022

ravinkohli Jan 12, 2022

ravinkohli Jan 12, 2022

ravinkohli Jan 12, 2022

ravinkohli Jan 12, 2022

nabenabe0928 Jan 28, 2022 •

edited

ravinkohli Jan 12, 2022

ravinkohli Jan 12, 2022

		return cost, status, info, additional_run_info


		class TargetAlgorithmQuery(AbstractTAFunc):

		search_space_updates = self.fixed_pipeline_params.search_space_updates
		self.logger.debug(f"Search space updates for {num_run}: {search_space_updates}")

	metric (autoPyTorchMetric):
	optimize_metric (autoPyTorchMetric):

[fix] Add first draft of the PR for issue#349 #365

Are you sure you want to change the base?

[fix] Add first draft of the PR for issue#349 #365

Conversation

nabenabe0928 commented Dec 22, 2021 • edited

Types of changes

Checklist:

Description

codecov bot commented Dec 22, 2021 • edited

Codecov Report

Choose a reason for hiding this comment

nabenabe0928 Jan 28, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ravinkohli Jan 12, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nabenabe0928 Jan 28, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nabenabe0928 commented Dec 22, 2021 •

edited

codecov bot commented Dec 22, 2021 •

edited

nabenabe0928 Jan 28, 2022 •

edited

ravinkohli Jan 12, 2022 •

edited

nabenabe0928 Jan 28, 2022 •

edited