preparation wrapper script for IMPROVE

[ymahlich]

A wrapper script that makes use of the coderdata API to prepare the datasets for ingestion of the IMPROVE model testing pipeline needs to be created.
This will reside in the scripts folder of the repo.

[sgosline]

From Natasha:

• The current benchmark dataset can be downloaded with wget --cut-dirs=8 -P ./ -nH -np -m --reject "index.html*,index.*" https://web.cels.anl.gov/projects/IMPROVE_FTP/candle/public/improve/benchmarks/single_drug_drp/benchmark-data-pilot1/csa_data/
• Documentation of the dataset is here: https://jdacs4c-improve.github.io/docs/content/app_drp_benchmark.html
• In order to use datasets, we specify the path to the data folder and the datasets and splits to use in the model config file (or by command line at run). So if we have a wrapper script that formats the data in the same way as the benchmark data we should just need to change the path to the data and the split names. If we can keep the file names for the x_data the same that would be ideal. Here's an example from a model config file:

x_data_canc_files = [["cancer_gene_expression.tsv", ["Gene_Symbol"]]]
x_data_drug_files = [["drug_mordred.tsv"]]
y_data_files = [["response.tsv"]]
train_split_file = CCLE_split_0_train.txt
val_split_file = CCLE_split_0_val.txt
test_split_file = CCLE_split_0_test.txt

[ymahlich]

As per conversation with Natasha:

• the ids in the files in splits/ reference the row number in data_y/response.tsv. This is truly a direct row number reference i.e. this "id" does not exist anywhere else
• mapping to omics data is done internally by their preparation method by mapping from splitfile:row_number -> response.tsv:improve_sample_id/improve_chem_id -> individual entries in the corresponding omics datafiles extracted by looking up improve_sample_id & improve_chem_id

[sgosline]

Is this complete?

[ymahlich]

I made some more fixes for Natasha the other day, but yes this is complete once the changes are merged into the main branch.
I'll create PR some time today.