COPD-MICCAI 2018] Apply the Sub2Vec method on new data from the hard drive

[sumedhasingla]

• Compare results on Sharp vs standard images

• Apply the methods as it is on the Phase -1

• Compare results on INSP only vs INSP + EXP images

• Repeat experiments while incorporating phase-2 images

• Perform 5 fold cross validation of the method as it is on Phase 1

• Repeat exploratory analysis on the cross validation data.

[kayhan-batmanghelich]

@sumedhasingla would you please update this issue. Thanks.

[kayhan-batmanghelich]

@sumedhasingla in the previous run, did you include the covariates to the network or it had only imaging?

The results from #57 show that the new features do help in predicting exacerbation. We might need to add exacerbation to the learning too (FEV1, FEVC, exacerbation). We should think about what loss function is appropriate for that task. Talk to @jrahimik and @mgong2 about it. Either one-class loss or loss of the positive-label learning can help there.

[sumedhasingla]

@Kayhan, In previous run, we are only predicting FEV1 and FVC.

[sumedhasingla]

There were some nan- issues in the earlier run of the cross validation experiments. The Nan issue is now fixed. We are waiting for our PSC reservation to execute cross validation.
Currently fold-1 for lambda = 10 is running on DBMI-GPU.
The reservation will start on 5:00 p.m on Thursday (09/06/2018)
If everything runs fine, we should have our results by next week.

[sumedhasingla]

In current implementation, one epoch is taking 5 hours to complete. We need to run about 25 epochs, so it will take 5 days to complete 1 experiment.
I am exploring ways to exploit multi GPU programming with Python.
@Kayhan I am sccessful in installing horovod library, but am working on getting it run with our experiment.
Also, I am trying the other method suggested by Payman.

[kayhan-batmanghelich]

@sumedhasingla Thanks for the update. If you get the multi-GPU working (in any way), this will be a major breakthrough for many of our project. Thanks for pushing for that.

Regarding your comment on FEV1, FEVC: yes I know. My question is that did network concatenated Age, gender, the pack of smoking with $\phi(x)$. I guess not. It might be helpful to provide those info so that the model focuses on the complementary info.

[sumedhasingla]

The cross-validation with lambda_1 = 10 is completed.
The results are here:
GitHub-Results
GitHub-Exploratory Analysis-PCA10
GitHub-Exploratory Analysis-PCA25