Question about TCA np.argsort()

[orvindemsy]

Hello, jindongwang, thanks for this amazing repo, I have question about the TCA implementation in python, about line 62-64

w, V = scipy.linalg.eig(a, b)
ind = np.argsort(w)
A = V[:, ind[:self.dim]]

So it takes the eigenvectors with smallest eigenvalues? Isn't it supposed to be the opposite? I also try to look at the w and found that the largest eigenvalues goes to inf, is this related to numerical rounding error in python?

I also try to switch the sort order to descending by ind = np.argsort(w)[::-1] but yeah, it doesn't work, it worsen the accuracy, so why does this work? thank you.

sorry if I am not allowed to ask here, but don't know where else I should ask

[jindongwang]

Good question. The code is correct. It is smallest eigenvectors that we need since we aim to minimize the objective function.
The function, as you know it, is solved by Rayleigh quotient:

• max --> largest eigenvectors.
• min --> smallest eigenvectors.

Thus, we use the smallest eigenvectors in our problem. You can refer to Page 11 of this slide: https://www.sjsu.edu/faculty/guangliang.chen/Math253S20/lec4RayleighQuotient.pdf

When do we need the largest eigenvectors? See this article: https://papers.nips.cc/paper/2001/file/d5c186983b52c4551ee00f72316c6eaa-Paper.pdf.

[orvindemsy]

Thanks!, if you don't mind I ask another question, let me know if I should open another thread instead.

[orvindemsy]

Thank you for getting back to me and providing the reference, will read that!

One more question if you don't mind, how does this work for new unseen data? Let's say Xt1 and Xt2 are two data coming from same experiment, Xs is source data

# ====== A. In the code =====
Xs_new, Xt1_new = tca.fit(Xs, Xt1)
clf = Classifier()                
clf.fit(Xs_new, Ys)       # fit source data to classifier
ypred = clf.eval(Xt1_new) # evaluate

# == B. What I am doing for Xt2 ==
Xs_new, Xt2_new = tca.fit(Xs, Xt2)
clf = Classifier()
clf.fit(Xs_new, Ys)
ypred = clf.eval(Xt2_new)

# == C. What I expect ==
Xt2_new = tca.fit(Xt2) # Using the projection matrix build from pairing Xs and Xt1
ypred = clf.eval (Xt2_new)

A

This is what it does in the code, is it train or test score?.

B

This is how I implement TCA for new data, so I generate new projection matrix, new classifier, each time new data comes, which I think is wrong, What I want to do is this, in the red box here (source: [TL book by Pan])(https://www.google.co.jp/books/edition/Transfer_Learning/dG_IDwAAQBAJ),
so I just want to project Xt2 onto latent space that I learn from A and use my classifier and projection matrix from I got from A too.

C

This is what I expect

Question Summary

1. About A the in the code, is it test score? or train score?
2. About B , do you think this correct? do I always have to build new projection matrix and classifier? Instead of mapping new data to latent space and use the pre-trained classifier?
3. About C, this is what I expected because it agrees with the statement on that book, but I got lost with what projection matrix and what source data should I use?

Example

Variables

Variable	Description	Value
Xs	source data	(ns, n_feat)
Xt1	target data 1	(nt1, n_feat
Xt2	target data 2	(nt2, n_feat)
ns	source data samples	100
nt1	target data 1	10
nt2	target data 2	5
n_feat	feature dimension of source and target	20
tca_dim	the dimension after PCA	3
K_A	kernel matrix source and target data 1	((ns + nt1), (ns +nt1)
K_C	kernel matrix source and target data 2	((ns + nt2), (ns +nt2)
W_proj_A	projection matrix from K_A	((ns + nt1), tca_dim)

K_A and K_C refers to approach A and C in the code above, now I want to use W_proj_A for Xt2, I try to pair it with Xs with ns sample size, but cannot cause the kernel matrix isn't same

Approach	K size	W_proj_A size
A	(110, 110)	(110, 3)
C	(105, 105)	(110, 3) (??)

The approach A maps K_A into latent space of 3 dimension, but as can be seen I cannot use W_proj_A on K_C due to different size. What should be the kernel matrix or projection matrix in C ?

[jindongwang]

I also thought about this several years ago. The answer is currently, TCA cannot deal with incremental data. Thus, B is correct.
Looking for incremental update or adaptation, you can refer to the deep version of TCA: https://github.com/jindongwang/transferlearning/tree/master/code/deep/DDC_DeepCoral

[orvindemsy]

I see, I found something which kinda do incremental update but it's for kpca. So they use the stack of [Xs + Xt1] as source to be paired with Xt2 when computing kernel matrix, following above example size we have..

K = (ker_type, X1 = Xt2, X2 = [Xs + Xt1])  # size of 5 x 110
Xt2_new = K @ W_proj                       # (5 x 110) @ (110 x 3) = (5 x 3)

What do you think about this? matrix size-wise it's working, but theoretically is it valid for TCA or only valid for KPCA?
I tried implementing this into your code, and it seems to work okay, judging from accuracy only.

[jindongwang]

It seems right. Treating the source1 and target1 as the new source domain, then use it to compute with target2 is working. But I doubt its accuracy. Could you please create a PR to submit your revised code? I'll check later.

[orvindemsy]

hello again jindongwang, I did it, btw this is my first PR to any github repo ever, my code may not be beautiful as I ignore any PEP8 format, feel free to point out my mistake, or any feedback is okay.

[jcalvaradop777]

Best regards

I am trying to make a comparison of the incremental versions of dimensionality reduction methods. Could someone tell me where I can find the code for Incremental Locally Linear Embedding, or Incremental Multidimensional Scaling or Incremental Laplacian EigenMaps ?

Thank you.