Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature add cosine proximity loss #30

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

WuZhuoran
Copy link
Contributor

This pull request closes #29 .

- What I did

  • Add Cosine Proximity Loss Function.

- How I did it

Refer class comments.

- How to verify it

  • Update test of loss function in tests.py
  • Test passed at my environment.

This pull request adds a new feature to Numpy-ml. Ask @ddbourgin to take a look.


y_true = l2_normalize(y, axis=-1)
y_pred = l2_normalize(y_pred, axis=-1)
return 1. - np.sum(y_true * y_pred, axis=-1)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I saw 2 different implementation. First one is

return -np.sum(y_true * y_pred, axis=-1)

or

return 1. - np.sum(y_true * y_pred, axis=-1)

Which one should we choose for our implementation? They are same in nature.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the first since it ranges between -1 and 1, similar to the cosine distance itself.

return 1. - np.sum(y_true * y_pred, axis=-1)

@staticmethod
def grad(y, y_pred, z, act_fn):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a sufficient way to do grad of cosine?

Copy link
Owner

@ddbourgin ddbourgin Jul 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doing this from my phone, so please check for errors:

If f(x, y) = (x @ y) / (norm(x) * norm(y)), then we have

df/dy = x / (norm(x) * norm(y)) - (f(x, y) * y) / (norm(y) ** 2)

where norm is just the 2-norm

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that since cosine loss == negative cosine distance, you should multiply df/dy by -1

vector_length_max = 100

for j in range(2, vector_length_max):
x = np.random.uniform(0., 1., [j, ])
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To generate random vector array, i set the bound from 0. to 1.

@ddbourgin
Copy link
Owner

@WuZhuoran - Just ping me when this is finished and I'll take a look.

@WuZhuoran
Copy link
Contributor Author

@WuZhuoran - Just ping me when this is finished and I'll take a look.

@ddbourgin Thank you. I think I need some help with the grad of cos loss and how to test grad function?

@ddbourgin
Copy link
Owner

General comment: It looks like right now the documentation is copied directly from scipy.spatial.distance. This needs to be rewritten before we merge.

@WuZhuoran
Copy link
Contributor Author

General comment: It looks like right now the documentation is copied directly from scipy.spatial.distance. This needs to be rewritten before we merge.

I will update documentation at next commits. Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature: Cosine Proximity Loss Function
2 participants