what do recommendation predict_one methods return for unknown items?

[victusfate]

I was doing some testing earlier and I passed a novel item_id to predict_one and got back a reasonable and consistent score for that item. I was wondering if this is intended?

print(self.model.predict_one({'user': user_id,'item': 'this_does_not_exist'}))

{"this_does_not_exist":0.3372315214069518}

I can also do this for unknown users. Normally when I pass predictions to unrecognized users I default to popular items, but in this case its an unknown user and an unknown item.

print(self.model.predict_one({'user': 'unknown_user','item': 'this_does_not_exist'}))

{"this_does_not_exist":-0.0014310086721113745}

I guess I expected 0 scores for novel items, or novel user/items.

[gbolmier]

Hi @victusfate!

It happens because of the weights initialization scheme. A desired behaviour in an online learning pipeline is to draw new weights for new features when they appear in order to continuously learn on new observations. This work internally by storing the model weights in defaultdicts so that they return a default value (can be zero or anything from a pre-defined distribution, cf. the weight_initializer param) when the key doesn't exist in the dict. So what is returned when the features are new is basically a dummy prediction based on the drawn weights.

If you need to check if an item has been seen or not by your model you can try:

feature_seen: bool = feature_name in my_model.<weights_attribute>

[victusfate]

Wonderful, so I can check for item existence and default to 0 or leave it as is, and get back dummy predictions (I'm worried that could conceal another error in my pipeline so I might default to 0).

Is there documentation on the weights attribute for models? I'm using a FM recommendation model, and double checking now.
https://riverml.xyz/latest/api/facto/FMRegressor/
so it looks like its just model.weights and I can check your above example with

  def predict(self,user_id,item_ids):
    scores = {}
    for item_id in item_ids:
      score = 0
      if item_id in self.model.weights:
        score = self.model.predict_one({'user': user_id,'item': item_id})
      scores[item_id] = score
    # sort desc by score
    scores = {k: v for k, v in sorted(scores.items(), key=lambda item: item[1],reverse=True)}  
    return scores

thanks again for the quick response!

update, not quite there yet

AttributeError: 'PredClipper' object has no attribute 'weights'

here's the full model declaration for reference

  # https://riverml.xyz/latest/examples/matrix-factorization-for-recommender-systems-part-2/
  #     yhat(x) = w0 + sumj=1->p(wjxj) + sumj=1->p(sumj'=j+1->p(<vj,vj'>xjxj'))
  #     <vj,vj'> = sumf=1->k(vj,f dot vj',f) (dot product of the latent factor vectors)
  def fm_model():
    fm_params = {
      'n_factors': 10,
      'weight_optimizer': optim.SGD(0.025),
      'latent_optimizer': optim.SGD(0.05),
      'sample_normalization': False,
      'l1_weight': 0.,
      'l2_weight': 0.,
      'l1_latent': 0.,
      'l2_latent': 0.,
      'intercept': 1, # mean of scoring
      'intercept_lr': .01,
      'weight_initializer': optim.initializers.Zeros(), # NOTE this is zeros
      'latent_initializer': optim.initializers.Normal(mu=0., sigma=0.1, seed=73),
    }

    regressor = compose.Select('user', 'item')
    regressor |= facto.FMRegressor(**fm_params)

    model = meta.PredClipper(
      regressor=regressor,
      y_min=constants.MIN_PLACE_SCORE,
      y_max=constants.MAX_PLACE_SCORE
    )
    return model

I got a chance to inspect the object and I can see the PredClipper object has a regressor member. In that member I can see the following

>>>fm = fm_model()
>>>dir(fm.regressor)
['__abstractmethods__', '__add__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__len__', '__lt__', '__module__', '__ne__', '__new__', '__or__', '__reduce__', '__reduce_ex__', '__repr__', '__ror__', '__setattr__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '__weakref__', '_abc_impl', '_add_step', '_get_params', '_is_stochastic', '_memory_usage', '_multiclass', '_raw_memory_usage', '_repr_html_', '_set_params', '_supervised', '_tags', '_transform_many', '_transform_one', '_unit_test_params', '_unit_test_skips', 'clone', 'debug_one', 'forecast', 'learn_many', 'learn_one', 'predict_many', 'predict_one', 'predict_proba_many', 'predict_proba_one', 'score_one', 'steps', 'transform_many', 'transform_one']

no weights yet...

[gbolmier]

Things are more nested than one may think 😉

At the top level we have the PredClipper object, that contains the Pipeline object, which contains the actual "model". So in order to access the weights you can try model.regressor.steps['FMRegressor'].weights

[victusfate]

Thanks much @gbolmier!
Making a note (going with the pseudo random scores for now) but it may be desirable down the road to pass back 0s for those predicted scores. On second thought, gonna try to make that change live and test it a bit.

[victusfate]

had to add one bit of modification to the keys (after logging them)

      score = 0
      model_item_id = 'item_' + str(item_id)
      if model_item_id in self.model.regressor.steps['FMRegressor'].weights:
        score = self.model.predict_one({'user': user_id,'item': item_id})
      scores[item_id] = score

[gbolmier]

it may be desirable down the road to pass back 0s for those predicted scores

Not sure it would benefit most use cases. Say we are trying to address a cold start problem and we have such a flag when doing a prediction on a new feature. Then the second time that feature appears it is quite likely that the model won't perform as well as we expect but unfortunately we don't have that flag anymore. The model will probably need a bit more than one observation to adapt reasonably. A better option could be to track the performance of the model on that new feature and then act accordingly to our expectations.

At the end I feel like it's something that should be handled more on the application side than on the library side, but don't hesitate to let us know if you think differently :)