U an M training

[deanditton]

Hello all,

I would like to ask for some feedback on how we are approaching training U and M values if that is okay Please let me know if this is not the area for it.

Does my current approach make sense? or am I missing something?

Currently we have are linking across two databases with these columns

[('id', 'string'),
 ('frn_array', 'array<string>'),
 ('title_long', 'string'),
 ('title_short', 'string'),
 ('start_date', 'date'),
 ('end_date', 'date'),
 ('year_award', 'string'),
 ('year_start', 'string'),
 ('year_end', 'string'),
 ('amount_aud', 'decimal(10,0)'),
 ('f_organisation_name_array', 'array<string>'),
 ('r_organisation_name_array', 'array<string>'),
 ('r_organisation_name_array', 'array<string>'),
 ('all_organisation_name_array', 'array<string>')]

This is my training function

u_rules = [
  f"{title_long} and array_size(array_intersect(l.frn_array, r.frn_array)) >= 1",
  dateRules,
  f"{amount_aud} and array_size(array_intersect(l.researcher_organisation_name_array, r.researcher_organisation_name_array)) >= 1",
  f"{amount_aud} and {year_start} and {year_end}"
]

def trainning_function(linker):
  # Estimate probability
  for rule in u_rules:
    linker.estimate_probability_two_random_records_match(rule, recall=0.6)

  # U values first run with blocks
  mlflow.log_param('estimate_probability_rules', rule)
  linker.estimate_u_using_random_sampling(max_pairs=1e4)
  # for block in [["doi", "full_title"], ["year"], ['isbn'], ['eisbn'], ['date']]:
  #   training_blocking_rule = block_on(block)

  # M values
  for col in columns:
    blocking_rule_for_training = block_on([col])
    linker.estimate_parameters_using_expectation_maximisation(blocking_rule_for_training)

  # U values no blocks
  linker.estimate_u_using_random_sampling(max_pairs=1e4)

[RobinL]

Some suggestions:

1. Order matters. You want to run estimate_u_using_random_sampling before estimate_parameters_using_expectation_maximisation since it anchors the estimation procedure
2. You want a much bigger sample size for estimate_u_using_random_sampling
   So something like

  # this takes a list of rules, each of which should capture matches but not non matches
  linker.estimate_probability_two_random_records_match(det_rules , recall=0.6)

  # Use a bigger sample size
  linker.estimate_u_using_random_sampling(max_pairs=1e7)


  # M values estimated once anchored by other params
  for col in columns:
    blocking_rule_for_training = block_on([col])
    linker.estimate_parameters_using_expectation_maximisation(blocking_rule_for_training)

[deanditton]

Thanks heaps! I appreciate you taking your time to help me out.