Duplicates in processed data

[pnoll1]

Describe the bug
There's multiple copies of same records in processed data

To Reproduce

• View us ri providence from 1/15/23
• view duplicates based on hash
  select hash, count(*) from us_ri_providence_addresses_city group by hash having count(*)>1;
  • 1aab47288c5ab757 is duplicated 21 times

Expected behavior
Duplicates removed after hashing

Additional context
It's unclear if this an intended limitation since I haven't seen any documentation on what guarantees are given for data

[iandees]

The hash attribute is a hash of the other attributes in the row, so if the row is empty then the hash will be the same.

[pnoll1]

Thanks for the link, very helpful.

The data I’m talking about above are valid addresses with all the fields filled out. I didn’t expect this because I thought addresses between files were deduped which is why so many files are hashes only.