detect duplicate records for pd types

[RabiaSajjad]

No description provided.

[JVickery-TBS]

@RabiaSajjad do you think that this might be a thing needed in the future for checking duplicate PD records when merging organizations? Or do you think this is just a very one time thing?

If we could use it in the future, it might just be easier making this into a ckan command in our Canada plugin. That way we can get all the information from the PD chromos (yaml files), and would not need to download the large CSV files as they would already be on the server in the backups directory.

Otherwise this one off script would be fine. Instead of doing pandas for this, you could try using the built-in csv library (or unicodecsv if python2), that way we would not need the extra dependency just for this one script.

[wardi]

pandas is good stuff, though. No reason to use worse tools to avoid adding something to dev-requirements.txt