How can one validate EML documents in python?

[mbjones]

Suzanne Dietrich asked:

Hello,

I am a Co-PI on an NSF-sponsored grant, Multispecies Ovary Tissue Histology Electronic Repository (MOTHER) [http://mother-db.org/], that is leveraging EML for data provenance and including our metadata as part of the additional metadata element of EML. I was wondering whether the emlparser (https://knb.ecoinformatics.org/emlparser/) is available to be called programmatically using python to automate our curation process? This is only for curation and there would not be much traffic.

We are currently in the process of extending ezEML to include MOTHER but I do not expect this to be production ready until later this fall.

Thank you for your consideration,
Suzanne

[mbjones]

Hi Suzanne --

The emlparser is a Java-based validation tool for which we created both a command line binding (in bash) and a web-service frontend. There is also an independent EML validator implemented in the R EML package. You should be able to automate the process through python through one of the following mechanisms:

1. call a system process to launch the commandline validator (shipped with the EML release on github)
2. call the web-service from within python (hosted on the KNB, or you can easily host your own copy from source)
3. use Python-Java language bindings to call the Java classes directly
4. Reimplement the validator in native python (it's not too complicated, and the R and Java language implementations would be a guide)    -- see https://eml.ecoinformatics.org/validation-and-content-references.html    -- if you chose this route, I'm sure others would be excited if you open sourced your code!

And probably several more possibilities. We're starting to guide people to use our DataONE Discussions site for these types of questions so that others can benefit from the answers, and so I have copied your question and my response there. If you have any other questions, feel free to follow up on that thread.

Best,
Matt

[servilla]

The most commonly used Python package IMO for managing (and parsing) XML is lxml (https://lxml.de/), but my experience with it for validating an XML instance document is limited. The basics are as follows (and your mileage may vary):

from lxml import etree

# XML DOM instance of EML document
xml_file = etree.parse("EML.xml")

# XML Schema validator instance for EML 2.2.0 schema - assumes local copy of schema
xml_validator = etree.XMLSchema(file="./eml-2.2.0/xsd/eml.xsd")

# Validate method returns a boolean
is_valid = xml_validator.validate(xml_file)

if is_valid:
    print("EML document is valid")
else:
    print("EML document is not valid")

To see more details about the validation exception, you can substitute assertValid for the validate method.

Perhaps the best introduction to lxml is John Shipman's "Python XML processing with lxml". I hope this helps.

Sincerely,
Mark

[jusana]

Hello,
I made one with generateDS (https://pypi.org/project/generateDS/).
.py38/bin/generateDS -o "emllib.py" --member-specs="list" ../xsd/eml.xsd --> got a 2.5Mb EMLlib.py file !!
So i can validate and CRUD eml files with a python API ... ;)

it's not perfect since the XSD has many imports and potential infinite loops ..
and the GenDS api is not that easy to play with but still fantastic and pure python !!

If anyone would create a repo to maintain an official D1 emllib in python ... it would be great !!

I tried metapype (https://github.com/PASTAplus/metapype-eml) but i didnt find it easy to use and furthermore the bindings seem to be made by hand !! GenDS is only based on source xsd !

[laurenwalker]

@jusana This looks really great, thank you for sharing! I think a lot of people would find a Python implementation with a GUI really useful.

DataONE is community maintained, so we could absolutely create a repo for your work. I think the Github "owners" of the Github DataONEorg organization would need to orchestrate that. (@mbjones ?)

[jusana]

Thanks @laurenwalker

I forgot to mention pyXB (which unfortunately seems unmaintained) which is used in the D1_python client to validate some D1 specific xml formats. I did not try it (cause of unmaintained lib) but i guess it would be worth a try , maybe you already tried pyxb on eml's xsd since you have experience with this lib.

Now for the GUI I screenshot above ... it is a little more complicated !! ;)
Actually the validator works fine for my needs as of now (i have a few standard test cases i picked from EML files shipped various repos, and from our own EMLs) .
For the html form generation , I use Django 3 (normaly genDS should be able to auto generate django (which version?) forms and models from xsd following the documentation but i failed to generate them (actually i found circular problems that prevent django3 to work properly) ... I did not contact the author of the lib Dave Khulman but it would certainly be a great help)
So i made the django form and render (with a js tree lib) by myself. BUT the real difficulty is to enforce a 'a apriori' correct/valid EML file and to propose/generate a GUI that adapts to the live building of the file (for instance the xs:sequence or xs:choice are really tricky in terms of UI auto generation) ...
Of course, it is a lot easier to validate 'a posteriori' when the eml file is already existing !

ezEML uses flask but has a really great workflow to enforce the correctness of the building of the EML but this workflow is "hardcoded" and correspond to an EML 'template' they wish to produce .... On my side, I am trying to create an generic/agnostic form generator only based on the xsd and python lib generated by genDS

... sorry for the long and certainly hard to understand answer .. !

[servilla]

Hi @jusana,

Just a couple of thoughts here regarding Metapype and ezEML...

First, Metapype is simply an experiment to see if we can decouple the XSD constraints from the EML metadata standard, which IMO are two very different things. We have found that one of the biggest stumbling blocks for our community to develop good EML (and EML applications) is understanding and using correctly the EML XML schema. The goal of Metapype was to provide an environment where EML could be generated in what we believe is a more natural programmatic approach to capturing structured metadata content - a directed acyclic graph. I suppose one could also do the same with hierarchical XML, but working with XML (also LXML) can be very non-intuitive at times. We have no idea if this experiment will pay off (and I agree that the Metapype API can be difficult to fully understand), but it is at the core of ezEML (I'm not sure if you realize that), and ezEML is being used quite often now by our community and does produce schema-valid EML.

Second, ezEML does use Flask as its underlying web framework, but it could use Django, Pyramid, or FastAPI just as easily. We don't consider Flask a detriment unless async is necessary to improve the performance. Also, the correctness of EML is not enforced by the workflow, but rather by the standard validation processing of Metapype. One could easily start an EML document by describing a data table and then validate it independent of any other XML schema required structure. This is some of the flexibility provided by Metapype. Note also that the workflow is not dictated by an EML "template" - the template capability was added long after ezEML was functional and only provides ready-made content that can be reused by community members affiliated with particular organizations without having to regenerate everything by hand.

And like your wonderful work, both Metapype and ezEML are pure Python! :-)

I'd be glad to discuss both of these projects further if you are interested. And it would be great to hear more about your work too!

Sincerely,
Mark

[jusana]

Hi @servilla

Indeed if the goal of metapype was to decouple constraints from XSD ... that explains it all (so what i felt as "made by hand" was not totally wrong). Actually I find it to be a very interesting approach. And i guess the typical api usage is to build the eml object piece by piece (title, then dataset, datatable etc ...) ... I'll try another look but i think the implementation in metapype of EML schema evolution (2.2.1 or 3.x.y ...) could be very tedious
and yes i perfectly realized it is core of ezEML ... and that goes with my point of a "predetermined" EML specific workflow ... rather than a generic/agnostic one .

And I agree ezEML provides a very convenient workflow with nice stats and suggestions to improve metadata quality !!
Surely the webframework is not really a point and any of them could totally be used with a few apropriate adaptation.

When I say "template" or "predetermined" I mean there is a guide a rail ... any markup,tag or attribute cannot be filled ex nihilo
Why this genericness thing is important to me is because i anticipate (in my very specific case) i will have to face schema evolution or even schema changes (iso or else) ... ( I know @yvanlebras well ;) )

I'll try to update you with my work if make notable progress

And I am looking forward to hearing from you python fans !!

Sincerely,
Julien