Interoperability tests wakaama/Leshan

[sbernard31]

I think that both projects could benefit to share efforts about interoperability tests between Leshan/wakaama. (and maybe more libraries)

I didn't think too much of this but I create this ticket to centralize discussion/idea about it.

I don't know if we can define a minimalist interface which could allow us to write test one for any combination of interoperability tests 🤔

[boaks]

For Californium I added such Californium - Interoperability Tests.

These test are mainly focus on several DTLS libraries (tinyDTLS, openssl, mbedtls, gnutls) and libcoap. Currently the tests are rather basic than advanced. That helped to localize some issues ahead (e.g. tinydtls ecc encoding).

I considered to extend them to the "ETSI plugtest", but I didn't find the time to do so.

[hannestschofenig]

In the OMA we are thinking about combining our next testfest with the IETF hackathon in March 2022.

In time for the testfest we would like to update our test specifications (see http://www.openmobilealliance.org/release/LightweightM2M/ETS/). Since we are currently reviewing the test object contributed by Simon (see OpenMobileAlliance/lwm2m-registry#639), the test spec might get augmented by it.

I am wondering whether it would make sense to have some of the interoperability tests between Wakaama & Leshan be based on automatic tests based on the LwM2M specification.

[sbernard31]

I had planned to refer the TestObject here too. @hannestschofenig thx for doing this. 🙏

I am wondering whether it would make sense to have some of the interoperability tests between Wakaama & Leshan be based on automatic tests based on the LwM2M specification.

Just to be sure when you say "based on the LwM2M specification", you talk about :

1. the LWM2M specification in a general way
2. or you specifically refer to ETS - Enabler Test Specification (Interoperability)for Lightweight M2M)?

for 1. it totally makes sense.
for 2. f I didn't look at ETS recently, but it probably make sense too.

In all case, TestObject could be a good start to think about that (so review at OpenMobileAlliance/lwm2m-registry#639 are welcomed 🙏 )
Finally, I guess we will quickly come the question how this should looks technically looks like ?

[hannestschofenig]

@sbernard31 The scope of testing is indeed an interesting topic. The LwM2M ETS focuses on testing LwM2M functionality -- not TLS, CoAP or any other of the referenced specs. The assumption is that the IETF specs we make use provide their own test cases and the implementations of those IETF specifications already have their test strategy. In practice, this is a somewhat bold assumption. While many authors of IETF specifications (or others in the working group) write implementations of their specifications, there is in general no test description published with RFCs nor is there a structured test approach used in general. Hence, the level of testing varies hugely throughout the IETF.

If we just focus on LwM2M-specific testing, then there is the question about how those tests should be done automatically. I have been working with the IETF OAuth community and they have created automated test (see https://github.com/rohe/oidctest). I was thinking about learning something from that test framework.

I am, of course, open to other suggestions.

[sbernard31]

If we just focus on LwM2M-specific testing, then there is the question about how those tests should be done automatically. I have been working with the IETF OAuth community and they have created automated test (see https://github.com/rohe/oidctest). I was thinking about learning something from that test framework.

I looked at this very quickly, for now this is not really clear to me how this tool looks like ? or how is it supposed to be used.
(E.g. how we add test ? how you test your implementation against it ? is it a kind of interoperability check or more spec validation test ?).

About testing LWM2M automatically, I can imagine several solutions
(:warning: I didn't think too much of this :warning: )

1. Test API

Define a kind of common API that client, server and bsserver should implements. (I don't know what could be the best ? CLI API ? REST API ? any more simple socket API ? ...)
We write our LWM2M automatic tests using only the defined API.
Then implementer need to provide binary which respect the defined API.

Pros : you can run your tests with any combination of Client / Server / Bsserver
Cons: All project implementation need to provide binaries which implement the common API defined.

2. Testing Service

Another solution would be to try to write tests which are only based on LWM2M API. Meaning you just need to provide something which support the LWM2M spec/API. You take your implementation and just connect it to a kind of testing service.

Pros: no need to implement a test dedicated binary.
Cons :

• I feel we can easily imagine a Testing Service which acts as a LWM2M server and BS server to easily test clients. But testing server or BS server this way seems more complicated to me.
• You don't really test interoperability but just validity against one Testing Service.

[sbernard31]

This Zephyr PR reuse leshan-server-demo and leshan-bsserver-demo to do some interoperability tests :
zephyrproject-rtos/zephyr#64013

Reusing demo REST API is not really recommended but what could be the right way to do that ? 🤔