How to configure to robustly de-serialize collections (get_collection_of_object_values)

[thelazydogsback]

Some of my entities do not have the correct type/value of properties which fail during deserialization.
How can I configure my python Kiota client to be robust to these errors?
Specifically, if I'm retrieving a page of objects, then the entire page will fail if there is an invalid entity.
I would like to be able to still get valid entities while skipping the bad ones, and possibly collecting the errors/ids of ones that caused the errors.
I had thought about setting my pageSize=1, but that doesn't work, as the entire request will fail and I'm unable to get the continuation token for the next page.

thanks

[baywet]

Hi @thelazydogsback
Thank you for using kiota and for reaching out.

You are correct in the sense that kiota's default serialization providers are fairly strict. We'd rather fail noisily when something is wrong than silently do something wrong when it comes to something as crucial as integrating with APIs.
On the client side, beyond correcting the description (e.g. use union types), or implementing your own serialization provider, there's not much that can be done to change/extend the serialization logic.

Let us know if you have any additional comments or questions.

[thelazydogsback]

Thanks, @baywet. This is another case (such as internal/k8s network authentication using plain http, etc.) where reality isn't quite as nice as we'd like when designing a framework.
I've had to write custom serialization logic for handling writing out dates without times, so I can try writing some custom deserialization logic for this case. (If so, I'll follow up here.) I guess my only other choice would be to add a new API on my backend to return a list of IDs only rather than full entities, and then retrieve them one-by-one on the client side, so that I can handle each failure.
I certainly agree that the default behavior should be strict - but it would be nice to be able to relax when needed.

[thelazydogsback]

@baywet Given this stack trace:

It looks like maybe either the exception-handler is to narrow, or you need to check for an empty string first:

So during a _try_get_anything, an empty string will fail in pendulm.parse

[thelazydogsback]

I've updated the deps and will let you know if the issue is resolved. (Also added custom deserialization logic to be less strict, but will log.)

[thelazydogsback]

Ok so the issue looks like another issue with round-tripping values.

JsonParseNode._try_get_anything() calls parse_timedelta_string(), which will create a timedelta object from a iso8601 string such as "P1Y".
However, when serializing, JsonSerializationWriter.write_timedelta_value() will just call str() on the timedelta, resulting in a string like "365 days, 00:00:00" - this value will fail during a PUT as it's no longer a valid duration.
Note that write_timedelta_value() is smart enough to try see if a string looks like an iso8601 string and keeps it that way, but as it has already been deserialized as a timedelta, it's too late for that. ISMT that the str() version of a td will never be the right choice here.
So to fix this and the writing of iso dates without times I've written the following:

However, this has some issues, because in general a timedelta and duration can't be converted to each other - e.g, for "1M", which month exactly? - the timedelta could be anywhere between 28 and 31 days. As Python doesn't have a standard way to represent non-exact durations, rather than converting to a timedelta, it may be better by default to keep duration strings as strings like "P1Y" so they round-trip properly. (Unless you wanted to introduce some sort of Duration model in kiota.abstractions that did round-trip - but doesn't really seem worth it.)

[thelazydogsback]

I think a good rule-of-thumb would be to have _try_get_anything() only perform an implicit conversion from a str to some object at read-time only if we know it will be written back out in the exact same format?

[thelazydogsback]

@baywet so to handle this on the read side assuming kiota keeps its default behavior:

from uuid import UUID
from kiota_abstractions.date_utils import (
    datetime_from_iso_format_compat, time_from_iso_format_compat, parse_timedelta_string
)
from datetime import date
import re

class CustomJsonParseNode(JsonParseNode):

    LeaveDurationsAsStrings = True

    def __init__(self, content: Dict[str, Any]):
        super().__init__(content)
        
    # This doesn't get called from _assign_field_values -- return CustomJsonParseNode in override/copy of _assign_field_values()?
    def _create_new_node(self, node: Any) -> JsonParseNode:
        new_node: JsonParseNode = CustomJsonParseNode(node)
        new_node.on_before_assign_field_values = self.on_before_assign_field_values
        new_node.on_after_assign_field_values = self.on_after_assign_field_values
        return new_node

    # Defined as private in superclass - redefine here
    def __is_four_digit_number(self, value: str) -> bool:
        pattern = r'^\d{4}$'
        return bool(re.match(pattern, value))

    def try_get_anything(self, value: Any) -> Any:
        if isinstance(value, (int, float, bool)) or value is None:
            return value
        if isinstance(value, list):
            return list(map(self.try_get_anything, value))
        if isinstance(value, dict):
            return dict(map(lambda x: (x[0], self.try_get_anything(x[1])), value.items()))
        if isinstance(value, str):
            try:
                if self.__is_four_digit_number(value): return value
                if value.isdigit(): return value
                datetime_obj = datetime_from_iso_format_compat(value)
                return datetime_obj
            except ValueError:
                pass

            try: return UUID(value) 
            except: pass

            if not CustomJsonParseNode.LeaveDurationsAsStrings:
                # Internally this calls parse_timedelta_string which matches both ISO8601 
                #   and hh:mm:ss formats - it may be valid to always convert the latter to a timedelta
                #   but requires refactoring.
                try: return parse_timedelta_string(value) 
                except ValueError: pass

            try: return date.fromisoformat(value) 
            except ValueError: pass

            try: return time_from_iso_format_compat(value) 
            except ValueError: pass

            return value

        raise ValueError(f"Unexpected additional value type {type(value)} during deserialization.")

    def _assign_field_values(self, item: Parsable) -> None:
         ...

[thelazydogsback]

BTW, my mistake _create_new_node() does get called from other places, like get_collection_of_object_values - so seems like also would be in _afv()

[thelazydogsback]

Also, I think the call to JsonParseNode() below maybe should be self._create_new_node() in case the user has a custom ParseNodeFactory?

[baywet]

If it's a different factory, the expectation is the implementation will be different as well. Which means the application would not go through that code path at all?

[thelazydogsback]

Well, I guess only if the new factory reimplements all methods that create a JsonParseNode (I guess just _assign_field_values()) - but if it only overrides select methods from JsonParseNode, like I was doing, this may not be that case.
I just assumed that's what _create_new_node() was for, like _create_new_writer() on the serialization side?
Anyway, as I'm overriding _afv() at the moment it's no big deal - I was just trying to use the correct pattern.