Make ToDer trait potentially more convenient by keeping state from to_der_len for write_der_content

[dequbed]

This is just a abstract idea I had because I'm currently writing a serializer using this crate:

The pattern I hit right now is that to be able to calculate the final encoded length for write_der_header I have to wrap all tagged fields and resolve things like their lengths, recursively.

But I have to throw away that state directly afterwards, only to reconstruct it again in the directly following call to write_der_content.

It's not a big overhead by any means, but I was wondering if I could remove it regardless.

Is that something that'd you be interested in as well and/or would want to work on together?

An idea I had is to make ToDer have an associated type that stores what is in essence a weak normalized form. The basic idea is that the WNF can be encoded into bytes directly without conditionals and without having to allocate/wrap any fields.

That solution would however make ToDer not object-safe until GATs for trait objects land. Not sure if that is a significant usability problem though.

(Fun fact: The last time I had this issue in asnom I actually inverted encoding, writing out the bodies of fields first and the headers afterwards, making heavy use of scatter/gather-IO. But that approach was absurdly complex, hard to use, and not all that efficient in most cases anyway.)

[chifflier]

Hi,

You are raising several good points here, on things that took me some time (and headaches) before getting to the current code. And I will humbly admit some better solution could exist :)
I'll reply here, but maybe it will be better to split the discussion on several issues.

About ToDer and length computation

I hesitated a long time between

• a streaming implementation, which would not need to allocate but would require computing the length recursively before writing the object (and thus, walking the DER tree twice).
• an allocating version. The code is much easier, though in some cases you have to allocate twice: once to serialize contents, and then you can compute length [1], and then you need to allocate again to write the header and the object
• finally I settled on using a Write trait so there is only one allocation

[1] Note that the fact that DER length is itself a variable-length fields does not make it easier to compute, and prevents leaving some placeholder bytes that could be updating after the computation.

I am not sure I get your idea of the weak normalized form, yet I'm really interested so if there is a way to write a minimal example or just some pseudo code it would be like, I'd gladly have a look. We can also discuss it on IRC/discord if you prefer.

Associated types and GAT on traits

I find it funny you are raising that, because it is something I also struggled on for other parts of this crate. In particular, I tried changing the Error parameter of the FromBer trait into an associated type (which would be better imho, since usually there is only one error type for a parse function), but quickly jumped into very complex errors. Roughly, they are related to the following:

• I need to defined bounds on the associated types
• FromBer is automatically implemented for any type providing TryFrom<Any>, but also needs to be implemented for Any. If the error type becomes too generic, then the compiler refuses because there is a default implementation of TryFrom<T> for T.
• I had other errors, and could never finally have something satisfying.

On that issue, I'd gladly take some advice/hints!

Note: for the ToDer trait, maybe the situation is different because the error is not generic. But it would be good to have a similar solution in the different traits used in that crate

[chifflier]

Update: ToDer has been re-rewritten in master, and will be published in next 0.8.
Note: this issue was not the main reason of the refactoring, but it is part of the problems targeted by the new version.
The new trait still uses different functions to get the length, and unfortunately may still requires multiple tree traversal (see

asn1-rs/src/to_der.rs

Lines 39 to 158 in c94c739

	pub trait ToDer {
	type Encoder: BerEncoder;
	/// Returns the length of the encoded content of the object
	///
	/// The length describes the _content_ only, not the header.
	fn der_content_len(&self) -> Length;
	/// Returns the total length (including header) of the encoded content of the object
	fn der_total_len(&self) -> Length {
	let (_, _, tag) = self.der_tag_info();
	let content_length = self.der_content_len();
	ber_total_length(tag, content_length)
	}
	/// Return the tag information to be encoded in header
	fn der_tag_info(&self) -> (Class, bool, Tag);
	/// Encode and write the content of the object to the writer `target`
	///
	/// Returns the number of bytes written
	fn der_write_content<W: Write>(&self, target: &mut W) -> SerializeResult<usize>;
	/// Encode and write the header of the object to the writer `target`
	///
	/// Returns the number of bytes written
	fn der_write_header<W: Write>(&self, target: &mut W) -> SerializeResult<usize> {
	let mut encoder = Self::Encoder::new();
	let mut sz = 0;
	let (class, constructed, tag) = self.der_tag_info();
	sz += encoder.write_tag_info(class, constructed, tag, target)?;
	// write length
	let length = self.der_content_len();
	sz += encoder.write_length(length, target)?;
	Ok(sz)
	}
	/// Encode and write the object (header + content) to the writer `target`
	///
	/// Returns the number of bytes written
	fn der_encode<W: Write>(&self, target: &mut W) -> SerializeResult<usize> {
	let sz = self.der_write_header(target)? + self.der_write_content(target)?;
	Ok(sz)
	}
	/// Encode and write the object (header + content) as TAGGED EXPLICIT and write it to the writer `target`
	///
	/// Usually, `class` is `Class::ContextSpecific`.
	///
	/// Returns the number of bytes written
	fn der_encode_tagged_explicit<W: Write>(
	&self,
	class: Class,
	tag_number: u32,
	target: &mut W,
	) -> SerializeResult<usize> {
	let length = self.der_total_len();
	let tagged_header = Header::new(class, true, Tag(tag_number), length);
	let sz = tagged_header.der_write_header(target)?
	+ self.der_write_header(target)?
	+ self.der_write_content(target)?;
	Ok(sz)
	}
	/// Encode and write the object (header + content) as TAGGED IMPLICIT and write it to the writer `target`
	///
	/// Usually, `class` is `Class::ContextSpecific`.
	///
	/// Returns the number of bytes written
	fn der_encode_tagged_implicit<W: Write>(
	&self,
	class: Class,
	tag_number: u32,
	target: &mut W,
	) -> SerializeResult<usize> {
	let length = self.der_content_len();
	let (_, constructed, _) = self.der_tag_info();
	let tagged_header = Header::new(class, constructed, Tag(tag_number), length);
	let sz = tagged_header.der_write_header(target)? + self.der_write_content(target)?;
	Ok(sz)
	}
	/// Write the DER encoded representation to a newly allocated `Vec<u8>`
	fn to_der_vec(&self) -> SerializeResult<Vec<u8>> {
	let mut v = Vec::new();
	self.der_encode(&mut v)?;
	Ok(v)
	}
	/// Encode in DER and write the object (header + content) to the writer `target`
	///
	/// Returns the number of bytes written
	fn write_der<W: Write>(&self, writer: &mut W) -> SerializeResult<usize> {
	self.der_encode(writer)
	}
	//--- DEPRECATED
	/// Get the length of the object (including the header), when encoded
	///
	// Since we are using DER, length cannot be Indefinite
	#[deprecated(since = "0.8.0", note = "Use `der_total_len()` instead.")]
	fn to_der_len(&self) -> crate::Result<usize> {
	match self.der_content_len() {
	Length::Definite(sz) => Ok(sz),
	Length::Indefinite => Err(crate::Error::IndefiniteLengthUnexpected),
	}
	}
	#[deprecated(since = "0.8.0", note = "Use `to_der_vec()` instead.")]
	fn to_der_vec_raw(&self) -> SerializeResult<Vec<u8>> {
	self.to_der_vec()
	}
	}

for new trait declaration).
There is now an Encoder object associated to traits, it might be part of the solution. Maybe a method .prepare() can be added somewhere to populate a stateful Encoder?