Why does System.Formats.Nrbf.ArrayRecord.GetArray require the type of the object in the array?

[qsc-jhndnn]

I'm using the prerelease Nrbf library from Nuget. It's not clear to me why I need to pass in the type into ArrayRecord.GetArray given that all I get from that call is an array of ClassRecords. In my use case I would like to load data that was serialized using the BinaryFormatter in an application that doesn't have any reference to the classes used during serialization. In moving to a new serializer we want to rearchitect the model so keeping the old models around isn't ideal. I've also determined that it's really just checking the type name - I was able to create an empty shell of the class with the same name/namespace without any members at all and was able to deserialize without any issues.

Here's a simple example.

using System.Runtime.Serialization.Formatters.Binary;
using System.Formats.Nrbf;

namespace SerializerTest
{
  [Serializable]
  public class Bar
  {
    public string Name { get; set; }
  }

  [Serializable]
  public class Foo
  {
    public string Name { get; set; }
    Bar[] Bars { get; set; }

  public static void Test()
    {
      Foo foo = new Foo
      {
        Name = "John",
        Bars = new [] 
        {
          new Bar { Name = "Bar1" },
          new Bar { Name = "Bar2" },
          new Bar { Name = "Bar3" },
        }
      };

#pragma warning disable SYSLIB0011 // Type or member is obsolete
      var bf = new BinaryFormatter();
#pragma warning restore SYSLIB0011 // Type or member is obsolete
      var ms = new MemoryStream();
      bf.Serialize(ms, foo);
      ms.Position = 0;

      SerializationRecord headerObject = NrbfDecoder.Decode(ms, null, true);
      var cr = headerObject as ClassRecord;
      Console.WriteLine($"### HEADER");
      Console.WriteLine($"Name: {cr.GetString("<Name>k__BackingField")}");

      var bars = cr.GetArrayRecord("<Bars>k__BackingField");

      // possible to get this without knowing the type?
      var _bars = bars.GetArray(typeof(Bar[]));
      Console.WriteLine($"### BARS");
    }
  }
}

In the case that I was generating the serialization data somewhere else I could just define this

  public class Bar
  {
  }

and would be able to still read the Name member so it's not clear what the type accomplishes.

[huoyaoyuan]

GetArray lets the NRBF library creates the array and objects for you. For security manner, it must not create a type that the caller code is not aware of.

I was able to create an empty shell of the class with the same name/namespace without any members at all and was able to deserialize without any issues.

This is safe because the code of the type has clearly no risk. It also guarantees that the payload doesn't contain other types.

[qsc-jhndnn]

Thank you for the response

GetArray lets the NRBF library creates the array and objects for you. For security manner, it must not create a type that the caller code is not aware of.

In my example, if I have a single Bar as a member I can create a ClassRecord for that using GetRawValue without any class defined. That seems to be functionally equivalent to the array of items I get back from GetArrayRecord. Why is GetRawValue considered safe without a type but GetArrayRecord isn't?

[huoyaoyuan]

Inspecting the implementation and documentation, GetRawValue is only allowed to return primitive types, or a SerializationRecord that you need to inspect yourself. The possible return types are a closed set.

[qsc-jhndnn]

I feel like I'm missing something here since I'm still confused...

For fields

1. Call GetRawValue on ClassRecord which returns a SerializationRecord
2. The returned value can be cast to a ClassRecord
3. Call GetString on returned ClassRecord to get a field from class

Compared to arrays

1. Call GetArrayRecord on ClassRecord which returns an ArrayRecord
2. Call GetArray on ArrayRecord which returns an array of SerivceRecords? Documentation is a little unclear since it says "serialized records themselves"
3. Cast each item in array to ClassRecord
4. Call GetString on returned ClassRecord to get a field from class

So in both cases I end up with a ClassRecord that I inspect via GetString. There doesn't seem to be a difference between the 2 approaches other than GetArray requiring a type.

[huoyaoyuan]

@adamsitnik can you answer this?

[adamsitnik]

I'm using the prerelease Nrbf library from Nuget.

Big thanks for using it and sharing your feedback! And appologies for such a late response (I was on a parental leave for a few weeks).

It's not clear to me why I need to pass in the type into ArrayRecord.GetArray

Short answer: There are multiple ways of attacking an app the reads payloads produced by BinaryFormatter. The GetArray method requires the user to provide the expected type to ensure that it's the array they are expecting (it has not been changed by the attacker).

Long answer: Just to give you one example of an attack: The NRBF format has a concept of a serialization record that represents multiple nulls with just a single Int32.
With such a record, an array of strings with 2 000 000 000 null elements can be represented with only 10 bytes.
It's also possible to represent jagged arrays, such as arrays of arrays of strings (with 2b null elements each). On top of that, it has a built in concept of references. So a payload can define one large array of nulls and then another arrays of arrays with just references to the large array.

Now imagine an app that expects an array of 20-40 integers. The developer checks the payload size and it's just 90 bytes, so it passes the payload to the NrfbDecoder. The decoder decodes the payload and returns a SerializationRecord that can be casted to ArrayRecord.
The user calls GetArray and gets an array of arrays of strings, each having 2b null elements. Few dozens of GB are allocated, the service most likely dies with OutOfMemory.

In moving to a new serializer we want to rearchitect the model so keeping the old models around isn't ideal.

To be honest this is a scenario that I have not taken into account.

The good news is that I expected arrays of classes to be most common use case and introduced SZArrayRecord<T> that derives from ArrayRecord and exposes T GetArray() method that does not require you to provide the type. The T can be any primitive type or SerializationRecord.

Here is an example of using it:

using System.Formats.Nrbf;
using System.Runtime.Serialization.Formatters.Binary;

namespace NrbfArrayOfClassses
{
    internal class Program
    {
        static void Main(string[] args)
        {
            MyClass[] instances =
            [
                new MyClass() { Text = "Some", Index = 0 },
                new MyClass() { Text = "Value", Index = 400 }
            ];

            using MemoryStream payload = Serialize(instances);
            SerializationRecord serializationRecord = NrbfDecoder.Decode(payload);

            if (serializationRecord is SZArrayRecord<SerializationRecord> arrayOfRecords)
            {
                SerializationRecord[] arrayItems = arrayOfRecords.GetArray()!; // no need to specify type
                foreach (SerializationRecord arrayItem in arrayItems)
                {
                    if (arrayItem is ClassRecord classRecord)
                    {
                        Console.WriteLine($"TypeName: {classRecord.TypeName.FullName}, Text: {classRecord.GetString(nameof(MyClass.Text))}, Index: {classRecord.GetInt32(nameof(MyClass.Index))}");
                    }
                }
            }
        }

        protected static MemoryStream Serialize<T>(T instance) where T : notnull
        {
            MemoryStream ms = new();

#pragma warning disable SYSLIB0011 // Type or member is obsolete
            new BinaryFormatter().Serialize(ms, instance);
#pragma warning restore SYSLIB0011 // Type or member is obsolete

            ms.Position = 0;
            return ms;
        }
    }

    [Serializable]
    public class MyClass
    {
        public string? Text;
        public int Index;
    }
}

Note: In our case the T is SerializationRecord not ClassRecord because it's possible to have an array of abstractions like IComparable that can contain both ClassRecord and PrimitiveTypeRecord.

I was able to create an empty shell of the class with the same name/namespace without any members at all and was able to deserialize without any issues.

That was very clever! And thank you for sharing the workaround to get others unblocked.

[qsc-jhndnn]

Thanks for the useful explanation.

In my trivial example program things work as you describe - I can cast the ArrayRecord to a SZArrayRecord<SerializationRecord> without any issues. The odd thing is that on my real data the cast returns null. In both cases I get what looks like the same data from GetArrayRecord() - a System.Formats.Nrbf.ArrayRecord {System.Formats.Nrbf.ArrayOfClassesRecord}. Any idea what would cause that cast to fail?

[qsc-jhndnn]

I was mistaken - neither the simple example or my real data worked :(. I realized I should update to the newer version and once I did that things worked as described - thanks!

One question - is there a way to mark the whole application as Experimental so I don't have to litter my code with [Experimental("SYSLIB5005")] on every class?

[adamsitnik]

is there a way to mark the whole application as Experimental

You can suppress this particular warning for the entire project by adding following line to your project file:

<NoWarn>$(NoWarn);SYSLIB5005</NoWarn>

You can find more details here: https://learn.microsoft.com/en-us/dotnet/fundamentals/syslib-diagnostics/experimental-overview#suppress-warnings