Serialization / deserialization speed optimizations

[KeithVinson]

Hello All,
I am wondering how best to control Message Pack to maximize the speed at which large objects can be read / written to disk. In my specific case the files on disk are approaching 10 GB in size. They contain fairly large Sorted Dictionaries. When I load them into memory the process can take upwards of 6-8 minutes to load. This process is running on a very beefy server with dual CPU totaling 32 cores and having 256 GB of memory. When reading the disk I can see periods of up to 750 MB / second transfer speeds, but there are periods where no disk transfers are occurring even though the load process has not completed. I am wonder if their are some options I should select for the MessagePack Serializer that could improve my performance. When accessing the disks (NVMe in a RAID 5 array) I see a very curious "ringdown" pattern on disk transfers as the load process proceeds. Any help you might suggest would be welcome.

Here is a sample of the code used to load the Dictionaries:
public static T ReadMsgPkFile<T> (string filePath) { using (Stream stream = File.Open (filePath, FileMode.Open, FileAccess.Read, FileShare.Read)) { return (T)MessagePackSerializer.Deserialize<T> (stream); } }

[yfakariya]

MessagePack for CLI simply uses System.IO.Stream, so you might improve performance of I/O. For example, you can adjust buffer size of your FileStream with its constructor parameter to be the write more chunky.

[KeithVinson]

Thank you, I will check it out! Keith Vinson CTO Image Access, Inc. 543 NW 77th Street, Boca Raton, FL 33487 Tel: 561-886-2951 Fax: 561-431-2766 ***@***.*** Image DLSG is a Division of Image Access, Inc. www.ImageAccess.com / www.DLSG.com DLSG is the Most Trusted Name in Digitization for Universities ▪ 49 of the top 50 universities are DLSG customers ▪ DLSG products serve over 70% of students at US universities ImageVisit us on Facebook to see what people are posting about KIC!

…

________________________________ From: Yusuke Fujiwara ***@***.***> Sent: Saturday, May 20, 2023 8:13:36 PM To: msgpack/msgpack-cli ***@***.***> Cc: Keith Vinson ***@***.***>; Author ***@***.***> Subject: Re: [msgpack/msgpack-cli] Serialization / deserialization speed optimizations (Issue #357) MessagePack for CLI simply uses System.IO.Stream, so you might improve performance of I/O. For example, you can adjust buffer size of your FileStream with its constructor parameter to be the write more chunky. — Reply to this email directly, view it on GitHub<#357 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AKVJS5JWV5ET255R3ZL5NY3XHFT4BANCNFSM6AAAAAAYHB4ZCI>. You are receiving this because you authored the thread.Message ID: ***@***.***>