alloc.c(177) : max_string_size reached

[mynameGY]

I have runned command "lime build flash", but it crashed with error:

Called from ? line 1
Called from CommandLineTools.hx line 1400
Called from CommandLineTools.hx line 25
Called from CommandLineTools.hx line 126
Called from CommandLineTools.hx line 619
Called from lime/project/PlatformTarget.hx line 70
Called from lime/tools/platforms/FlashPlatform.hx line 260
Called from lime/tools/helpers/FlashHelper.hx line 872
Called from lime/tools/helpers/FlashHelper.hx line 685
Called from format/swf/Writer.hx line 60
Called from format/swf/Writer.hx line 1425
Called from format/tools/Deflate.hx line 33
Called from /usr/lib/haxe/std/neko/_std/haxe/zip/Compress.hx line 48
Called from /usr/lib/haxe/std/haxe/io/Bytes.hx line 457
Uncaught exception - alloc.c(177) : max_string_size reached

When I delete some of the resources under my assets, I can build again successfully.
Thank's!

[AdamHarte]

This is driving me a bit crazy! Are there any plans for a fix? Or any work arounds? I am maintaining a legacy Flash/AIR project, and it just started producing this error after adding a few updated assets.

[player-03]

Can you give us your stack trace? Nearly all of the line numbers have changed since mynameGY's post. If I'm not mistaken, it will point to FlashHelper line 714, Writer lines 60 and 1452, Deflate line 34, Compress line 48, and Bytes line 551, but we should make sure.

Assuming it does point to those lines, the problem appears to be that Neko allocates a string in order to compress data, and has a maximum string length. Unfortunately, Neko is no longer being maintained, so I don't think a bug report would work.

Maybe this could be fixed if Lime's tools ran in HashLink or something, but it that's only built for Neko at the moment and can't easily switch. You could disable asset compression, but that would hurt the end user experience. The Deflate algorithm breaks its data into blocks, so it might be possible to compress only a bit at a time, but I have no idea how.

The most practical solution I can think of is to outsource the compression. For instance, 7-zip should be able to handle any amount of data.

if( compressed )
{
	var sevenZip = new haxe.Process("7za", ["a", "dummy.gz", "-si", "-so"]);

	sevenZip.stdin.writeBytes(bytes, 0, bytes.length);
	sevenZip.stdin.flush();
	sevenZip.stdin.close();

	var exitCode = sevenZip.exitCode();
	if( exitCode != 0 )
		throw '7-zip exited with code $exitCode';

	bytes = sevenZip.stdout.readAll();

	sevenZip.close();
}

Insert that code in Writer.hx, run lime rebuild tools, and then retry. Make sure you have 7-zip installed and that you can run 7za from your project folder. (7z is fine too but has more moving parts.)

Disclaimer: this code is untested.

[AdamHarte]

Thanks! I will try that out ASAP. Here is my stack:

Called from ? line 1
Called from CommandLineTools.hx line 1904
Called from CommandLineTools.hx line 22
Called from a C function
Called from CommandLineTools.hx line 125
Called from CommandLineTools.hx line 640
Called from lime/tools/PlatformTarget.hx line 98
Called from AIRPlatform.hx line 257
Called from lime/tools/FlashHelper.hx line 880
Called from lime/tools/FlashHelper.hx line 714
Called from format/swf/Writer.hx line 60
Called from format/swf/Writer.hx line 1450
Called from /Users/runner/hostedtoolcache/haxe/4.2.5/x64/std/haxe/io/BytesOutput.hx line 143
Called from /Users/runner/hostedtoolcache/haxe/4.2.5/x64/std/haxe/io/BytesBuffer.hx line 204
Uncaught exception - alloc.c(177) : max_string_size reached

Note that I am actually targeting AIR. lime test air.

After a discussion on Discord, and a few tests, I have found that the error seems to occur when my assets folder goes over 256MB in size. Seems like it is related to this? HaxeFoundation/neko#189

[player-03]

Called from format/swf/Writer.hx line 1450

Oh, interesting, for you it's happening a little earlier, during o.getBytes(). Perhaps mynameGY just below the limit, but when __deflate_bound() added some extra bytes of padding, it pushed it over.

I will try that out ASAP.

It isn't going to work in your case. According to your stack trace, it's crashing before it even reaches the if( compressed ) line.

Instead, you'd have to rewrite the whole writeEnd() function to avoid ever allocating the full buffer. I'll get back to you on that. Edit: that isn't even supported in Neko. You have to allocate the full buffer and it crashes if it's too big.

Seems like it is related to this?

Looks like the same issue, yeah. And they're right about switching to HashLink, but that'll take a while. Lime's tools have only ever been tested in Neko, and will probably break in all kinds of subtle ways anywhere else.

[tobil4sk]

It might still be a good idea to address this in the swf writer in format. There should be no need to allocate memory for the entire swf file at once, since it's possible for the swf's bytes to be deflated and written to the disk in chunks.

This would require working out the uncompressed size of the file in advance, to complete the header information. Here is some example code from the haxe codebase which does this: https://github.com/HaxeFoundation/haxe/blob/731dcd71f10c495a5a820449249fbb3d4b40a7c1/libs/swflib/swfParser.ml#L2062-L2066.

It can then use the haxe.zip.Compress.execute function to compress in parts, instead of .run which does everything at once. Using the (default) NO_FLUSH option gives control for deflate to decide how to group the data blocks, even though it is sent in parts. Then, the bytes that have been compressed can be written immediately to disk and the buffers can be reset. This way a whole swf file sized memory block doesn't ever have to be allocated at a single moment.

[player-03]

That sounds like it could work, though I doubt it'll be high priority. Since you seem more familiar with the details than I am, do you want to submit the issue?

[tobil4sk]

I doubt it'll be high priority

Makes sense. I've opened the issue anyway: HaxeFoundation/format#107