Can't get the custom formatter working

[DCNick3]

I have a format describing a filesystem image. To save on clicks, I want to see the filename of a directory entry directly in the "Value" column of the "Pattern Data" view. However, I just can't get it to convert the entry's name to be displayable. I've tried returning e.name directly, and converting it with either str or std::string::to_string...

Pattern code:

// necessary for large files
#pragma pattern_limit 0x200000

#include "type/magic.pat"
#include "std/string.pat"

struct RomHeader {
    type::Magic<"ROM "> magic;
    u32 version;
    u32 index_size;
    u32 unk;
};

bitfield NameAndFlags {
    name_offset : 31;
    is_directory : 1;
};

using Directory;

struct Entry<auto DirOffset> {
    NameAndFlags name_off [[color("FF0000")]];
    u32 offset [[color("00FF00")]];
    u32 size [[color("0000FF")]];
    
    char name[] @ (DirOffset + name_off.name_offset);
    
    if (name_off.is_directory) {
        if (name != ".\x00" && name != "..\x00") {
            Directory directory_data @ (offset * 0x10);
        }
    } else {
        // this creates too much patterns for large files
        // u8 file_data[size] @ (offset * 0x800);
    }
} [[format_read("entry_formatter")]];
fn entry_formatter(auto e) {
    return "hello " + str(e.name);
};

struct Directory {
    u64 directory_offset = $;

    u32 entry_count;
    Entry<directory_offset> entries[entry_count];
};

RomHeader header @ 0x00;
Directory root @ 0x10;

Example dummy file: hello.rom.zip

P.S. Not sure if there's a better way to handle the DirOffset thing than with a non-type template parameter.. Is what I came up with a good idea?

P.P.S. Sometimes the strings I encounter are NOT utf-8 encoded (they are in Shift-JIS). Is there a way to specify the string encoding in pattern language?

[paxcut]

isnt directory_offset always 0x10? you don't need a template if the argument never changes. You can then create a pointer relative to 0x10 by offset that will point to the name. Try to avoid using @ inside structs if possible that creates a view of the data which doesnt advance the cursor. When using strings as arguments it may be necessary to use ref and make them references. The way you read the name assumes it is zero terminated. is that assumption correct? Finally you don't state what actually is displayed or any error messages shown. What actually happens when you try?

[DCNick3]

isnt directory_offset always 0x10

Sorry, my example file doesn't show this, but it does change if there are nested directories, like here: hello.rom.zip. Basically, the name offset is relative to the start of Directory structure.

Try to avoid using @ inside structs if possible that creates a view of the data which doesnt advance the cursor.

I am not sure how to implement reading of directory trees or entry names this way.. The offsets stored in the file require some math operations to actually get the real position in bytes (they are relative and/or require a multiplication). AFAIK, you can't really do that with pointers?

Also, what is the difference between pointers and just using @ in the structure. Isn't the "does advance the cursor" thing the whole point of a pointer?

The way you read the name assumes it is zero terminated. is that assumption correct?

Yes, the name is always zero-terminated and it is the only way you can determine its length.

Finally you don't state what actually is displayed or any error messages shown. What actually happens when you try?

Using e.name directly: the Value column is empty

Using str(e.name): Type error. Cannot cast value of type 'str' to type 'str'.

Using std::string::to_string: the Value column is empty

[paxcut]

yeah, don't worry about the posts on the offsets and the pointer. I was just curious as to why it needed a template. To clarify a pointer is a variable that stores an address. its value is read from the input file and the cursor is advanced. using @ is different because the data is read but the cursor does not advance so the next variable will read the same data again.

[paxcut]

this seems to work

fn entry_formatter(ref auto e) {
    return "hello " + e.name;
};

[DCNick3]

Aha, so I've needed ref. Can you please explain/refer me to the docs that explain why is it necessary? I saw the page mentioning that ref can be used to avoid unnecessary copying, but I don't really understand the semantics of the language to be able to infer how this helps with strings..

[paxcut]

The problem is that strings are not constant size structures. if you try to pass it by value the part that copies to data doesn't know how far to go. using a reference avoids the problem and this behavior is not documented afaik and only implied in the section that introduces references. This is something I had problems understanding when somebody said that refs needed to be used for strings based on their experience. When I pressed the issue I was told about dynamic sizes but nobody responded to my questions about the docs not mentioning it or if this was necessary or just recommended.

[DCNick3]

I guess this answers the core question. Thanks a lot!

[rigservice]

I have a format describing a filesystem image. To save on clicks, I want to see the filename of a directory entry directly in the "Value" column of the "Pattern Data" view. However, I just can't get it to convert the entry's name to be displayable. I've tried returning e.name directly, and converting it with either str or std::string::to_string...

Pattern code:

// necessary for large files
#pragma pattern_limit 0x200000

#include "type/magic.pat"
#include "std/string.pat"

struct RomHeader {
    type::Magic<"ROM "> magic;
    u32 version;
    u32 index_size;
    u32 unk;
};

bitfield NameAndFlags {
    name_offset : 31;
    is_directory : 1;
};

using Directory;

struct Entry<auto DirOffset> {
    NameAndFlags name_off [[color("FF0000")]];
    u32 offset [[color("00FF00")]];
    u32 size [[color("0000FF")]];
    
    char name[] @ (DirOffset + name_off.name_offset);
    
    if (name_off.is_directory) {
        if (name != ".\x00" && name != "..\x00") {
            Directory directory_data @ (offset * 0x10);
        }
    } else {
        // this creates too much patterns for large files
        // u8 file_data[size] @ (offset * 0x800);
    }
} [[format_read("entry_formatter")]];
fn entry_formatter(auto e) {
    return "hello " + str(e.name);
};

struct Directory {
    u64 directory_offset = $;

    u32 entry_count;
    Entry<directory_offset> entries[entry_count];
};

RomHeader header @ 0x00;
Directory root @ 0x10;

Example dummy file: hello.rom.zip

P.S. Not sure if there's a better way to handle the DirOffset thing than with a non-type template parameter.. Is what I came up with a good idea?

P.P.S. Sometimes the strings I encounter are NOT utf-8 encoded (they are in Shift-JIS). Is there a way to specify the string encoding in pattern language?