Regression in 5.1.7379-dev (f55a1eef) breaks propagation of return types when overriding a call type for calls inlined from a function

[WeiN76LQh]

Version and Platform (required):

• Binary Ninja Version: 5.1.7379-dev (f55a1eef)
• OS: macOS
• OS Version: 15.2
• CPU Architecture: M1

Bug Description:
Prior to the regression call type overrides applied via the API or through the UI as a user, would work as expected. However version 5.1.7379-dev kind of broke them and now calls from inlined functions do not propagate the return type of the applied call type override at the site where the function was inlined. This is specifically an issue with calls in functions that have been inlined. Overrides elsewhere seem to work fine.

Steps To Reproduce:
Please provide all steps required to reproduce the behavior:

1. Load a copy of DYLD Shared Cache.
2. Load a library like FeatureStore or anyone that makes calls to stub functions.
3. Find a call to a stub function. I've been testing with _objc_alloc.
4. Try to apply a call type override to it with a different return type and observe that the variable being assigned the result of the _objc_alloc doesn't change to use the return type of the call type override. You might need to be in MLIL to observe a variable assignment, often in HLIL the result of the call is passed directly to another call.
5. Now stop the function being inlined by going to j__objc_alloc and editing its function properties and turning off function inlining. I find often times toggling this setting doesn't work. Sometimes it requires multiple attempts but usually changing the function workflow from inherited to the same workflow just not the inherited option, at the same time as unticking the function inlining checkbox, will actually get the change to stick.
6. Try setting a call type override, with a different return type, to the j__objc_alloc call and observe that the variable being assigned the result will take on the return type of the call type override.

Expected Behavior:
Overridding the type of a call in an inlined function (at the location where its been inlined, not the actual function) should result in the return type being propagated to the variable its being assigned to.

Additional Information:

• This is not a problem for non-inlined calls (calls that aren't from inlined functions).
• In 5.1.7372-dev (c3488220) things worked as expected. I originally noticed this issue in the latest version (was 5.1.7399 at the time) and went back and found where the regression occurred so its still broken on latest.

[WeiN76LQh]

Actually I'm finding on latest (5.1.7400-dev (6eeff4a8)) even for non-inlined functions, the call type override is there (if I go to change it I see it has updated from being changed before), but the return type isn't being propagated to the variable its being assigned to. Manually re-applying the call type override as a user does nothing. I have to manually set the variable's type.

Although there seems to be some inconsistency because sometimes it does work for non-inlined calls.

[bdash]

Thanks for filing this! I had noticed myself that this wasn't working but was still in the "am I doing something wrong" phase so hadn't gotten around to writing up a bug report.

[WeiN76LQh]

I believe there is a related issue I'm also seeing in MLIL where direct assignments are not propagating types as I'd expect. I have a variable that is assigned a value and then used as an argument to a call. Thats its entire existence. Instead of taking on the type of the value it is being assigned it is choosing the type of the argument for the call it is passed to. If I override the call type, the variable will adjust its type according to the argument of the call type override.

I can kind of see the logic here but shouldn't the type of the assigned value take precedence? I made sure the value it was being assigned had a type specified by the user and that didn't help either. It will only ever take on the type of the call parameter unless I explicitly set the variable's type.

[WeiN76LQh]

Another thing I'm seeing, that may or may not be part of the issue, is that modifying IL, specifically MLIL to change the target of a call is not updating the call type. So I'll notice a variable which is assigned the return result of a modified MLIL call will have the type of the original call target's return type. If I then go to override the call type then that will also show the type of the original call target. If I then override the call type with the updated call target's type then the variable will end up with the expected return type.

Thinking this might be an unrelated issue that I'm just also observing now I modified my plugin to not always override call types. Also I'm unsure if this is occurring for all instances or just some.

[plafosse]

@emesare This seems to correspond to

commit f55a1eef2bb62212a934c01058b2a32faddbbcec
Author: Mason Reed <mason@vector35.com>
Date:   Sun May 4 21:20:32 2025 -0400

    Bump API (skip-note)

which corresponds with

commit 178887ba4d503838c444a0c976ac5538117fd2fb
Author: Mark Rowe <mrowe@bdash.net.nz>
Date:   Sat May 3 23:19:08 2025 -0700

    [Rust] Fix a pre-existing formatting issue
    
    CI is complaining about it.

commit 9bd1a1c5d54f46e250274538e8faee23d5520a88
Author: Mark Rowe <mrowe@bdash.net.nz>
Date:   Sat May 3 22:39:50 2025 -0700

    [Rust] Avoid leaking a reference within {Medium,High}LevelILFunction::ssa_form
    
    Return a Ref<_> so that the underlying core object will have its
    reference count decremented when the caller drops the object.

commit c9253993ee4085880468cf9345519f45adfa77a3
Author: Daniel Roethlisberger <daniel@roe.ch>
Date:   Sat May 3 14:00:57 2025 +0200

    [SharedCache] Avoid reading header fields outside of the bounds of the header
    
    Use mappingOffset as an upper bound for the header size, and avoid
    reading any header fields from beyond that offset.
    
    Co-authored-by: Mason Reed <mason@vector35.com>

I'm not sure if this is related IDK how it could be.