Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

class-dump XCFramework only creates library for ios-arm64_x86_64-simulator #409

Open
2 tasks done
Meowcat285 opened this issue Feb 26, 2024 · 26 comments
Open
2 tasks done
Assignees
Labels
bug Something isn't working

Comments

@Meowcat285
Copy link

What happened?

class-dump's XCFramework feature only creates the library for ios-arm64_x86_64-simulator and not for other platforms like macOS

How can we reproduce this?

Use class-dump's XCFramework on a dylib from macOS and see that it doesn't output a library for other platforms

ipsw version

Version: 3.1.455, BuildTime: 2024-02-22T19:12:37Z

Search

  • I did search for other open and closed issues before opening this

Code of Conduct

  • I agree to follow this project's Code of Conduct

Additional context

No response

@t0rr3sp3dr0
Copy link
Contributor

t0rr3sp3dr0 commented Apr 13, 2024

I see two alternatives here:

  1. Generate XCFW only for the platforms and architectures of the source DSC/Mach-O

    1. Pros: Makes the generated framework 100% reliable for that specific DSC/Mach-O.
    2. Cons: Increases the amount of work necessary to build a universal XCFW, given that DSC's/Mach-O's from all platforms and architectures would need to be provided to generate such XCFW.
  2. Generate a generic XCFW supporting all platforms and architectures, regardless of the source DSC/Mach-O

    1. Pros: Minimal work required to build a universal framework, given that the DSC/Mach-O of only one platform and architecture needs to be provided.
    2. Cons: Programs built with a generic XCFW may find that symbols are not available on that platform or architecture during runtime and crash.

As a general note, I would add that when dealing with Private Frameworks, there is no guarantee that an XCFW will be portable between architectures, platforms, or different OS versions. You are only guaranteed to have 100% reliability when generating an XCFW in the same system you intend to use it, while being aware that OS updates may break it.

@t0rr3sp3dr0
Copy link
Contributor

I think we can provide both solutions by adding an extra flag to ipsw class-dump. Maybe --xcfw for option 1 and --xcfw-generic for option 2.

WDYT, @blacktop?

@blacktop
Copy link
Owner

blacktop commented Apr 16, 2024

ya I always like to lean on the side of less flags (as I've found that people don't really read the docs/help 😏 but for this weird edge case a flag might make the most sense for those that accept the possible errors of a generic tbd

but as I mentioned in your PR, that we could chose the correct platform subset based on the ARCH/platform of the DSC for the non-generic case, but not 💯 about old DSCs

@t0rr3sp3dr0
Copy link
Contributor

I'll take a look at this in the weekend

@blacktop blacktop removed the triage label Apr 27, 2024
@blacktop
Copy link
Owner

@Meowcat285 @t0rr3sp3dr0 do you know what the structure of an XCFramework would be that supports macOS as well? Is it a separate folder or can I just append macos to the from of ios-arm64_x86_64-simulator ?

@blacktop
Copy link
Owner

So I just pushed out a release that I "believe" creates much more accurate XCFrameworks, however, this also means that it will only make them for the LibraryIdentifier that the source is, meaning a DSC from macOS will make a macos-ARCH and iOS DSC will make an ios_arm etc, so to make a XCFramework that you need you'd need a DSC for that same 'target'

We can work on adding 'generic' XCFrameworks too, I just think there's an edge case where the class-dump headers and the .tbd symbols wouldn't match.

You can see in the libsystem.B.dylib.tbd that it has target arch break downs where if it's a fat/universal macho it'll have different symbols for the different arches etc

This is a pretty complex issue 😩

I STILL think I need more examples of good XCFrameworks to use to make sure I'm filling out all the required Info.plist fields etc.

@blacktop
Copy link
Owner

I think to make a BIG generic XCFramework we'd need to supply it with the DSCes from an macOS IPSW which would have x86_64, x86_64h and arm64 AND the corresponding iOS version to get the arm64, arm64e AND maybe a watchOS, tvOS and visionOS ?? there's also the Runtimes to get the _simulator variants 🤮

@blacktop
Copy link
Owner

Please at least try and use the output and let me know if it actually compiles and is useful and has code-completions when calling into the private ObjC funcs etc

@NSAntoine if you have any time I'd love your input as well 🙏

@NSAntoine
Copy link

I think to make a BIG generic XCFramework we'd need to supply it with the DSCes from an macOS IPSW which would have x86_64, x86_64h and arm64 AND the corresponding iOS version to get the arm64, arm64e AND maybe a watchOS, tvOS and visionOS ?? there's also the Runtimes to get the _simulator variants 🤮

The entire DSC? or just the binary of the specified lib/framework? can't you extract just one specific lib from a DSC?

@blacktop
Copy link
Owner

blacktop commented Apr 28, 2024

I think to make a BIG generic XCFramework we'd need to supply it with the DSCes from an macOS IPSW which would have x86_64, x86_64h and arm64 AND the corresponding iOS version to get the arm64, arm64e AND maybe a watchOS, tvOS and visionOS ?? there's also the Runtimes to get the _simulator variants 🤮

The entire DSC? or just the binary of the specified lib/framework? can't you extract just one specific lib from a DSC?

Ya, 'extracting' from the DSC is sooooooooo 2019 😏 now we just analyze the DSC as a whole.

But my comment was that is the output product that XCFramework consumer REALLY want is a generic one that supports all teh things, then the input to ipsw would prob have to be the macOS IPSW and the iOS IPSW (or at the very least their DSCs) I "think" we could lie and say it also supports the _simulator but I know that where you actually get those FW bins are in the XCode runtimes (i.e. iOS_17.5_beta_2_Simulator_Runtime.dmg)

@NSAntoine
Copy link

I think to make a BIG generic XCFramework we'd need to supply it with the DSCes from an macOS IPSW which would have x86_64, x86_64h and arm64 AND the corresponding iOS version to get the arm64, arm64e AND maybe a watchOS, tvOS and visionOS ?? there's also the Runtimes to get the _simulator variants 🤮

The entire DSC? or just the binary of the specified lib/framework? can't you extract just one specific lib from a DSC?

Ya, 'extracting' from the DSC is sooooooooo 2019 😏 now we just analyze the DSC as a whole.

But my comment was that is the output product that XCFramework consumer REALLY want is a generic one that supports all teh things, then the input to ipsw would prob have to be the macOS IPSW and the iOS IPSW (or at the very least their DSCs) I "think" we could lie and say it also supports the _simulator but I know that where you actually get those FW bins are in the XCode runtimes (i.e. iOS_17.5_beta_2_Simulator_Runtime.dmg)

Lying about that sounds like a bad idea in theory, I think an option to specify the targets would be best here

@blacktop
Copy link
Owner

@NSAntoine have you tried the ipsw class-dump DSC CoreUI --xcfw --output /tmp yet? I'd be curious if it works, needs fixes/tweaks?

@blacktop
Copy link
Owner

The next ipsw feature we can finish once this one is done is ipsw class-dump --spm which would also gen the Package.swift and anything else needed etc

@NSAntoine
Copy link

@NSAntoine have you tried the ipsw class-dump DSC CoreUI --xcfw --output /tmp yet? I'd be curious if it works, needs fixes/tweaks?

Haven't, been really really busy w studying, I could after ~may 16

@NSAntoine
Copy link

The next ipsw feature we can finish once this one is done is ipsw class-dump --spm which would also gen the Package.swift and anything else needed etc

package.swift generation??

@blacktop
Copy link
Owner

@NSAntoine have you tried the ipsw class-dump DSC CoreUI --xcfw --output /tmp yet? I'd be curious if it works, needs fixes/tweaks?

Haven't, been really really busy w studying, I could after ~may 16

Ya I understand and there's no real rush. tx ❤️

@blacktop
Copy link
Owner

blacktop commented Apr 28, 2024

The next ipsw feature we can finish once this one is done is ipsw class-dump --spm which would also gen the Package.swift and anything else needed etc

package.swift generation??

ya I supposed there's REALLY nothing to it other than filling out a few fields, with the XCFramework being the .binaryTarget etc, but thought it'd be a nice-to-have for those that have 0 experience w/ SPM etc

My idea is that I would create a NEW github org and it's output would be SPMs for the popular FWs so people could just import then and use them like any other Swift pkg?

@t0rr3sp3dr0
Copy link
Contributor

@Meowcat285 @t0rr3sp3dr0 do you know what the structure of an XCFramework would be that supports macOS as well? Is it a separate folder or can I just append macos to the from of ios-arm64_x86_64-simulator ?

I don't know if this is the recommended way, but take a look at ./lib/mdk.xcframework in this archive: https://github.com/wang-bin/mdk-sdk/releases/download/v0.26.0/mdk-sdk-apple.zip

mdk.xcframework
├── Info.plist
├── _CodeSignature
│   └── ...
├── ios-arm64
│   └── mdk.framework
│       └── ...
├── ios-arm64_x86_64-maccatalyst
│   └── mdk.framework
│       └── ...
├── ios-arm64_x86_64-simulator
│   └── mdk.framework
│       └── ...
├── macos-arm64_x86_64
│   └── mdk.framework
│       └── ...
├── tvos-arm64
│   └── mdk.framework
│       └── ...
├── tvos-arm64_x86_64-simulator
│   └── mdk.framework
│       └── ...
├── xros-arm64
│   └── mdk.framework
│       └── ...
└── xros-arm64_x86_64-simulator
    └── mdk.framework
        └── ...

@t0rr3sp3dr0
Copy link
Contributor

Please at least try and use the output and let me know if it actually compiles and is useful and has code-completions when calling into the private ObjC funcs etc

I'll try the new version during this week and let you know

@t0rr3sp3dr0
Copy link
Contributor

My idea is that I would create a NEW github org and it's output would be SPMs for the popular FWs so people could just import then and use them like any other Swift pkg?

That would be amazing. I'm personally using these XCFWs with SwiftPM and would love not have to generate them manually and store them together with the source code.

@t0rr3sp3dr0
Copy link
Contributor

About the generic XCFW, I now believe that's not a good idea unless you provide the DSC for all architectures and systems.

I was debugging the Objective-C type encoding in go-macho and found out that BOOL is signed char (encoded as c) on Intel Macs while it is _Bool (encoded as B) in Apple Silicon and PowerPC Macs. https://developer.apple.com/documentation/objectivec/bool#discussion

I also learned that long double is actually just double (64-bit Float) in Apple Silicon but it gets encoded as D (128-bit Float). I know that affects struct sizes and method signatures, but not sure if that is a problem for our purpose here. https://developer.apple.com/documentation/xcode/writing-arm64-code-for-apple-platforms#Handle-data-types-and-data-alignment-properly

Similar to double and long double, long and unsigned long also vary depending on the architecture. They are encoded as l and L in 32-bit architectures and as q and Q in 64-bit architectures. https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Articles/ocrtTypeEncodings.html

So if we have this class:

@implementation C : NSObject

- (void)b:(BOOL)b {
}

- (void)l:(long)l {
}

- (void)ld:(long double)ld {
}

@end

We may decompile it to this:

@interface C : NSObject

- (void)b:(signed char)b; // v20@0:8c16

- (void)l:(long)l; // v20@0:8l16

- (void)ld:(long double)ld; // v32@0:8D16

@end

Or this:

@implementation C : NSObject

- (void)b:(_Bool)b; // v20@0:8B16

- (void)l:(long)l; // v24@0:8q16

- (void)ld:(long double)ld; // v24@0:8D16

@end

@t0rr3sp3dr0
Copy link
Contributor

t0rr3sp3dr0 commented May 8, 2024

@blacktop, I just tried the new version, the TBD generation based on the specific system and architecture of the DSC is working perfectly. But there are still some problems with the XCFW generation.

Using CoreFoundation as an example:

  • Somehow CoreFoundation has both regular macOS and Mac Catalyst. I have no idea how this is compiled, given that xcodebuild archive is unable to handle multiple platforms at once and that there is no way to describe this in a XCFramework. So I'm assuming we have one binary for each platform.
  1. It generated an invalid Info.plist. x86_64 is listed twice in SupportedArchitectures, SupportedPlatform has the wrong value, SupportedPlatformVariant is missing, and LibraryIdentifier doesn't follow pattern used by Xcode.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>AvailableLibraries</key>
        <array>
            <dict>
                <key>BinaryPath</key>
                <string>CoreFoundation.framework/CoreFoundation.tbd</string>
                <key>LibraryIdentifier</key>
                <string>macos_x86_64maccatalyst_x86_64</string>
                <key>LibraryPath</key>
                <string>CoreFoundation.framework</string>
                <key>SupportedArchitectures</key>
                <array>
                    <string>x86_64</string>
                    <string>x86_64</string>
                </array>
                <key>SupportedPlatform</key>
                <string>maccatalyst</string>
            </dict>
        </array>
        <key>CFBundlePackageType</key>
        <string>XFWK</string>
        <key>XCFrameworkFormatVersion</key>
        <string>1.0</string>
    </dict>
</plist>

Fixing everything, we should have this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>AvailableLibraries</key>
	<array>
		<dict>
			<key>BinaryPath</key>
			<string>CoreFoundation.framework/CoreFoundation.tbd</string>
			<key>LibraryIdentifier</key>
			<string>ios-x86_64-maccatalyst</string>
			<key>LibraryPath</key>
			<string>CoreFoundation.framework</string>
			<key>SupportedArchitectures</key>
			<array>
				<string>x86_64</string>
			</array>
			<key>SupportedPlatform</key>
			<string>ios</string>
			<key>SupportedPlatformVariant</key>
			<string>maccatalyst</string>
		</dict>
		<dict>
			<key>BinaryPath</key>
			<string>CoreFoundation.framework/CoreFoundation.tbd</string>
			<key>LibraryIdentifier</key>
			<string>macos-x86_64</string>
			<key>LibraryPath</key>
			<string>CoreFoundation.framework</string>
			<key>SupportedArchitectures</key>
			<array>
				<string>x86_64</string>
			</array>
			<key>SupportedPlatform</key>
			<string>macos</string>
		</dict>
	</array>
	<key>CFBundlePackageType</key>
	<string>XFWK</string>
	<key>XCFrameworkFormatVersion</key>
	<string>1.0</string>
</dict>
</plist>

And if we had a universal Mach-O, it would this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>AvailableLibraries</key>
	<array>
		<dict>
			<key>BinaryPath</key>
			<string>CoreFoundation.framework/CoreFoundation.tbd</string>
			<key>LibraryIdentifier</key>
			<string>ios-arm64_x86_64-maccatalyst</string>
			<key>LibraryPath</key>
			<string>CoreFoundation.framework</string>
			<key>SupportedArchitectures</key>
			<array>
				<string>arm64</string>
				<string>x86_64</string>
			</array>
			<key>SupportedPlatform</key>
			<string>ios</string>
			<key>SupportedPlatformVariant</key>
			<string>maccatalyst</string>
		</dict>
		<dict>
			<key>BinaryPath</key>
			<string>CoreFoundation.framework/CoreFoundation.tbd</string>
			<key>LibraryIdentifier</key>
			<string>macos-arm64_x86_64</string>
			<key>LibraryPath</key>
			<string>CoreFoundation.framework</string>
			<key>SupportedArchitectures</key>
			<array>
				<string>arm64</string>
				<string>x86_64</string>
			</array>
			<key>SupportedPlatform</key>
			<string>macos</string>
		</dict>
	</array>
	<key>CFBundlePackageType</key>
	<string>XFWK</string>
	<key>XCFrameworkFormatVersion</key>
	<string>1.0</string>
</dict>
</plist>
  1. As mentioned above, LibraryIdentifier is wrong and this is the name of the directory containing the Framework. In the case of CoreFoundation, it should be macos-x86_64 and ios-x86_64-maccatalyst. Or ios-arm64_x86_64-maccatalyst and macos-arm64_x86_64 in case of a universal binary. Never macos_x86_64maccatalyst_x86_64

  2. The headers are also in the wrong path. Currently they are located at CoreFoundation.framework/Headers/CoreFoundation, but the correct path is just CoreFoundation.framework/Headers. Otherwise, you would have to import the library with #import < CoreFoundation/CoreFoundation/CoreFoundation.h>.

  3. The contents of module.modulemap are not correct for a Framework.

module CoreFoundation [system] {
    header "Headers/CoreFoundation.h"
    export *
}

I expected this instead:

framework module CoreFoundation [system] {
  umbrella header "CoreFoundation.h"
  export *

  module * { export * }
}

Note that we can only assume the module to be a system module if we get it from a DSC (at least I think, not sure if there is some info about this in the Mach-O). Modules for regular binaries should be like this:

framework module CoreFoundation {
  umbrella header "CoreFoundation.h"
  export *

  module * { export * }
}
  1. There shouldn't be a Info.plist inside the Framework, just the XCFramework has that file. So we should delete CoreFoundation.framework/Info.plist.

  2. ipsw cd -x is panicking for any Mach-O that isn't from a DSC. This should be usable with non-system binaries. Even just taking into consideration system binaries, in the latest version of macOS, we still have some stuff on disk and not included in the DSC.

% ipsw cd -x /System/Library/PrivateFrameworks/CoreFP.framework/Versions/A/CoreFP
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x104709692]

goroutine 1 [running]:
github.com/blacktop/ipsw/pkg/dyld.(*File).GetDylibIndex(0x0, {0x7ff7bc4a9b4d, 0x6})
	/Users/pedro/Documents/blacktop/ipsw/pkg/dyld/closure.go:1020 +0x32
github.com/blacktop/ipsw/pkg/dyld.(*File).Image(0x0, {0x7ff7bc4a9b4d, 0x6})
	/Users/pedro/Documents/blacktop/ipsw/pkg/dyld/file.go:1717 +0x37
github.com/blacktop/ipsw/internal/commands/macho.(*ObjC).XCFramework(0xc000040180)
	/Users/pedro/Documents/blacktop/ipsw/internal/commands/macho/objc.go:726 +0x173
github.com/blacktop/ipsw/cmd/ipsw/cmd.init.func22(0xc0007a9a00?, {0xc0000fa280, 0x1, 0x104b4c63f?})
	/Users/pedro/Documents/blacktop/ipsw/cmd/ipsw/cmd/class_dump.go:245 +0xe71
github.com/spf13/cobra.(*Command).execute(0x105ddc6c0, {0xc0000fa260, 0x2, 0x2})
	/Users/pedro/go/pkg/mod/github.com/spf13/[email protected]/command.go:983 +0xaca
github.com/spf13/cobra.(*Command).ExecuteC(0x105ddc9a0)
	/Users/pedro/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
	/Users/pedro/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039
github.com/blacktop/ipsw/cmd/ipsw/cmd.Execute()
	/Users/pedro/Documents/blacktop/ipsw/cmd/ipsw/cmd/root.go:67 +0x1a
main.main()
	/Users/pedro/Documents/blacktop/ipsw/cmd/ipsw/main.go:27 +0xf

@blacktop
Copy link
Owner

blacktop commented May 9, 2024

@t0rr3sp3dr0 thank you for this great analysis. Just fyi I'm traveling a lot recently so I'll be slow to respond for a while.

@t0rr3sp3dr0
Copy link
Contributor

Quick note, if you do something like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>AvailableLibraries</key>
        <array>
            <dict>
                <key>BinaryPath</key>
                <string>FridaGum.framework/FridaGum</string>
                <key>LibraryIdentifier</key>
                <string>macos-arm64</string>
                <key>LibraryPath</key>
                <string>FridaGum.framework</string>
                <key>SupportedArchitectures</key>
                <array>
                    <string>arm64</string>
                </array>
                <key>SupportedPlatform</key>
                <string>macos</string>
            </dict>
            <dict>
                <key>BinaryPath</key>
                <string>FridaGum.framework/FridaGum</string>
                <key>LibraryIdentifier</key>
                <string>macos-x86_64</string>
                <key>LibraryPath</key>
                <string>FridaGum.framework</string>
                <key>SupportedArchitectures</key>
                <array>
                    <string>x86_64</string>
                </array>
                <key>SupportedPlatform</key>
                <string>macos</string>
            </dict>
        </array>
        <key>CFBundlePackageType</key>
        <string>XFWK</string>
        <key>XCFrameworkFormatVersion</key>
        <string>1.0</string>
    </dict>
</plist>

Xcode will fail:

error: Failed to load XCFramework at 'FridaGum.xcframework': Both 'macos-arm64' and 'macos-x86_64' represent two equivalent library definitions.

So SupportedPlatform + SupportedPlatformVariant work as a primary key. This means that, if we want to support providing multiple DSC to generate a single XCFW, we need to produce a single TDB that is the intersection of the TDBs of each architecture.

@blacktop
Copy link
Owner

Congrats on your CVE @t0rr3sp3dr0 🎉

@t0rr3sp3dr0
Copy link
Contributor

Thanks! 😁

You gave me the news even before Apple did, lol. Their communication has been spotty at best.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants