Importing Cocoa SDKs with HI2

SDK import is done by a combination of tools:

• HI2 provides high-level logic for importing SDKs and other related tasks. HI2 does not ship as a command line tool, as it is meant to be tweaked as needed for each new Xcode version, as a "work in progress", but its code is open source on GitHub at github.com/remobjects/HI2, so you can review what it does (and build, tweak and run it yourself, if you wish). The HI2 code is also what used to be embedded in Fire for the integrated SDK import functionality there.

• HeaderImporter, integrated into the compiler, does the actual processing of .h and related files, and generating of .fx files. This codebase is also used for Import Projects.

Simply put, HI2 knows about how Cocoa SDKs are structured. It looks at Xcode, finds all the SDKs, their frameworks and core rtl files, includes and excludes the right parts, and creates an "import recipe" in form of a .json file (and some additional flags). It then passes that on to HeaderImporter to do the dirty work.

There's a lof of knowledge about the SDKs hardcoded in HI2, and the code base is designed to be flexible, but in reality needs adjustments for most major new Xcode versions (say to handle new architectures, such as with Xcode 12). The Constants.pas source file gives a good overview of that. HI2 also knows what files from the core rtl to include and exclude (this is information arrived on by continous tweaks, and cannot be inferred automatically), as well as what files (and what whole frameworks) to not import due to problems (for example, some frameworks include Objective-C++ code APIs, or bad headers).

For each version of Xcode, HI2 can import all five supported SDKs (macOS, iOS, tvOS, watchOS and Mac Catalyst. For iOS, tvOS and watchOS, the import is run twice, separately for device and simulator versions.

Given the path to an Xcode version, HI2 automatically finds what versions of SDKs are included.

Running HI2 yourself

As mentioned above, HI2 is not mean to be a read-to-use tool you just run, it is meant to have its source tweaked to the right options, and then run form the IDE. Unless specified differenly, HI2 looks for Xcode versions in ~/Applications (if that folder exists) or in /Applications, if the former does not.

Before you run HI2, you will want to open the ./Importer.Darwin.Tasks.pas source file and change the ImportCurrentXcode method to point to the version of Xcode you want to import. For this, the Xcode app bumdle must be renamed to Xcode-${Version)-Beta(BetaNumber).app, e.g. Xcode-13.4.app or Xcode-14.0-Beta3.

You might also want to tweak the ImportSDKs method to enable/disable certain aspects of the import (by default, all SDKs will be imported).

Once ready, hit "Run" (**⌘R) to build and run HI2.

How the Import Works

Importing an SDK consists of two main parts:

• Importing the base rtl library (./usr/include)
• Importing the actual named frameworks (./System/Library/Frameworks)

For rtl, HI2 has a fixed list of files to include, based on SDK type and architecture. These are defined in Constants.Darwin.RTL.pas. That list of files, as well as paths to ./usr/include and ./usr/lib are passed to HeaderImporter.

For the named frameworks, HI2 dynamically discovers all frameworks found in ./System/Library/Frameworks for the current SDK – so newly added frameworks will be picked up automatically, as Apple adds them. As mentioned before, a hardcoded blacklist exists for frameworks that cannot be imported successfully, in Constants.Darwin.Blacklists.pas.

For each framework, HI2 collects a number of details:

1. Most frameworks are Cocoa frameworks, written in Objective-C. For these, the main source for information comes from the .h files included in the frameworks, and the classes defined therein will get imported as Cocoa Object Model classes, available to both Toffee and Island/Darwin projects.

2. Some frameworks are pure Swift frameworks. For these, the core information comes from .swiftinterface files contained within. Import of Swift frameworks is experimental (not shipping yet), and the imported types will be Swift Object Model types, available only on Island/Darwin.

3. In addition, ./usr/lib/swift includes additional "shadow frameworks" with Swift-specific addendums to Cocoa frameworks. These will be imported and also be available on Island/Darwin only.

API Notes

"API Notes" files provide the "Grand Rename" mapping that turn the beautiful Cocoa API names into cryptic unreadable names seen by the Swift language.

HI2 will locate .apinotes for the base rtl in /usr/include/swift, as well as within the named frameworks, where present. API Notes will import alternative/optional names, and both real and mapped names will be available to user code, in both languages. In Swift source files, Code Completion will favor to showing renamed APIs, while in all other languages it will show the original and more expressive Cocoa names.

Finally, the Import

With all this information gathered, HI2 passes off the import to the core header importer, as a .json file and additional parameters.

A typical command line looks like this:

HeaderImporter.exe
import 
-o ".../Toffee/macOS 11.0/arm64"
--json=.../import-Toffee-macOS-11.0-arm64.json
--sdk=/Applications/Xcode-12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk
--toolchain=/Applications/Xcode-12.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain
--libpath=/Applications/Xcode-12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/usr/lib
-i /Applications/Xcode-12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/usr/include
-f /Applications/Xcode-12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/System/Library/Frameworks
-i /Users/mh/Code/Elements/Frameworks/Toffee

and the (trimmed for readability) Json file:

{
  "TargetString": "arm64-apple-macosx",
  "Version": "11.0",
  "SDKVersionString": "11.0",
  "Imports": [
    ...
    {
      "Name": "Accelerate",
      "Framework": true,
      "Prefix": "",
      "FrameworkPath": "{sdk}/System/Library/Frameworks/Accelerate.framework",
      "APINotes": [
        "{sdk}/System/Library/Frameworks/Accelerate.framework/Headers/Accelerate.apinotes"
      ]
    },
    ...
    {
      "Name": "SwiftUI",
      "Framework": true,
      "Prefix": "",
      "FrameworkPath": "{sdk}/System/Library/Frameworks/SwiftUI.framework",
      "Swift": true,
      "SwiftModule": "{sdk}/System/Library/Frameworks/SwiftUI.framework/Modules/SwiftUI.swiftmodule",
      "SwiftInterface": "{sdk}/System/Library/Frameworks/SwiftUI.framework/Modules/SwiftUI.swiftmodule/arm64.swiftinterface"
    },
    ...
    {
      "Name": "rtl",
      "Framework": false,
      "Prefix": "",
      "Core": true,
      "ForceNamespace": "rtl",
      "DropPrefixes": [
        "NS"
      ],
      "Files": [
        "assert.h",
        ...
        "xar/*.h"
      ],
      "IndirectFiles": [
        "_wctype.h",
        ...
        "os/*.h"
      ],
      "ImportDefs": [
        {
          "Name": "dyld_stub_binder",
          "Library": "/usr/lib/libSystem.B.dylib",
          "Version": "81395766,65536"
        },
        ...
      ],
      "APINotes": [
        "{sdk}/usr/include/Darwin.apinotes",
        "{sdk}/usr/include/objc/ObjectiveC.apinotes",
        "{sdk}/usr/include/xpc/XPC.apinotes",
        "{toolchain}/usr/lib/swift/apinotes/Dispatch.apinotes",
        "{toolchain}/usr/lib/swift/apinotes/os.apinotes"
      ]
    }
  ],
  "Defines": [
    "__arm__",
    "DARWIN",
    ...
    "__ELEMENTS",
    "__TOFFEE__",
  ],
  "Blacklist": [
    "sys/_symbol_aliasing.h",
    ...
    "Foundation/FoundationLegacySwiftCompatibility.h"
  ],
  "Platform": "macOS",
  "SDKName": "macOS",
  "Island": false,
  "OverrideNamespace": [
    {
      "Key": "objc/NSObject.h",
      "Value": "Foundation"
    },
    {
      "Key": "objc/NSObjCRuntime.h",
      "Value": "Foundation"
    }
  ],
  "VirtualFiles": {
    "os/availibility.h": "#include <os/availability.h>"
  }
}