Under the Hood

Under the hood, SDK import is by a combination of tools:

  • HI2 provides high-level logic for importing SDKs and other related tasks. HI2 does not ship as a command line tool, as it is meant to be tweaked as needed for each new Xcode version, as a "work in progress", but its code is open source on GitHub at github.com/remobjects/HI2, so you can review what it does (and build, tweak and run it yourself, if you wish). The HI2 code is also what is embedded in Fire for the integrated SDK import functionality there.

  • HeaderImporter integrated into the compiler, does the actual processing of .h and related files, and generating .fx files. This codebase is also used fior Import Projects.

Simply put HI2 knows about how Cocoa SDKs are structured. It looks at Xcode, finds all the frameworks and core rtl files, includes and excludes the right parts, and creates in "port recipe" in form of a json file (and some additional flags). It then passes it on to HeaderImporter to do the dirty work.

There's a lof of knowledge ab out the SDKs hardcoded in HI2, and the code base is designed to be flexible, but in reality needs adjustments form most major new Xcode versions (say to handle new architectures, such as with Xcode 12). The Constants.pas source file gives a good overview of that. HI also knows what files from the core rtl to include and exclude (this is information arrived on by contonous tweaks, and cannot be inferred automatically), as well as what files (and what whole frameworks) to not import due to problems (for example, some frameworks include Objective-C++ code APIs, or bad headers).

For each version of Xcode, HI2 can import all five supported SDKs (macOS, iOS, tvOS, watchOS and Mac Catalyst. For iOS, tvOS and watchOS, the import is run twice, separately for device and simulator versions.

Importing an SDK

Importing an SDK consists of two main parts:

  • Importing the base rtl library (/usr/include)
  • Importing the actual named frameworks (/System/Library/Frameworks)

For rtl, HI2 has a fixed list of files to include, based on SDK type and architecture. These are defined in Constants.Darwin.RTL.pas. That list of files, as well as paths to /usr/include and /usr/lib are passed to HeaderImporter.

For the named frameworks, HI2 dynamically discovers all frameworks found in /System/Library/Frameworks for the current SDK – so newly added frameworks will be picked up automatically, as Apple adds them. As mentioned before, a hardcoded blacklist exists for frameworks that cannot be imported successfully, in Constants.Darwin.Blacklists.pas.

For each framework, HI2 collects a number of details:

  1. Most frameworks are Cocoa frameworks, written in Objective-C. For these, the main source for information comes from the .h files included in the frameworks, and the classes define therein will get imported as Cocoa Object Model classes, available to both Toffee and Island/Darwin projects.

  2. Some frameworks are pure Swift frameworks. For these, the core information comes from .swiftinterface files contained within. Import of Swift frameworks is experimental (not shipping yet), and the types found in the will be Swift Object Model types, available only on Island/Darwin

  3. In addition /usr/lib/swift includes additional "shadow frameworks" with Swift-specific addendums to Cocoa frameworks. These will be imported and also be available on Island/Darwin only.

API Notes

"API Notes" files provide the "Grand Rename" mapping that turn the beautiful Cocoa API names into cryptic unreadable names seen by the Swift language.

HI2 will locate .apinotes for the base rtl in /usr/include/swift, as well as within the named frameworks, where present. API Notes will import alternative/optional names, and both real and mapped names will be available to user code, in both languages. in Swift source files, Code Completion will favor to showing renamed APIs, while in all other languages will will show the original and more expressive Cocoa names.

Finally, the Import

With all this information gathered, HI2 passes off the import to the core header importer, as a .json file and additional parameters.

A typical command line looks like this:

HeaderImporter.exe
import 
-o ".../Toffee/macOS 11.0/arm64"
--json=.../import-Toffee-macOS-11.0-arm64.json
--sdk=/Applications/Xcode-12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk
--toolchain=/Applications/Xcode-12.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain
--libpath=/Applications/Xcode-12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/usr/lib
-i /Applications/Xcode-12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/usr/include
-f /Applications/Xcode-12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/System/Library/Frameworks
-i /Users/mh/Code/Elements/Frameworks/Toffee

and the (trimmed for readability) Json file:

{
  "TargetString": "arm64-apple-macosx",
  "Version": "11.0",
  "SDKVersionString": "11.0",
  "Imports": [
    ...
    {
      "Name": "Accelerate",
      "Framework": true,
      "Prefix": "",
      "FrameworkPath": "{sdk}/System/Library/Frameworks/Accelerate.framework",
      "APINotes": [
        "{sdk}/System/Library/Frameworks/Accelerate.framework/Headers/Accelerate.apinotes"
      ]
    },
    ...
    {
      "Name": "SwiftUI",
      "Framework": true,
      "Prefix": "",
      "FrameworkPath": "{sdk}/System/Library/Frameworks/SwiftUI.framework",
      "Swift": true,
      "SwiftModule": "{sdk}/System/Library/Frameworks/SwiftUI.framework/Modules/SwiftUI.swiftmodule",
      "SwiftInterface": "{sdk}/System/Library/Frameworks/SwiftUI.framework/Modules/SwiftUI.swiftmodule/arm64.swiftinterface"
    },
    ...
    {
      "Name": "rtl",
      "Framework": false,
      "Prefix": "",
      "Core": true,
      "ForceNamespace": "rtl",
      "DropPrefixes": [
        "NS"
      ],
      "Files": [
        "assert.h",
        ...
        "xar/*.h"
      ],
      "IndirectFiles": [
        "_wctype.h",
        ...
        "os/*.h"
      ],
      "ImportDefs": [
        {
          "Name": "dyld_stub_binder",
          "Library": "/usr/lib/libSystem.B.dylib",
          "Version": "81395766,65536"
        },
        ...
      ],
      "APINotes": [
        "{sdk}/usr/include/Darwin.apinotes",
        "{sdk}/usr/include/objc/ObjectiveC.apinotes",
        "{sdk}/usr/include/xpc/XPC.apinotes",
        "{toolchain}/usr/lib/swift/apinotes/Dispatch.apinotes",
        "{toolchain}/usr/lib/swift/apinotes/os.apinotes"
      ]
    }
  ],
  "Defines": [
    "__arm__",
    "DARWIN",
    ...
    "__ELEMENTS",
    "__TOFFEE__",
  ],
  "Blacklist": [
    "sys/_symbol_aliasing.h",
    ...
    "Foundation/FoundationLegacySwiftCompatibility.h"
  ],
  "Platform": "macOS",
  "SDKName": "macOS",
  "Island": false,
  "OverrideNamespace": [
    {
      "Key": "objc/NSObject.h",
      "Value": "Foundation"
    },
    {
      "Key": "objc/NSObjCRuntime.h",
      "Value": "Foundation"
    }
  ],
  "VirtualFiles": {
    "os/availibility.h": "#include <os/availability.h>"
  }
}