Back to Blog Home
← all posts

What happens when a SwiftUI and NativeScript developer embrace each other? visionOS Goodies!

January 24, 2025 — by Ben Ormos and Nathan Walker

In the ever-evolving world of app development, the journey to innovation often involves crossing paths with different programming paradigms, tools, and mindsets. Recently, we embarked on a unique adventure: building a visionOS app by combining the strengths of SwiftUI and NativeScript. Here's a timeline of how it all unfolded, including lessons we learned along the way and a sharing of some goodies you can take with you on your journey.

Step 1: Building the visionOS app

A SwiftUI developer, Ben Ormos, began building a visionOS app from scratch using SwiftUI. This allowed myself (NathanWalker), a fullstack developer with NativeScript understanding, and Ben to learn how the visionOS ecosystem works aided by Apple's developer portal, template apps, and WWDC demos and samples. These resources provided:

  • Insights into RealityKit and ARKit: Key frameworks for creating immersive 3D environments.
  • Guidance on AR Tracking: Examples showcasing world tracking, hand tracking, and plane tracking.
  • Best Practices for Spatial UI: Since Apple's XR platform was quite new when we started, it was useful to rely on Apple's guidelines on how the user interactions and how the user interface should look like.

Having acquired this knowledge we could build out a fully functional SwiftUI-based visionOS app, capable of AR tracking and 3D interactions.

Step 2: Integrating the visionOS app into a NativeScript Project

The next step was integrating the visionOS app into an existing larger NativeScript project. The NativeScript project was well architected with great composability, state management and loads of features with sophisticated hardware interactions. Considerations during this phase included:

  • Data Flow: Establishing seamless data flow between SwiftUI and NativeScript.
  • Immersive scene lifecycle: Handling critical immersive session background/foreground behavior.
  • SwiftUI Developer keeping same DX in Xcode: Allowing the SwiftUI developer to continue feeling at home in Xcode if they desired.

But how to take an existing visionOS app and use it within a NativeScript project, especially without affecting a SwiftUI developers preferred Xcode experience?

This was a pivotal moment where the realization of endless possibilities elevated.

A blog post was shared in 2024 demonstrating adding visionOS to an existing NativeScript app, which covers good fundamental steps but let's look at some additional things.

Using the @nativescript/swift-ui plugin and putting the SwiftUI files into place backed by a provider will get you a lot to work with. Let's briefly walk through doing just that and more.

Start by creating a visionOS app with Xcode. We'll make this starter template work exactly as shown in Xcode from NativeScript.

visionOS Xcode starter app

Incorporate Swift files into a NativeScript project

Any Swift files inside App_Resources/visionOS/src will be automatically incorporated into the Xcode project for NativeScript when ns run|debug is run for the given platform (eg, ns run visionos --no-hmr).

But what about the @main app protocol from Xcode?

That would be this file from the Xcode visionOS starter template: visionOS Xcode starter app protocol

Every NativeScript visionOS app already contains a @main App Protocol file: visionOS NativeScript starter app protocol

We definitely don't need both. There should only be one @main App protocol for the boostrap of our visionOS app. Given NativeScriptMainWindow is already our WindowGroup, we just want to use @nativescript/swift-ui to allow us to use the ContentView from the Xcode starter template.

We can do this by adding a provider as discussed here for the ContentView which allows us to include it any NativeScript view we need. We can move Xcode's ContentView.swift to App_Resources/visionOS/src/ContentView.swift and add a provider for it:

@objc class ContentViewProvider: UIViewController, SwiftUIProvider {
    private var swiftUI: ContentView?
    
    required init?(coder aDecoder: NSCoder) {
        super.init(coder: aDecoder)
    }
    
    required public init() {
        super.init(nibName: nil, bundle: nil)
    }
    
    public override func viewDidLoad() {
        super.viewDidLoad()
    }
    
    /// Receive data from NativeScript
    func updateData(data: NSDictionary) {
        
        if (self.swiftUI == nil) {
            swiftUI = ContentView()
            setupSwiftUIView(content: swiftUI)
        }
    }
    
    /// Send data to NativeScript
    var onEvent: ((NSDictionary) -> ())?
}

Now just register it for use with our NativeScript project:

import { SwiftUI, UIDataDriver, registerSwiftUI } from '@nativescript/swift-ui'

registerSwiftUI(
  'contentView',
  (view) => new UIDataDriver(ContentViewProvider.alloc().init(), view)
)

// each flavor of NativeScript has their own similar API to register custom elements
registerElement('SwiftUI', () => SwiftUI);

This allows any of our NativeScript views to place the SwiftUI ContentView anywhere we'd like, for example:

<GridLayout>
  <SwiftUI swiftId="contentView" [data]="data">
  </SwiftUI>
</GridLayout>

Each declarative <SwiftUI/> component can specify which SwiftUI to use by it's swiftId which we defined via registerSwiftUI above. While at the same time we can bind TypeScript data to SwiftUI through it's data attribute.

But hang on, what about Swift Packages for the 3D assets?

NativeScript has supported Swift Package Manager since March of 2023. We can add the same Packages folder from the starter to our NativeScript projects root directory and reference it in nativescript.config.ts like this:

visionos: {
    SPMPackages: [
        {
            name: "RealityKitContent",
            libs: ["RealityKitContent"],
            path: "./Packages/RealityKitContent"
        }
    ]
}

If we run the app now via ns run visionos --no-hmr, we'll see exactly the same app running from NativeScript as we did from Xcode, while retaining the ability to edit our project in VS Code or Xcode, so both development workflows are supported.

visionOS NativeScript Preview in Xcode

NativeScript can be developed in TypeScript with VS Code and Vision Pro simulators, while the SwiftUI developer can still use Xcode with Previews if they desire as well.

With the visionOS app now part of the NativeScript project, we now have the ability to use those SwiftUI views in our NativeScript project.

Step 3: Adding NativeScript UIs to the visionOS app

In addition, we now have the best of all worlds, meaning we can also incorporate NativeScript UIs into the visionOS app as well. This phase was all about:

  • Enhancing the visionOS Experience: Leveraging NativeScript's dynamic UI capabilities to add reusable components.
  • Maintaining Consistency: Ensuring NativeScript UI blended seamlessly with visionOS's spatial interface.

We can use any NativeScript component within SwiftUI like this:

  • ContentView.swift
struct ContentView: View {
    var body: some View {
        VStack {
            Model3D(named: "Scene", bundle: realityKitContentBundle)
                .padding(.bottom, 50)

            // Use any NativeScript view component inside SwiftUI
            NativeScriptView(id: "hello")
        }
        .padding()
    }
}

We identify NativeScript components with an id by registering them with the SwiftUIManager like this:

  • app.ts: You can register views via TypeScript anywhere it makes sense in your NativeScript project
import { SwiftUIManager } from '@nativescript/swift-ui'
import { Hello } from './hello'; // NativeScript component

SwiftUIManager.registerNativeScriptViews(
    {
        hello: Hello,
    },
    {
        create: (id, component) => {
            console.log('SwiftUIManager creating:', id)
            return component.view;
        },
        destroy: (id) => {
            console.log('SwiftUIManager destroy:', id)
        },
    }
);

If we're using NativeScript with any flavor like Angular, React, Solid, Svelte or Vue, we can also take a moment to handle any flavor specific lifecycle. For example with Angular we can make sure any ComponentRef is also destroyed when SwiftUI destroys the component:

import { ComponentRef, inject, Injector, Injectable } from "@angular/core";
import { generateNativeScriptView } from "@nativescript/angular";
import { SwiftUIManager } from "@nativescript/swift-ui";
import { View } from "@nativescript/core";
import { HelloComponent } from "./hello.component";

@Injectable({ providedIn: 'root' })
export class AppService {
    ngRefs: Map<string, ComponentRef<View>>;
    injector = inject(Injector);

    constructor() {
        SwiftUIManager.registerNativeScriptViews(
            {
                hello: HelloComponent,
            },
            {
                create: (id: string, component) => {
                    if (!this.ngRefs) {
                        this.ngRefs = new Map();
                    }
                    const injector = Injector.create({
                        providers: [],
                        parent: this.injector,
                    });
                    const ngView = generateNativeScriptView(component, {
                        injector,
                    });
                    this.ngRefs.set(id, ngView.ref as ComponentRef<View>);
                    return ngView.firstNativeLikeView;
                },
                destroy: (id: string) => {
                    if (this.ngRefs.has(id)) {
                        this.ngRefs.get(id).destroy();
                        this.ngRefs.delete(id);
                    }
                },
            }
        );
    }
}

Now anytime SwiftUI renders NativeScriptView(id: "hello"), it will look up the component by it's id, in this case "hello" which maps to HelloComponent, and construct the component via the create method we registered with the SwiftUIManager. Lastly, the destroy method will be invoked allowing any cleanup you may want for your NativeScript components.

Step 4: Handling Immersive Sessions with Background/Foreground state

When a Vision Pro user taps the home button (top right button on goggles), any currently running immersive space in your app will immediately shut down as expected. You cannot have immersive spaces running in the background (for probably obvious reasons!). However, we want to allow the user to easily reenter the immersive space upon bringing the app back to the foreground so we need to ensure a clean exit to allow state restoration when the app moves between background/foreground states.

We needed to customize NativeScriptMainWindow slightly for this case. Let's consider an example.

We can have as many Immersive Spaces defined in our NativeScriptApp.swift as our app needs:

@main
struct NativeScriptApp: App {

    var body: some Scene {
        NativeScriptMainWindow()

        ImmersiveSpace(id: "MyImmersiveSpace") {
            LightView()
        }.immersionStyle(selection: .full, in: .full)
    }
}

We can open and close these immersive spaces using NativeScript as follows:

import { XR } from '@nativescript/swift-ui';

// open any immersive space by it's id
XR.requestSession('MyImmersiveSpace');

// close immersive space (you can only have one open at a time)
XR.endSession();

When we open an immersive space from NativeScript, we want to also match it with a closing call via XR.endSession(). The solution was quite simple but a great case to handle nonetheless:

@Observable
class AppState {
    var wasInForeground: Bool = false
}

@main
struct NativeScriptApp: App {
    @Environment(\.scenePhase) private var scenePhase
    @StateObject private var appState = AppState()

    var body: some Scene {
        NativeScriptMainWindow()
            .onChange(of: scenePhase) {
                if scenePhase == .inactive {
                    if appState.wasInForeground {
                        appState.wasInForeground = false

                        // let NativeScript close the immersive session
                        NotificationCenter.default.post(name: Notification.Name("NativeScriptExitImmersiveSession"), object: nil, userInfo: nil)
                    }
                } else if scenePhase == .active {
                    appState.wasInForeground = true
                }
            }

        ImmersiveSpace(id: "MyImmersiveSpace") {
            LightView()
        }.immersionStyle(selection: .full, in: .full)
    }
}

We can listen for that notification in NativeScript as follows:

import { Application } from '@nativescript/core';
import { XR } from '@nativescript/swift-ui';

export class StateService {

    constructor() {
        Application.ios.addNotificationObserver(
            'NativeScriptExitImmersiveSession',
            (notification: NSNotification) => {
                // ensure immersive sessions exit
                XR.endSession();
            }
        );
    }
}

Now if the app was backgrounded, when the user reopens the app we can allow them to properly restore the immersive session on their own.

The Result

Immensely creative collaboration opportunities with exciting, effective and versatile options at the developers disposal. Here's a glimpse at the exciting Vision Pro app demonstrated at LDI 2024 in Las Vegas from this magical pairing of technologies.

Some other references:

Takeaways

When SwiftUI and NativeScript developers embrace each other, magic happens ✨.

It's a blend of structure and flexibility + power and adaptability.

Building a visionOS app together was more than a technical endeavor; it was a journey of collaboration, learning, friendship and growth.

Additional Resources