Screen Share

Flutter SDK provides support for sharing the entire screen of the device to the room.

Please note that for a peer to share their screen, their role must have Screenshare enabled in the dashboard. Also select the appropriate resolution for the Screen share quality. 1080p is recommended for better text readability.

Prerequisites

Let's first do some setup required for both the platforms

Android Setup

You also need to pass the intent from android native side to HMS SDK in the following way :

In your app's MainActivity add -

import live.hms.hmssdk_flutter.HmssdkFlutterPlugin
import android.app.Activity
import android.content.Intent
import live.hms.hmssdk_flutter.Constants

override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {
super.onActivityResult(requestCode, resultCode, data)

    if (requestCode == Constants.SCREEN_SHARE_INTENT_REQUEST_CODE && resultCode == Activity.RESULT_OK) {
        data?.action = Constants.HMSSDK_RECEIVER
        activity.sendBroadcast(data?.putExtra(Constants.METHOD_CALL, Constants.SCREEN_SHARE_REQUEST))
    }
}

<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />

<!---- Required for android 14 and above --->
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_MEDIA_PROJECTION" />

import live.hms.hmssdk_flutter.HmssdkFlutterPlugin
import io.flutter.embedding.android.FlutterFragmentActivity
import android.content.Intent
import live.hms.hmssdk_flutter.Constants

override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {
super.onActivityResult(requestCode, resultCode, data)

    if (requestCode == Constants.SCREEN_SHARE_INTENT_REQUEST_CODE && resultCode == Activity.RESULT_OK) {
        data?.action = Constants.HMSSDK_RECEIVER
        this.sendBroadcast(data?.putExtra(Constants.METHOD_CALL, Constants.SCREEN_SHARE_REQUEST))
    }
}

Migrating from older HMSSDK version to 1.5.0 or above

There is a change in the Screenshare setup in HMSSDK version 1.5.0 as compared to older versions. Let's see how we can migrate from older versions to 1.5.0 or above.

Replace these two lines of application's MainActivity.kt file:

if (requestCode == Constants.SCREEN_SHARE_INTENT_REQUEST_CODE && resultCode == Activity.RESULT_OK){
    HmssdkFlutterPlugin.hmssdkFlutterPlugin?.requestScreenShare(data)
}

with these lines:

if (requestCode == Constants.SCREEN_SHARE_INTENT_REQUEST_CODE && resultCode == Activity.RESULT_OK) {
    data?.action = Constants.HMSSDK_RECEIVER
    activity.sendBroadcast(data?.putExtra(Constants.METHOD_CALL, Constants.SCREEN_SHARE_REQUEST))
}

Here's a Screenshot showing the Git Diff between older and the implementation in version 1.5.0 and above.

That's it 🥳🥳

You can run the application now.

iOS Setup

You need to create an iOS broadcast upload extension. It uses Apple's ReplayKit framework to record the device screen and delivers frame samples to your broadcast extension. You can share not only your own app but also the entire device sceeen including other apps on the device.

Step 1 - Open project

Open your iOS Xcode project, such as ios/Runner.xcworkspace for full-Flutter apps.

Step 2 - Add Broadcast Upload Extension

Click on your project in the Project Navigator to show the project settings. Press + at the bottom of the target list to add a new target.

Select the Broadcast Upload Extension type for your new target.

Enter your new target detail in the dialog. Uncheck Include UI Extension option.

In the following dialog, activate the new scheme for the new target.

Step 3 - Add App Group

Click + icon in Signing & Capabilities section. Select App Group from the list of Capabilities.

New section should be added under Signing & Capabilities named App Groups. Click + icon under that.

Enter App group name (create unique app group name ex: group.your.domain.name)

Step 4 - Edit Podfile

In ios folder of your flutter project and open Podfile. Paste the following code and replace the extension name you just created:

target 'Your Extension Name here' do
  use_modular_headers!
  pod 'HMSBroadcastExtensionSDK'
end

In terminal change directory to ios and run pod install command.

Step 5 - Edit SampleHandler

Expand Runner > ExtensionName and open SampleHandler file.

Replace the code with the code below and pass app group name to the respected field:

import ReplayKit
import HMSBroadcastExtensionSDK

class SampleHandler: RPBroadcastSampleHandler {

    let screenRenderer = HMSScreenRenderer(appGroup: "Enter App Group Name")

    override func broadcastStarted(withSetupInfo setupInfo: [String : NSObject]?) {
        // User has requested to start the broadcast. Setup info from the UI extension can be supplied but optional.
    }

    override func broadcastPaused() {
        // User has requested to pause the broadcast. Samples will stop being delivered.
    }

    override func broadcastResumed() {
        // User has requested to resume the broadcast. Samples delivery will resume.
    }

    override func broadcastFinished() {
        // User has requested to finish the broadcast.
        screenRenderer.invalidate()
    }

    override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) {
        switch sampleBufferType {
        case RPSampleBufferType.video:
            // Handle video sample buffer
            if let error = screenRenderer.process(sampleBuffer) {
                if error.code == .noActiveMeeting {
                    finishBroadcastWithError(NSError(domain: "ScreenShare",
                                                     code: error.code.rawValue,
                                                     userInfo: [NSLocalizedFailureReasonErrorKey : "You are not in a meeting."]))
                }
            }
            break
        case RPSampleBufferType.audioApp:
            // Handle audio sample buffer for app audio
            break
        case RPSampleBufferType.audioMic:
            // Handle audio sample buffer for mic audio
            break
        @unknown default:
            // Handle other sample buffer types
            fatalError("Unknown type of sample buffer")
        }
    }
}

Key Notes

To start Screenshare from iOS devices you need to pass App Group and Preferred Extension name to HMSSDK constructor as shown below.

You can find appGroup and preferredExtension name in Xcode under Signing and Capabilities section under Target > yourExtensionName.

Once you have the correct App Group & Preferred Extension values created in Xcode & linked to your Apple Developer Account, you can now use them to start Screenshare from iOS devices (iPhone / iPad).

// Pass the correct App Group & Preferred Extension parameters in HMSIOSScreenshareConfig class.
HMSIOSScreenshareConfig iOSScreenshareConfig = HMSIOSScreenshareConfig(
    appGroup: "appGroup", // App Group value linked to your Apple Developer Account
    preferredExtension: "preferredExtension" // Name of the Broadcast Upload Extension Target created in Xcode
);

HMSSDK hmsSDK = HMSSDK(iOSScreenshareConfig: iOSScreenshareConfig);

await hmsSDK.build(); // ensure to await while invoking the `build` method

After completing the setup let's see how we can Start the Screenshare from iOS.

How to Start/Stop Screenshare from the app

To start screen share, app needs to call the startScreenshare method of HMSSDK and similar is it's counterpart stopScreenShare to stop screen share.

Following is the snippet on how to use this:

class Meeting implements HMSUpdateListener, HMSActionResultListener{

    void startScreenShare() {
        ///[hmsActionResultListener]: an instance of a class that implements HMSActionResultListener
        //Here this is an instance of a class that implements HMSActionResultListener, that is, Meeting
        hmsSDK.startScreenShare(hmsActionResultListener: this);
    }

    void stopScreenShare() {
        ///[hmsActionResultListener]: an instance of a class that implements HMSActionResultListener
        //Here this is an instance of a class that implements HMSActionResultListener, that is, Meeting
        hmsSDK.stopScreenShare(hmsActionResultListener: this);
    }

    @override
    void onSuccess(
      {HMSActionResultListenerMethod methodType =
          HMSActionResultListenerMethod.unknown,
      Map<String, dynamic>? arguments}) {
          switch (methodType) {
          ...


            case HMSActionResultListenerMethod.startScreenShare:
            //Screen share started successfully
            break;

            case HMSActionResultListenerMethod.stopScreenShare:
            //Screen share stopped successfully
            break;
      }
  }


  @override
  void onException(
      {HMSActionResultListenerMethod methodType =
          HMSActionResultListenerMethod.unknown,
      Map<String, dynamic>? arguments,
      required HMSException hmsException}) {
          switch (methodType) {
          ...

            case HMSActionResultListenerMethod.startScreenShare:
            // Check the HMSException object for details about the error
            break;

            case HMSActionResultListenerMethod.stopScreenShare:
            // Check the HMSException object for details about the error
            break;

      }
  }

}

How to get Screen Share Status

Application needs to call the isScreenShareActive method of HMSSDK.

This method returns a Boolean which will be true in case ScreenShare is currently active and being used, and false for inactive state.

hmsSDK.isScreenShareActive();

How to display screenshare in apps

A peer in Room can broadcast their Screen from any platform like Web, Android, iOS, etc. If a peer shares their Screen from Web & the viewer is on a Mobile platform, some content on the Screen can get cropped. It's necessary to configure the HMSVideoView correctly to ensure the complete Screen share content is visible without any clipping/cropping from edges.

Always create the HMSVideoView with ScaleType as ScaleType.SCALE_ASPECT_FIT for correctly showing Screenshare Tracks.

Let's look at the implementation:

 HMSVideoView(
  track: screenShareTrack,  // pass the screen share track here
  scaleType: ScaleType.SCALE_ASPECT_FIT, // always set to Aspect Fit for Screenshare Tracks
  key: Key(videoTrack.trackId),   // set a unique identifier using the trackId
)

To learn more about Rendering any Video, refer the guide here.

Troubleshooting Guide

For starting Screenshare from iOS devices (iPhones or iPads) following are some common setup you should already have within your Apps -

Bitcode Disabled

Bitcode has been disabled by Apple from Xcode 14 & iOS 16 and above. So 100ms packages also have Disabled Bitcode to ensure compatibility. Ensure that in your Xcode project Bitcode is Disabled for all Targets.

Podfile with Bitcode Disabled

You can use the following Podfile which has post_install script to Disable Bitcode for all Pods. Ensure that you modify the target for your Main App and the Broadcast Upload Extension.

In this sample Podfile, the Target names are Runner and FlutterBroadcastUploadExtension. Change these to the actual Target names defined in your Xcode project.

platform :ios, '13.0'

project 'Runner', {
  'Debug' => :debug,
  'Profile' => :release,
  'Release' => :release,
}

def flutter_root
  generated_xcode_build_settings_path = File.expand_path(File.join('..', 'Flutter', 'Generated.xcconfig'), __FILE__)
  unless File.exist?(generated_xcode_build_settings_path)
    raise "#{generated_xcode_build_settings_path} must exist. If you're running pod install manually, make sure flutter pub get is executed first"
  end

  File.foreach(generated_xcode_build_settings_path) do |line|
    matches = line.match(/FLUTTER_ROOT\=(.*)/)
    return matches[1].strip if matches
  end
  raise "FLUTTER_ROOT not found in #{generated_xcode_build_settings_path}. Try deleting Generated.xcconfig, then run flutter pub get"
end

require File.expand_path(File.join('packages', 'flutter_tools', 'bin', 'podhelper'), flutter_root)

flutter_ios_podfile_setup

# ENSURE TO SET THE CORRECT MAIN APP TARGET NAME BELOW
target 'Runner' do
  use_modular_headers!
  use_frameworks!

  flutter_install_all_ios_pods File.dirname(File.realpath(__FILE__))
end


# ENSURE TO SET THE CORRECT SCREENSHARE EXTENSION TARGET NAME BELOW
target 'FlutterBroadcastUploadExtension' do
  use_modular_headers!
  use_frameworks!
  pod 'HMSBroadcastExtensionSDK'
end

# Post install script to Disable Bitcode from all Pods
post_install do |installer|
  installer.pods_project.targets.each do |target|
    flutter_additional_ios_build_settings(target)
    target.build_configurations.each do |config|
      config.build_settings['ENABLE_BITCODE'] = 'NO'
    end
  end
end

Background Modes Enabled

In majority use-cases, playing audio from Room would be required when app is in Background Mode. Mostly, if users are starting Screenshare from their iPhones/iPads they would want to continue listening to audio from the Room. So, ensure that you have Background Modes Enabled in your Xcode project.

App Groups Enabled

Ensure that you have enabled App Groups for both your Main App Target & the newly created Broadcast Extension Target. If the same App Group is not enabled on both Targets then the App & the Screenshare Extension won't be able to communicate & starting Screenshare from your iOS device will fail. Also, ensure that there's no typo / spelling mis-matches between the App Group enabled on Main App Target & the Screenshare Broadcast Extension Target.

Permission Denied EXC_BAD_ACCESS error

100ms Example Apps already have configurations for starting Screenshare on iOS devices. The values for App Group & Preferred Extension used in 100ms Example Apps cannot be reused by any other apps as they won't be linked to your Apple Developer Account.

If the 100ms values for App Group which is "group.flutterhms" & Preferred Extension which is "FlutterBroadcastUploadExtension" are resued by your apps then you will have an Xcode exception containing Permission Denied EXC_BAD_ACCESS error message similar to the one shown below -

CFMessagePort: bootstrap_register(): failed 1100 (0x44c) 'Permission denied', port = 0xbc03, name = 'group.flutterhms.88C5E70E-F40F-4C36-A48C-E65736E85CAC.audio.mach.port'
See /usr/include/servers/bootstrap_defs.h for the error codes.
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x8)
    frame #0: 0x00000001c03ddc78 CoreFoundation_CFGetNonObjCTypeID + 92
...

To resolve this issue ensure that you create a Unique App Group which is linked to your Apple Developer Account. If you do not have an Active Apple Developer Account or do not have permissions to create a New App Group, then you won't be able to this feature.

Same Minimum iOS Deployment Version on Extension Target and Main App

Ensure that the Minimum iOS Deployment Target Version is set to 13.0 or above in your Xcode project & the Podfile. Ensure that both Targets - Main app & the Screenshare Extension have the same Minimum iOS Deployment version.

Show only my app name when starting iOS Screenshare

Ensure that your app's Screenshare extension name is passed as the preferredExtension name to HMSIOSScreenshareConfig.

Ensure that there are no typos/spelling mistakes in the name passed.

Your preferredExtension name in Xcode is under Signing and Capabilities section under Target > yourExtensionName.

// Pass the correct App Group & Preferred Extension parameters in HMSIOSScreenshareConfig class.
HMSIOSScreenshareConfig iOSScreenshareConfig = HMSIOSScreenshareConfig(
    appGroup: "appGroup", // App Group value linked to your Apple Developer Account
    preferredExtension: "preferredExtension" // pass correct preferredExtension to show only your app name while starting Screenshare
);

HMSSDK hmsSDK = HMSSDK(iOSScreenshareConfig: iOSScreenshareConfig);

await hmsSDK.build(); // ensure to await while invoking the `build` method

Running on Simulator

Starting Screenshare from an iOS Simulator is not supported by Apple. You can start Screenshare only from an actual iOS device like an iPhone or iPad.

iOS Deployment Target Version

100ms Flutter Package is supported for iOS 12 and above versions. Ensure that the Minimum iOS Deployment Target Version is set to 12.0 or above in your Xcode project & the Podfile.

Flutter version 3.3.0+

100ms Flutter Package is supported for Flutter versions 3.3.0 or above. You can check your current Flutter version by running the flutter doctor command.

Role has Screenshare Permission

Ensure that the Role used to Join the Room has Screenshare permission Enabled from the 100ms Dashboard. If the Screenshare permission is not Enabled from the Dashboard, any users joining with this Role won't be able to start Screenshare. These users would still be able to see Screenshare performed by other Peers who have Screenshare permissions.

Avoid infinite looping of video when local peer is sharing screen

When you share your screen, you'll notice an infinite looping of the shared screen within the user interface, resembling something like the left image. To prevent this, you can display a custom widget when the local peer shares their screen by utilizing the isLocal check on the HMSPeer object like the image on the right.

Widget screenShareWidget(HMSPeer peer){

    /// If the peer sharing screen is a local peer then we can render a custom widget
    if(peer.isLocal){
        ///Render a custom widget
    }
    else{
        ///Render the screen share widget using HMSVideoView
    }
}

Explore the code for the aforementioned UI to eliminate any screen looping concerns here