Enhancing Live Teaching Experience with Tap-to-Focus in 100ms SDK

Online classes have become a more common part of the education system worldwide since the pandemic. We at 100ms understand the importance of providing a smooth and pleasant experience for both teachers and students during live teaching classes. We’ll talk about a common problem that teachers face in live classes and the ‘Why, What and How’ of fixing it.

Autofocus and the Whiteboard

One of our EdTech customers utilizing our live SDK contacted us recently regarding an issue they encountered with a teacher using a digital whiteboard for instruction. Specifically, the camera focus would shift whenever the teacher moved in front of the whiteboard.

In a traditional WebRTC call, the camera uses the default autofocus and exposure mode. This can sometimes lead to the whiteboard becoming out of focus or over/under exposed. And this makes it difficult for students to see what is written on the board clearly.

Fixing the Focus

We decided to fix this issue by implementing tap-to-lock-focus functionality as part of the 100ms iOS SDK. This allows teachers to easily lock the focus of the camera onto the whiteboard, ensuring that the board remains in focus throughout the class. The tap-to-focus functionality is integrated directly into the HMSVideoView, which is an easy-to-use API that developers can use to render video during a WebRTC call.

The feature includes three different focus modes:

1. kHMSFocusModeAuto - The default focus mode, where the camera continuously sets the focus and exposure based on the center point of the scene. This mode is useful when the user wants the camera to automatically adjust to the changing environment or when the user is moving around.
2. kHMSFocusModeTapToAutoFocus - This allows the user to tap on an object to focus on it. Once the user moves the camera enough to change the subject of the scene, the camera switches back to the default kHMSFocusModeAuto. This mode is useful when the user wants to quickly adjust the focus to a specific object, but still wants the camera to adjust to the changing environment.
3. kHMSFocusModeTapToLockFocus - This allows the user to tap on an object to lock focus on it. The focus will not be changed until the user taps again to focus on another object. This mode is useful when the user wants to keep the focus on a specific object, even if the camera is moved.

To use this feature, developers simply had to enable it in the settings—by setting the camera focus mode to one of the modes mentioned above. We made it easy to implement and roll out, allowing teachers to start using it as soon as possible. Now, let’s have a look at how we implemented it!

Behind the Scenes

The class that does the capturing from device camera on WebRTC is RTCCameraVideoCapturer. This gives us an array of camera devices available. We can use either the front or the back camera to capture the local video. This capture device is an AVCaptureDevice that represents a physical camera on iPhone.

Implementing tap-to-focus consisted mainly of two parts:

1. Enabling the user to be able to tap on the local video tile to let the SDK know what part of the video should get focused.
2. Sending the Focus information to the video capturer.

100ms SDK provides a simple interface to render video called HMSVideoView. We installed a tap handler on HMSVideoView such that if the view is being used to render local video, it calculates the fractional coordinates (range 0-1.0) of the tap location inside the view. Note that fractional coordinates was essential because then you don't have to deal with scaling of the video layer itself. It's important to note that we must factor in the device's orientation when calculating the tapping point.

UIInterfaceOrientation currentOrienation;
    
    if (@available(iOS 13.0, *)) {
        currentOrienation = UIApplication.sharedApplication.windows.firstObject.windowScene.interfaceOrientation;
    } else {
        // Fallback on earlier versions
        currentOrienation = UIApplication.sharedApplication.statusBarOrientation;
    }
    
    CGPoint focusPoint;
    
    if (currentOrienation == UIInterfaceOrientationLandscapeLeft || currentOrienation == UIInterfaceOrientationLandscapeRight) {
        focusPoint = CGPointMake([touchPoint locationInView:self].x / viewSize.width, [touchPoint locationInView:self].y / viewSize.height);
    }
    else {
        focusPoint = CGPointMake([touchPoint locationInView:self].y / viewSize.height, 1.0 - [touchPoint locationInView:self].x / viewSize.width);
    }

Now comes the second part. We send these fractional coordinates to the video capturer. Here we set the AVCaptureDevice's focal point and exposure according to the focused point on screen.

[self.device lockForConfiguration:nil];
    
if (self.device.isFocusPointOfInterestSupported) {
	self.device.focusPointOfInterest = point;
	self.device.focusMode = AVCaptureFocusModeAutoFocus;
}
    
if (self.device.isExposurePointOfInterestSupported) {
	self.device.exposurePointOfInterest = point;
	self.device.exposureMode = AVCaptureExposureModeAutoExpose;
}
    
[self.device unlockForConfiguration];

There is one more thing remaining. If you see the tap-to-focus feature on the native iOS camera app, you can notice that the camera switches back to auto focus when you move the camera too much from the current scene. This is the same functionality we wanted to offer in our TapToAutoFocus mode.

To accomplish this we have to know exactly ‘when’ the scene has changed enough, so that we can reset the focus and exposure to auto. To do this, we observe AVCaptureDeviceSubjectAreaDidChange notification from iOS notification center, and reset to autofocus.

subjectAreaDidChangeObserver = [NSNotificationCenter.defaultCenter addObserverForName:AVCaptureDeviceSubjectAreaDidChangeNotification object:nil queue:NSOperationQueue.mainQueue usingBlock:^(NSNotification * _Nonnull note) {
	[self resetFocusAndExposure];
}];

The End

This feature ensures that the whiteboard remains in focus during the class, making it easy for students to see what is written on the board and follow along with the class. And finally, the teacher was happy with the tap-to-lock-focus implemented in the SDK and so was our customer. Thanks for reading!

At 100ms, we emphasize developer-friendly APIs that make it easy for our customers to implement new features. The tap-to-focus is no exception, and we have made it as simple as possible to use. To learn more about this feature, please visit our developer documentation at Tap-to-focus.