Virtual Background with Effects SDK

Virtual background plugin helps in customising the peer's background. The customising options are blurring the background or replacing it with a static image. This guide provides an overview of usage of the virtual background plugin of 100ms.

Pre-requisites

Get the 100ms VirtualBackground Package (Supported since version 1.11.28)

npm install --save @100mslive/hms-virtual-background@latest

Features

The following features are currently supported under the HMSEffectsPlugin class:

/**
 * Sets the blur intensity.
 * @param {number} blur - The blur intensity, ranging from 0 to 1.
 * @returns {void}
 */
setBlur(blur) {}

/**
 * Sets the virtual background using the provided media URL.
 * @param {HMSEffectsBackground} url - The media URL to set as the virtual background.
 * Alternatively, the background image can be downloaded beforehand and passed to setBackground as objectURL
 * @returns {void}
 */
setBackground(url) {}

/**
 * Retrieves the name of the plugin.
 * @returns {string} The name of the plugin, 'HMSEffects'.
 */
getName() {}

/**
 * Retrieves the currently enabled background type or media URL.
 * @returns {string|MediaStream|MediaStreamTrack|HTMLVideoElement} The background type or media URL.
 */
getBackground() {}

/**
 * Sets the preset quality of the virtual background.
 * Options:  "balanced" | "quality"
 * The 'quality' preset has a higher CPU usage than the 'balanced' preset which is the default
 * @param {string} preset - The preset quality to set.
 * @returns {Promise<void>}
 */
setPreset(preset) {}

/**
 * Retrieves the active preset quality of the virtual background.
 * @returns {string} The active preset quality.
 */
getPreset() {}

/**
 * Clears all applied filters.
 * @returns {void}
 */
removeEffects() {}

/**
 * Stops the plugin.
 * @returns {void}
 */
stop() {}

Callbacks supported by the plugin:

On initialization, after the required resources are downloaded by the plugin:

const effectsPlugin = new HMSEffectsPlugin(<key>, () => console.log("Plugin initialised"));

On resolution change (on device rotation or when the video aspect ratio changes):

effectsPlugin.onResolutionChange = (width: number, height: number) => {
  console.log(`Resolution changed to ${width}x${height}`)
}

Instantiate Virtual Background

The SDK key for effects is needed to instantiate the HMSEffectsPlugin class:

  const effectsKey = useHMSStore(selectEffectsKey);

It is recommended to initialise the object in a separate file to prevent multiple initialisations on re-renders and keep the UI level code independent of internal calls by the SDK.

import { HMSEffectsPlugin, HMSVirtualBackgroundTypes } from '@100mslive/hms-virtual-background';

export class VBPlugin {
  private effectsPlugin?: HMSEffectsPlugin | undefined;

  initialisePlugin = (effectsSDKKey?: string) => {
    if (this.getVBObject()) {
      return;
    }
    if (effectsSDKKey) {
      this.effectsPlugin = new HMSEffectsPlugin(effectsSDKKey);
    }
  };

  getBackground = () => {
    return this.effectsPlugin?.getBackground();
  };

  getBlurAmount = () => {
    return this.effectsPlugin?.getBlurAmount();
  };

  getVBObject = () => {
    return this.effectsPlugin;
  };

  getName = () => {
    return this.effectsPlugin?.getName();
  };

  setBlur = async (blurPower: number) => {
    this.effectsPlugin?.setBlur(blurPower);
  };

  setBackground = async (mediaURL: string) => {
    this.effectsPlugin?.setBackground(mediaURL);
  };

  setPreset = (preset: string) => {
    this.effectsPlugin.setPreset(preset);
  };

  getPreset = () => {
    return this.effectsPlugin?.getPreset() || '';
  };

  removeEffects = async () => {
    this.effectsPlugin?.removeEffects();
  };

  reset = () => {
    this.effectsPlugin = undefined;
  };
}

export const VBHandler = new VBPlugin();

Building the UI

The following snippet illustrates how to add the plugin to the video and manage the UI state to preserve configuration:

import {
  selectEffectsKey,
  selectIsLocalVideoPluginPresent
  selectLocalVideoTrackID,
  useHMSStore,
} from '@100mslive/react-sdk';
import { 
  HMSEffectsPlugin, 
  HMSVirtualBackgroundTypes
} from '@100mslive/hms-virtual-background';
import { VBHandler } from './VBHandler';

export const VirtualBackgroundPicker = () => {
  const hmsActions = useHMSActions();
  // Get the effects SDK key here
  const effectsKey = useHMSStore(selectEffectsKey);
  const trackId = useHMSStore(selectLocalVideoTrackID);
  const isPluginAdded = useHMSStore(selectIsLocalVideoPluginPresent(VBHandler?.getName() || ''));
  
  // State can be used to show active selection
  const [background, setBackground] = useState<string | HMSVirtualBackgroundTypes>(
    VBHandler.getBackground() as string | HMSVirtualBackgroundTypes,
  );

  useEffect(() => {
    if (!track?.id) {
      return;
    }
    if (!isPluginAdded) {
      let vbObject = VBHandler.getVBObject();
      if (!vbObject) {
        VBHandler.initialisePlugin(effectsKey);
        vbObject = VBHandler.getVBObject();
        if (effectsKey) {
          hmsActions.addPluginsToVideoStream([vbObject as HMSEffectsPlugin]);
        } 
      }
    }
  }, [hmsActions, isPluginAdded, effectsKey, track?.id]);

  // UI code for media picker can go here
}

This handles initialisation and adding the plugin to the video stream. The plugin takes a few seconds on first load during initialisation. Subsequent filter and effect selections should take less than a second to reflect.

The methods can be called via the VBHandler object:

const setBackground = async(mediaURL : string) => {
  await VBHandler?.setBackground(mediaURL);
  // The selection can be highlighted using the activeBackground state
  setActiveBackground(mediaURL);
}

const setBlur = async(blurAmount: number) => {
  await VBHandler?.setBlur(blurAmount);
  setActiveBackground(HMSVirtualBackgroundTypes.BLUR);
}

const removeEffects = async() => {
  await VBHandler.removeEffects();
  setActiveBackground(HMSVirtualBackgroundTypes.NONE);
}

The full implementation can be viewed in the roomkit-react package.

Supported Browsers

Effects virtual background is supported on Safari, Firefox and Chromium based browsers.