How to moderate chat in 100ms? - Part 2

In the first part of our series on Chat Moderation in 100ms, we shared some simple techniques for keeping chat in check. You might want to look here if you have not seen it.

Now, let's dive into more ways to handle text chats in video apps using 100ms. This time, we'll use cloud services and LLMs to do the job automatically. Just like before, we'll tweak the chat feature of 100ms Prebuilt to show you how it's done.

Keep in mind, that the methods we're talking about are just one way to help with moderation. They're here to give developers ideas, though there's room for improvement.

Getting started

We'll begin by setting up 100ms Prebuilt for this tutorial as it comes with Chat configured.

• Fork your copy of the web-sdks repository from here.
• Clone it to your system and open it in a code editor of your choice.
• Run the command yarn to validate and resolve packages using the Yarn package manager.
• Navigate to web-sdks/packages/roomkit-react/src/Prebuilt/components/Chat/ChatFooter.tsx

In the 100ms Prebuilt, the message is typed in the Chat Footer and before it is sent it is stored in the message variable as follows:

const message = inputRef?.current?.value;

• Run the yarn build command to build the packages.
• Next, navigate to the prebuilt-react-integration under the examples directory and run the yarn dev commands in the terminal, to run the 100ms Prebuilt example locally.

Using a Cloud Service

Various cloud services offer options for moderating text. For instance, Google Cloud provides Natural Language API, while Microsoft Azure offers Content Moderator for text moderation, among others.

By leveraging these cloud-based moderation tools, developers can implement robust chat moderation systems without having to build everything from scratch. This saves time and resources while ensuring that chat conversations remain constructive and free from offensive content thus creating a safer environment for users and also reducing the burden on human moderators.

Using Google Cloud API

The first step is to find the API and enable it.

• Search for the Cloud Natural Language API in the Google Cloud Console.

• Click on the 'Cloud Natural Language API' under Marketplace and click on the 'Enable' button.

• Then, we'll need to create credentials to access the enabled APIs. We create an API key as follows:

To utilize the Natural Language API, we'll need to send a POST request to the provided endpoint with the API Key as a parameter.

POST https://language.googleapis.com/v2/documents:moderateText?key=API_KEY

The API provides us with a list of label names along with their corresponding confidence values. These labels represent different categories such as "Toxic", "Insult", etc. each with a confidence score indicating the likelihood that the text falls into that category.

For instance, if the confidence score for the "Toxic" category is significantly high, we can alert the user about the potentially harmful content and take measures to mitigate its impact.

{
  "moderationCategories": [
    {
      "name": "Toxic",
      "confidence": 0.8611886
    },
    {
      "name": "Insult",
      "confidence": 0.5483594
    },
  ...
  ],
  "languageCode": "en",
  "languageSupported": true
}

Additionally, the provided language code indicates the language of the text being analyzed, and the "languageSupported" field confirms whether the API supports the specified language for moderation purposes. This information helps ensure that the moderation process is accurate and effective across different languages.

In our code, we find the message variable. We can pass this message to the API request body, before it is sent on the public chat, as follows:

    await fetch(
      'https://language.googleapis.com/v2/documents:moderateText?key=API_KEY',
      {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
        },
        body: JSON.stringify({
          document: {
            content: message,
            type: 'PLAIN_TEXT',
          },
        }),
      },
    )
      .then(response => {
        if (!response.ok) {
          ToastManager.addToast({
            title: 'Response not ok. Please try again.',
          });
        }
        return response.json();
      })
      .then(data => {
        const moderationCategories = data.moderationCategories;
        let maxConfidence = 0;
        let maxCategory = '';

        moderationCategories.forEach((category: { confidence: number; name: string }) => {
          if (category.confidence > maxConfidence && category.confidence > 0.8) {
            maxConfidence = category.confidence;
            maxCategory = category.name;
          }
        });

        if (maxCategory !== '') {
          ToastManager.addToast({
            title: 'Message contains ' + maxCategory + ' information. Please refrain from it!',
          });
          return;
        } 
      })
      .catch(error => {
        ToastManager.addToast({
          title: 'There was an error moderating the message. Please try again.' + error,
        });
      });

Notice how we iterate through the list of moderation categories, each containing a confidence value and a corresponding name. Once we determine the category with the maximum confidence, we display a message to the user via a toast notification. By providing real-time feedback about potentially problematic content, we contribute to fostering a safer and more respectful online environment.

Using Microsoft Azure API

Similar to the Natural Language API, we can utilize Content Moderator's text moderation models to examine text content and manage chat in 100ms.

It also offers various categories with scores, presented in the following format:

"Classification": {
    "ReviewRecommended": true,
    "Category1": {
        "Score": 1.5113095059859916E-06
    },
    "Category2": {
        "Score": 0.12747249007225037
    }
}

Furthermore, in addition to its primary functions, the text moderation system also identifies potential instances of personal data, including common forms such as email addresses, US mailing addresses, IP addresses, and US phone numbers.

Moreover, the text moderation response can provide the text with basic auto-correction applied, if preferred. This feature can help improve the readability and clarity of the text while maintaining its original meaning.

The system can be customized to screen for terms that are tailored to your business requirements.

For instance, you may have specific terms or phrases that you want to filter out from user-generated content, such as competitive brand names or sensitive information. This customization ensures that the moderation process aligns with your business objectives and helps maintain a consistent brand image across your platform.

Learn more about Microsoft Azure’s Natural Language API here.

Using an LLM

Using Large Language Models (LLMs) for chat moderation can sometimes be excessive, primarily due to the associated costs. However, one intriguing use case is to prompt the AI model to adjust the text to generate a message that is safe to share. This approach not only helps mitigate the risks associated with inappropriate or harmful content but also provides a solution for maintaining a positive user experience.

By automatically adjusting the text to align with community guidelines or platform policies, this method contributes to fostering a safer and more inclusive online environment.

A local LLM like Gemma

Gemma is a set of lightweight, advanced open models constructed using the same research and technology behind the Gemini models. The model weights are accessible in two sizes: Gemma 2B and Gemma 7B, each accompanied by pre-trained and instruction-tuned variations.

• The first step is to download a supported model and store it within the project directory:

<dev-project-root>/assets/gemma-2b-it-gpu-int4.bin

• A task can then be created as follows:

const genai = await FilesetResolver.forGenAiTasks(
    // path/to/wasm/root
    "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-genai@latest/wasm"
);
const llmInference = await LlmInference.createFromOptions(genai, {
    baseOptions: {
        modelAssetPath: '/assets/gemma-2b-it-gpu-int4.bin'
    },
});

Here we use createFromOptions() function and specify the path of the model with the baseOptions object modelAssetPath parameter.

• Finally, run the task as follows:

const response = await llmInference.generateResponse(message);

The LLM Inference API employs the generateResponse() function to initiate inferences and provides a string as its output, containing the generated response text.

An API like Open AI’s API

If the asset file is too large to include with your project, you might opt to utilize an online AI model such as OpenAI's GPT-4 for moderation purposes instead.

Implementing the Open AI’s API for this task is relatively simple - send your message and prompt to the API endpoint, and it will return the edited message if the original was found to be inaccurate or insufficient.

	 await fetch('https://api.openai.com/v1/chat/completions', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        Authorization: 'Bearer OPENAI_API_KEY',
      },
      body: JSON.stringify({
        model: 'gpt-3.5-turbo',
        messages: [
          {
            role: 'user',
            content:
              'You are a chat moderator for a live audio video session. Your task is to make sure the text shared in chat is appropriate. If the user message shared is not appropriate rewrite it to make it follow community guidlines. If it is appropriate, send it back as is. Do not reply with anything else. The user intends to send this message: ' +
              message,
          },
        ],
        temperature: 0.7,
      }),
    })
      .then(response => {
        if (!response.ok) {
          throw new Error('Network response was not ok');
        }
        return response.json();
      })
      .then(data => {
        const completion = data.choices[0].message.content.trim();
        console.log(completion);
        message = completion;
        ToastManager.addToast({
          title: 'Message modified to comply with community guidelines.',
        });
      })
      .catch(error => {
        console.error('There was a problem with the fetch operation:', error);
  });

We've explored various options for chat moderation, both through cloud services and utilizing LLMs. Thank you for taking the time to read through it. I trust it’ll provide insights into constructing your chat moderation system.

Should you have any inquiries, our team is ready to assist you on Discord. You can find us here.