Android SurfaceViewRenderer Explained

The mighty SurfaceViewRenderer! It's the very end of a long WebRTC chain that gets video from other people to your screen.

Learn how to build Twitter Spaces with 100ms iOS SDK

It can also be a bit difficult to use if it's your first time around. Especially in a dynamic android app with lots of views flicking in and out.

Are you a Flutter developer? check out our comprehensive guide on Flutter WebRTC

Let's take a tour through the many ways it can go wrong and what you should know to keep your video conference calls running smoothly.

Here's the lifecycle of a SurfaceViewRenderer:

Which means:

1. Once you create the SurfaceViewRenderer, you need to initialize it before it can be used at all. This kicks off a render thread.
2. addSink should be called after this. Where is addSink you ask? Well it's not on the SurfaceViewRenderer, it's a method on the VideoTrack you want to add to it.
3. removeSink should only be called when you're done showing the video and now it's time to either destroy the view (or recycle it), this stops downloading data from the peer's video too. Another function that's only on the VideoTrack, not the SurfaceViewRenderer though it takes the SurfaceViewRenderer as a parameter.
4. Finally release so the render thread can stop.

Once released the SurfaceViewRenderer can be inited again and the cycle renews. Now let's look at what happens when any one of these is missed or not called with the right parameters.

1. Initing without a shared EglContext

This one is actually mentioned in the source code of SurfaceViewRenderer, when you call SurfaceViewRenderer.init(context) it must only be with a shared EGL context.

What does that mean? That you call org.webrtc.EglBase.create() once in something like a singleton and then always initialize all the SurfaceViewRenderers with that saved context's eglBaseContext.

If you call create() for each SurfaceView, you'd end up with no errors and no video playing at all. It's easy to get stuck here and wonder why SurfaceViewRenderer is not showing any video when seemingly nothing went wrong.

2. Not releasing before initing

If you don't release after you've removed a video sink, you're going to get the same result as if you initialized twice.

java.lang.IllegalStateException: videoSurfaceViewAlready initialized
    at org.webrtc.EglRenderer.init(EglRenderer.java:212)

So you've just got to release at the right time.

3. What if you never init at all?

The view just never shows up ¯\_(ツ)_/¯
Behind the scenes a render thread was never created and the loop to draw wasn't kicked off so, of course, nothing would show up.

4. Double the addSinks, double vision

Calling addSink on the same SurfaceViewRenderer is a pretty fun bug. You won't realize anything's wrong if you just added all the videos for people twice until the order of the videos is changed.
Then you'll see flicker as two different video sources try to play onto the same surface. Both video sources will take race condition turns rendering.
Oh, and you'll keep downloading the video from the source peer if you only called removeSink once when disposing of the view.

Here's an image of what the two video streams look like separately and what happens when addSink for both was called on the same SurfaceViewRenderer.

5. How many SurfaceViewRenderers can you create? What happens beyond?

Without initializing them, you can create a heck of a lot. But there's a limit to how many different initialized SurfaceViewRenderers you can prepare at a time.

The limit varies from device to device, when I tried on one modestly powered Nokia 3.4 phone it got up to 30 contexts initialized until it failed with:

E/AndroidRuntime: FATAL EXCEPTION: main process: com.example.myvideocallapp, PID: 532java.lang.RuntimeException: java.lang.RuntimeException: Failed to create EGL context: 0x3003at org.webrtc.EglBase14Impl.createEglContext(EglBase14Impl.java:282)at org.webrtc.EglBase14Impl.(EglBase14Impl.java:78)at org.webrtc.EglBase.createEgl14(EglBase.java:215)at org.webrtc.EglBase.create(EglBase.java:158)

On others, like an Oppo F19 it gets beyond 150 and keep going. Considering that that's the number of videos you're displaying simultaneously, you may not need to get anywhere near that much.

Now let's look at something even tricker than managing a single or a handful of a static number of SurfaceViewRenderers.

Managing the lifecycle within a RecyclerView

As a refresher, the RecyclerView is an essential View on Android that lets you avoid creating expensive view objects when scrolling through a large list.

What's even more expensive than a typical view object? The SurfaceViewRenderer of course.

Let's think about what a RecyclerView does in android.

1. It creates a view when a new one is required.
2. It keeps a cache of older views that aren't being used right now.
3. It may need to render a view in a different place because its order in the layout changed. (In case of a high score board, there may be a rapid change in positions)

How does that compare to the lifecycle of a SurfaceView?

This poses a natural problem because there are multiple things that could cause the view to be changed.

1. The position of the view changes. A teacher who's supposed to be shown first, may join after the students.
2. The peer representing this video leaves, so has to be removed.
3. New people keep joining so new ones have to be made.
4. Someone scrolls the list, so no new views are made but they are constantly recycled.

To further complicate matters, someone's video may not have been on at first, so you may just show the person's initials, but later they do turn it on, so now the SurfaceViewRender has to be initialized.

If you're to Android, rebinding views is straightforward, you'd just do that in the AdapterView's bind method but a slightly unconventional approach is required here because anyone's video can go on or off at any time.

So the big question becomes, when do you initialize the video?
You'd quickly see a 'bind' method is insufficient.

What we need here are some lesser-known adapter methods:
onViewAttachedToWindow
Called when a new view is added to the window or a cached one is restored.

onViewDetachedFromWindow
Called when a view is removed, we don't know if it's going into cache or out permanently.

Take a look at the PeerViewHolder and the PeerAdapter to see how all the disparate cases are handled. This is the simplest example of how you could manage the tricky SurfaceViewRenderer in an Android RecyclerView.
This is part of the "hello world" demonstration of how the 100ms SDK can be used.

To look into more advanced use cases, take a look at our fully featured sample app.