Controls are mirrored on local stream & local stream is unmirrored in fullscreen mode

[Philzen]

When one enables the controls on a stream via the context menu, it reveals that the local stream is mirrored – which is a feasible setting in itself, coming from

livekit-react/packages/core/src/components/VideoRenderer.tsx

Line 59 in 7a2a96f

transform: isLocal && isFrontFacing ? 'rotateY(180deg)' : '',

However all the controls are mirrored as well, which makes a really awkward interface 😉
As a workaround i applied a rule such as

video::-webkit-media-controls {
  transform: scaleX(-1)
}

(btw that does the same job as rotateY(180deg) – not sure which is faster to the GPU or preferable in general).
Workaround works fine in Firefox and WebKit-based browsers – until you enter fullscreen mode: In Firefox it's the least bad, controls are fine in fullscreen but become mirrored when exiting fullscreen for a while, until the browser catches up again. In Chromium the controls are effectively mirrored in fullscreen which sort of negates the point of the workaround.

---

Worse: in both Firefox and Chromium the mirroring of the video stream is not active in fullscreen mode – the picture flips back and forth between non-mirrored and mirrored mode when switching between fullscreen and non-fullscreen mode.

The mirroring also is not applied when using picture-in-picture mode.

---

I understand from my issue title this technically are two separate issues, but i believe they are two birds that can only be killed with the same stone. I experimented with ::fullscreen-css rule to now avail – i believe it only works on HTML elements along with their content, but seems not to work the video-element itself, presumably b/c it is rendered natively by the client.

We could probably best solve both issues by providing custom controls along with a wrapper around <video> which should react fine to ::fullscreen-styles.

Some interesting article on how this is done: https://css-tricks.com/custom-controls-in-html5-video-full-screen/

[Philzen]

Related issues:

• Gets a mirrored video stream #29
• would be great to add fullScreen button in controlsView #78

[davidzhao]

Thank you for the detailed description! You are making it easy for us to understand.

However all the controls are mirrored as well
Which controls are you referring to? Do you have a screenshot demonstrating the issue?

For full screen mode, are you just entering it with element.requestFullscreen() ?

[Philzen]

Which controls are you referring to? Do you have a screenshot demonstrating the issue?

This concerns the native controls that the user can enable via the context menu (via right-click on the video-element).

For full screen mode, are you just entering it with element.requestFullscreen()?

No, this is the native functionality activated by clicking the button. Note that the mirroring also doesn't work when choosing picture-in-picture mode.

This is exactly why my current hunch is that implementing custom controls as described in the linked article might be the only way to provide a consistent experience – wondering if there is also a solution for the picture-in-picture mode. Excited to learn / wondering if there is another approach though!

[davidzhao]

ah ha! that makes sense. I didn't realize the controls could be shown that way. For the most part, these controls aren't too meaningful with WebRTC streams.

the bit about full screen is interesting.. and I think that's much harder to resolve. While full screening yourself doesn't seem like an extremely common use case, we should make it more clear what's happening.

[Philzen]

I didn't realize the controls could be shown that way.

Me neither, until our project sponsors and a test user showed this to us 😆

For the most part, these controls aren't too meaningful with WebRTC streams

Some of them are a real nuisance – you can actually pause or mute streams with them... 🤯

[Philzen]

I did some quick tests to find out when and how the transform is applied correctly:

	Firefox	Chromium (Blink)
Normal	✔️	✔️
<video>.requestFullscreen()	❌	❌
<div><video></div>.requestFullscreen()	✔️	✔️
<video> in PiP	✔️ (¹)	❌ (²)

1. there is no requestPictureInPicture() method available in FF, only native button can be used (so we have no way to implement a custom control).
   scaleX(-1) & rotate(180deg) on the <video>-element works fine for the PiP-window (note that any non full-mirroring value is not applied to the PiP, but anyway this does the job for us).
   But: the current behaviour actually is a violation of the standard, which says:

   Styles applied to video (such as opacity, visibility, transform, etc.) MUST NOT apply in the Picture-in-Picture window. Its aspect ratio is based on the video size.

   *sigh*

2. for the life of it, i couldn't get :picture-in-picture class to do anything or even show up as being applied anywhere in the devtools, despite MDN indicating it is should be supported in Blink. Would be gladful if anybody could experiment with this and find out how to make it work and where the limitations are.

---

So, introducing custom fullscreen controls as suggested in #78 clearly is a viable solution to the fullscreen problem here, the fullscreen request just needs to be made on the container instead. There also is a simple explanation why it cannot work on the fullsceen element itself – there is a native browser style that prevents this:

:not(:root):fullscreen {
    /* … */
    transform: none !important;
    /* … */
}

PiP seems simply broken w/o a way to consistently implement the mirroring, unless somebody works out if something can be done via the :picture-in-picture-pseudo class.

[Philzen]

For full screen mode, are you just entering it with element.requestFullscreen() ?

BTW – in my tests above it did not seem to make any difference whether using the javascript trigger or the native button control.

[davidzhao]

@Philzen thank you for the detailed notes. That's interesting that the transform are displayed correctly when the container goes full screen. I think the right thing here is as you suggested: overriding the stylesheet to have custom fullscreen controls. Since most of these controls don't apply to WebRTC, I think v1 of custom controls could be just hiding them. What do you think?

P.S. Just a minor correction: Chromium uses Blink, not WebKit. I am curious how it works in Safari though.

[Philzen]

I think v1 of custom controls could be just hiding them. What do you think?

Unless we have custom controls that replace the native ones (for PiP and fullscreen – the others are of course not feasible), i would not hide the native ones. It may be better to have a feature with a known bug (which actually only applies to the local / mirrored stream) , than taking the option away for all users. Once they are implemented, it's fine to hide the native controls.

P.S. Just a minor correction: Chromium uses Blink, not WebKit. I am curious how it works in Safari though.

Oh yes, you're right (Blink is a forked off WebKit though). I'm updating my table above accordingly.

[davidzhao]

Curious, which controls do you find to be useful to the end user? In the context of live video.

[Philzen]

Fullscreen and PiP. Not sure if anybody ever would want to pause / freeze the video – that may be an interesting feature, but so far i've never seen it in video conferencing and never heard anyone ask for it 😉