Video black screen after app returns from background during active video call (Android)

[narayan298]

Describe the bug

Environment

• PJSIP Version: 2.x (pjsua2 Android Java binding)
• Platform: Android
• Android API tested: API 29, 31, 33, 34
• Language: Java (Android)
• Build: pjsua2 JNI / AAR integrated into Android app
• Device tested: Multiple Android devices (Samsung, Oppo, stock Android)

---

Problem Description

During an active video call, when the user presses the Home button or switches
to another app (sending the app to background), and then returns to the app,
the video screen shows a complete black screen — both the local preview
(self camera) and the remote video (incoming stream) fail to display.

The SIP call itself remains alive and connected — audio continues working
normally. Only the video rendering breaks.

Steps to reproduce

1. Start the Android app
2. Make or receive a video call (pjsua2 with video enabled, videoCount > 0)
3. Wait for call to reach PJSIP_INV_STATE_CONFIRMED state
4. Confirm that both local preview and remote video are rendering correctly
5. Press Home button to send app to background
6. Wait 2–5 seconds
7. Tap the app icon or notification to bring app back to foreground
8. Result: Both local preview and remote video show black screen

PJSIP version

2.17

Context

• The issue happens on Samsung Galaxy S21, Oppo Reno, and stock Android
  (Pixel 6,Pixel 8) devices running Android 11, 12, 13,14 and 16.

• App is built using Android NDK r21 or newer.

• 'configure-android' param:
  TARGET_ABI=arm64-v8a

• 'config_site.h' contents:
  #define PJ_CONFIG_ANDROID 1
  #define PJMEDIA_HAS_VIDEO 1
  #define PJMEDIA_VIDEO_DEV_HAS_ANDROID_OPENGL 0
  #define PJMEDIA_HAS_OPENH264_CODEC 1
  #define PJMEDIA_HAS_LIBYUV 1
  #include <pj/config_site_sample.h>

• Related third-party lib & version:
  OpenSSL 1.1.1b
  pjsua2 Android AAR (pjsip 2.13 or 2.14)
  Android Gradle Plugin 8.x
  compileSdk 34, minSdk 24

• Applied patch(es): None

Log, call stack, etc

## Actual Behavior

- Both `surfacePreviewCapture` (local) and `surfaceIncomingVideo` (remote) 
  show solid black
- No crash or exception is thrown
- Audio continues working normally
- PJSIP call state remains `PJSIP_INV_STATE_CONFIRMED`
- Calling `vidPrev.getVideoWindow()` returns a non-null `VideoWindow` object
- Calling `setWindow()` on the `VideoWindowHandle` does not throw but has no visual effect


## What We Have Tried

### 1. Blocking and unblocking surface handlers on pause/resume
We implemented a custom `SurfaceHolder.Callback` wrapper (`VideoSurfaceHandler`) 
that blocks all PJSIP calls in `onPause` and unblocks in `onResume`. 
On `block()` we null the `videoWindow` reference to force a fresh fetch on resume.

### 2. Stopping and restarting video transmit
In `onPause`:

currentCall.vidSetStream(
    pjsua_call_vid_strm_op.PJSUA_CALL_VID_STRM_STOP_TRANSMIT,
    callVidPrm
);

In `onResume`:

currentCall.vidSetStream(
    pjsua_call_vid_strm_op.PJSUA_CALL_VID_STRM_START_TRANSMIT,
    callVidPrm
);

This keeps the call alive but video still does not render on screen after resume.

### 3. Re-fetching VideoWindow and calling setWindow after resume
After `onResume`, we fetch a fresh `VideoWindow` from `vidPrev.getVideoWindow()` 
and from `CallMediaInfo.getVideoWindow()`, then call:

VideoWindowHandle wh = new VideoWindowHandle();
wh.getHandle().setWindow(surface);
videoWindow.setWindow(wh);

The call completes without exception but the surface remains black.

### 4. Using SurfaceHolder callbacks to reattach
We use `surfaceCreated()` callback to detect when Android recreates the surface 
after returning from background, then immediately reattach the PJSIP video window.
The callback fires correctly but video still does not render.

### 5. Retrying with delays
We tried delays of 300ms, 500ms, 600ms, 1000ms before calling `setWindow()`.
None of the delays consistently fix the issue.

### 6. Calling reinvite after resume
We tried calling `currentCall.reinvite(param)` after returning to foreground.
This sometimes fixes the remote video but causes a brief call interruption 
which is not acceptable UX.

### 7. Z-order management
We ensured correct Z-order:

surfaceIncomingVideo.setZOrderOnTop(false);
surfacePreviewCapture.setZOrderOnTop(true);
surfacePreviewCapture.setZOrderMediaOverlay(true);

Z-order is not the cause.

---

## Key Technical Questions

**Q1.** When an Android app goes to background and `SurfaceHolder.surfaceDestroyed()` 
is called, does PJSIP internally release or invalidate the `VideoWindow` native 
object? Or does it remain valid and reusable?

**Q2.** After returning to foreground and `surfaceCreated()` fires with a new 
`Surface`, is it safe and sufficient to call `videoWindow.setWindow(newHandle)` 
with the new surface — or does PJSIP require any additional steps to re-render?

**Q3.** Is there a recommended PJSIP API sequence for handling 
**background → foreground** transitions during an active video call on Android? 
For example, should we:
- Call `PJSUA_CALL_VID_STRM_STOP_TRANSMIT` then `START_TRANSMIT`?
- Call `reinvite()`?
- Destroy and recreate the video window entirely?
- Something else?

**Q4.** Does `vidPrev.getVideoWindow()` always return the same native `VideoWindow` 
object for the lifetime of the call, or can it change after a background/foreground 
cycle? Should we cache the reference or always re-fetch it?

**Q5.** Is there a known issue or limitation with `VideoWindow.setWindow()` being 
called after the Surface has been destroyed and recreated by Android? Are there 
any internal PJSIP flags or state that prevent re-rendering after surface recreation?

**Q6.** For `CallMediaInfo.getVideoWindow()` (remote video), the same question — 
does the `VideoWindow` object remain valid after surface destruction, or does it 
need to be re-obtained via `getInfo()` after every background/foreground cycle?

---

## Relevant Code

### onPause (background handling)

@Override
protected void onPause() {
    super.onPause();

    // Block surface handler — stops all PJSIP surface calls
    if (localVideoHandler != null) localVideoHandler.block();
    if (remoteVideoHandler != null) remoteVideoHandler.block();

    // Stop video transmit to release camera gracefully
    if (videoCount > 0 && confirmedCallView && currentCall != null) {
        try {
            CallVidSetStreamParam callVidPrm = new CallVidSetStreamParam();
            callVidPrm.setCapDev(isFront ? 2 : 1);
            currentCall.vidSetStream(
                pjsua_call_vid_strm_op.PJSUA_CALL_VID_STRM_STOP_TRANSMIT,
                callVidPrm
            );
            wasVideoPausedByBackground = true;
        } catch (Exception e) {
            Log.e(TAG, "onPause stop transmit: " + e.getMessage());
        }
    }
}


### onResume (foreground handling)

@Override
protected void onResume() {
    super.onResume();

    // Unblock handlers
    if (localVideoHandler != null) localVideoHandler.unblock();
    if (remoteVideoHandler != null) remoteVideoHandler.unblock();

    // Resume video transmit
    if (wasVideoPausedByBackground && currentCall != null) {
        try {
            CallVidSetStreamParam callVidPrm = new CallVidSetStreamParam();
            callVidPrm.setCapDev(isFront ? 2 : 1);
            currentCall.vidSetStream(
                pjsua_call_vid_strm_op.PJSUA_CALL_VID_STRM_START_TRANSMIT,
                callVidPrm
            );
            wasVideoPausedByBackground = false;
        } catch (Exception e) {
            Log.e(TAG, "onResume start transmit: " + e.getMessage());
        }
    }

    // Reattach video surfaces
    new Handler(Looper.getMainLooper()).postDelayed(() -> {
        forceReattachAllVideoSurfaces();
    }, 300);
}


### Surface reattach logic

private void forceReattachAllVideoSurfaces() {
    MyCall safeCall = currentCall;
    if (safeCall == null) return;

    // Local preview
    if (safeCall.vidPrev != null) {
        VideoWindow localVw = safeCall.vidPrev.getVideoWindow();
        if (localVw != null) {
            localVideoHandler.resetVideoWindow();
            binding.surfacePreviewCapture.post(() -> {
                localVideoHandler.setVideoWindow(localVw);
            });
        }
    }

    // Remote video
    CallInfo ci = safeCall.getInfo();
    for (int i = 0; i < ci.getMedia().size(); i++) {
        CallMediaInfo cmi = ci.getMedia().get(i);
        if (cmi.getVideoIncomingWindowId() == pjsua2.INVALID_ID) continue;
        VideoWindow remoteVw = cmi.getVideoWindow();
        if (remoteVw == null) continue;
        remoteVideoHandler.resetVideoWindow();
        binding.surfaceIncomingVideo.post(() -> {
            remoteVideoHandler.setVideoWindow(remoteVw);
        });
        break;
    }
}


### setWindow call inside surface handler

private void applySurfaceNow(SurfaceHolder sh) {
    try {
        VideoWindowHandle wh = new VideoWindowHandle();
        boolean valid = sh != null && sh.getSurface() != null && sh.getSurface().isValid();
        wh.getHandle().setWindow(valid ? sh.getSurface() : null);
        videoWindow.setWindow(wh);
    } catch (Exception e) {
        Log.e(TAG, "applySurfaceNow failed: " + e.getMessage());
        active = false;
        videoWindow = null;
    }
}


---

## Additional Notes

- The issue is **100% reproducible** on all tested Android devices
- The issue happens specifically when the **Surface is destroyed and recreated** 
  by Android during the background/foreground transition
- No `CameraAccessException` is thrown — the camera hardware appears to resume 
  correctly at the OS level
- PJSIP native logs show no errors during the transition
- The `VideoWindow` object references appear valid (non-null, no exception on access)
  but produce no visual output after reattachment
- We are using a **Foreground Service** to keep the process alive in background,
  so the issue is not caused by process death

---

## Request

We would greatly appreciate:
1. Official documentation or sample code for handling 
   **background/foreground lifecycle** in Android video calls
2. Clarification on the correct API sequence to reattach video surfaces 
   after Android destroys and recreates them
3. Any known workarounds or patches for this specific scenario

Thank you for your time and for maintaining PJSIP.