Streaming from my cell phone camera to a rtmp server

[asokone]

Hi
I am new to stream development. I ask chatGPT to provide me with some code in python for streaming from my cell phone camera to a rtmp server. The RTMP server is working fine. I have tested with OBS and VLC Media Player.

When I executed this code on Jupyter Notebook I got this error below
Can someone help figure out what is wrong?

Regards
A Sokone.

------------------------- PYTHON CODE START HERE
import cv2
import gi

# ---- I added these lines
import ctypes
import numpy as np
import cairo
# ---- End of added lines

gi.require_version('Gst', '1.0')
from gi.repository import Gst

# Initialize GStreamer
Gst.init(None)

# Define the RTMP server URL
rtmp_server_url = 'rtmp://192.168.0.30/live'

# Set up the GStreamer pipeline
pipeline = Gst.Pipeline.new()

# Create the elements for video capture and encoding
src = Gst.ElementFactory.make("appsrc", "source")
caps = Gst.Caps.from_string("video/x-raw,format=BGR")
filter = Gst.ElementFactory.make("capsfilter", "filter")
filter.set_property("caps", caps)
video_convert = Gst.ElementFactory.make("videoconvert", "video_convert")
x264enc = Gst.ElementFactory.make("x264enc", "x264enc")
mux = Gst.ElementFactory.make("flvmux", "mux")
sink = Gst.ElementFactory.make("rtmpsink", "sink")
sink.set_property("location", rtmp_server_url)

# Add elements to the pipeline
for element in [src, filter, video_convert, x264enc, mux, sink]:
    pipeline.add(element)

# Link elements in the pipeline
src.link(filter)
filter.link(video_convert)
video_convert.link(x264enc)
x264enc.link(mux)
mux.link(sink)

# Start the pipeline
pipeline.set_state(Gst.State.PLAYING)

# Open the video capture device
cap = cv2.VideoCapture(0)

while True:
    ret, frame = cap.read()
    if not ret:
        break

    
    # Convert frame to Cairo surface
    height, width, _ = frame.shape

    stride = width * 1  # Assuming 3 channels (RGB)
    
    surface = cairo.ImageSurface.create_for_data(frame.data, cairo.FORMAT_RGB24, width, height)
    
    print("surface.get_data", surface.get_data())
    
    # Push frame to the GStreamer pipeline
    buf = Gst.Buffer.new_wrapped(surface.get_data())
    
    print("buf", buf)
    buf.pts = buf.dts = Gst.CLOCK_TIME_NONE
    buf.duration = 1
    src.emit("push-buffer", buf)

# Clean up
cap.release()
pipeline.set_state(Gst.State.NULL)

--------------------------- End of Python Code

Error below

TypeError Traceback (most recent call last)
Cell In[1], line 65
61 print ("height", height, "width", width, "stride", stride)
63 #surface = cairo.ImageSurface.create_for_data(frame.data, cairo.FORMAT_RGB24, width, height, stride)
---> 65 surface = cairo.ImageSurface.create_for_data(frame.data, cairo.FORMAT_RGB24, width, height)
67 print("surface.get_data", surface.get_data())
69 # Push frame to the GStreamer pipeline

TypeError: buffer is not long enough

[stuaxo]

It's hard to see what's happening in this format, Jupyter has an option do output to a single .py file, can you do that somewhere and it might be easier to read the code.

[stuaxo]

In my limited experience of this stuff: gpt 3.5 is much more likely to suggest code that doesn't work Vs 4.

You can also paste your error back into it and see what it says.

All of this is a little far away from pycairo specific advice.

[stuaxo]

OK, I stared at this more, the line stride = width * 1 looks wrong.

You should calculate it using stride_for_width(width: int) it's not going to be width * 1, except on an 8 bit per channel image.

Or you can just use 4 (cairo RGB uses 4 bytes as an optimisation, where one is unused).

[stuaxo]

create_for_data needs the input buffer to be the right width, I changed the line

height, width, _ = frame.shape

To

height, width, channels = frame.shape and channels is 3.

You need it to be 4, you can either add a zeros or use an API to convert the colours - I'd add zeros since the channel is unused

https://stackoverflow.com/a/54008568

I have no way of testing if this works, but you might end up with something like this:

import cv2
import gi

# ---- I added these lines
import ctypes
import numpy as np
import cairo
# ---- End of added lines

gi.require_version('Gst', '1.0')
from gi.repository import Gst

# Initialize GStreamer
Gst.init(None)

# Define the RTMP server URL
rtmp_server_url = 'rtmp://192.168.0.30/live'

# Set up the GStreamer pipeline
pipeline = Gst.Pipeline.new()

# Create the elements for video capture and encoding
src = Gst.ElementFactory.make("appsrc", "source")
caps = Gst.Caps.from_string("video/x-raw,format=BGR")
filter = Gst.ElementFactory.make("capsfilter", "filter")
filter.set_property("caps", caps)
video_convert = Gst.ElementFactory.make("videoconvert", "video_convert")
x264enc = Gst.ElementFactory.make("x264enc", "x264enc")
mux = Gst.ElementFactory.make("flvmux", "mux")
sink = Gst.ElementFactory.make("rtmpsink", "sink")
sink.set_property("location", rtmp_server_url)

# Add elements to the pipeline
for element in [src, filter, video_convert, x264enc, mux, sink]:
    pipeline.add(element)

# Link elements in the pipeline
src.link(filter)
filter.link(video_convert)
video_convert.link(x264enc)
x264enc.link(mux)
mux.link(sink)

# Start the pipeline
pipeline.set_state(Gst.State.PLAYING)

# Open the video capture device
cap = cv2.VideoCapture(0)
while True:
    ret, frame = cap.read()
    if not ret:
        break

    # frame = cv2.cvtColor(frame, cv2.COLOR_RGB2RGBA)
    frame = np.dstack((frame, np.zeros(frame.shape[:-1])))

    # Convert frame to Cairo surface
    height, width, channels = frame.shape

    stride = width * 4  # Hardcoded, but should consider using stride_for_width
    
    surface = cairo.ImageSurface.create_for_data(frame.data, cairo.FORMAT_RGB24, width, height, stride)
           
    # Push frame to the GStreamer pipeline
    buf = Gst.Buffer.new_wrapped(surface.get_data())
    
    print("buf", buf)
    buf.pts = buf.dts = Gst.CLOCK_TIME_NONE
    buf.duration = 1
    src.emit("push-buffer", buf)

[stuaxo]

@lazka in playing with this I actually hit the same problem as asok -
In going from some other API (cv2) to cairo, it's easy to accidentally have data for an image of the right dimensions, but the amount of channels is wrong.

I might see if create_for_data could output a little more info here if the size of the data is width / 3, which it should be width / 4 (or maybe just an error the the amount of channels in general)

PEP678 has a way of adding notes to exception which could be useful -
https://peps.python.org/pep-0678/