RTSP support

Currently the Gateway has ONVIF support, but no RTSP support. What would it take to create an RTSP adapter? Are there things to look out for, or things to be aware of when potentially creating such an addon?

[benfrancis]

Currently the Gateway has ONVIF support, but no RTSP support.

I don't think that's quite correct. As I understand it the ONVIF adapter should support any ONVIF Profile S compatible camera, and all Profile S devices and clients must support RTSP.

RTSP just provides the media transport, ONVIF provides a wrapper around the RTSP stream which does a lot of other things like device discovery, authentication, capability description, network configuration, PTZ controls etc.

So I guess what you're really asking is whether an RTSP stream could be directly turned into a web thing, without using something like PSIA or ONVIF as a wrapper. I would guess that is possible, if users manually enter an rtsp:// URL it could be possible to create a Thing Description which exposes that video stream as a property (trans-coding the video as necessary) and offers some basic actions like play and pause.

I guess the more important question is what are you trying to achieve? Is there a device you'd like to support which doesn't support ONVIF, but does expose an RTSP stream directly?

[mrstegeman]

As an outside example, @atirage managed to get the Raspberry Pi camera module running as a native web thing: https://github.com/atirage/picamera-webthing

@benfrancis Interesting!

The main goal is to have a privacy-focussed doorbell that could also double as a security camera, where the shutter and microphone can be trusted to be off. A physical camera shutter is essential to really offer peace of mind.

Here's an early prototype:
https://www.candlesmarthome.com/exploring-a-privacy-friendly-smart-doorbell

That one's based on an ESP32, so it's not very powerful. It only supports motion jpeg, altough RTSP theoretically supports both MJPEG and H26X. So it should work?

A big advantage would be that it's cheap, safe to unplug, easy to create open source privacy-focused software for, and could be pre-programmed with a wifi password when flashed from Candle Manager (increasing transparency and trust).

A more powerful option would be to base this on a Raspberry Pi Zero W. This is probably a good idea, since it might also make the return audio channel easier. Theoretically it could even just run the WebThings Gateway software again (a security camera running Voco might offer attractive convergence). It also has downsides though. Having a larger attack surface is one. A principle for Candle projects is to use the 'least viable hardware', so if an ESP32 can handle it, that would be the preference. It's all research in progress.

I was thinking of creating an addon that receives a stream from the doorbell/cameras. It could store streams to the SD card or (ideally) an externally plugged in drive. This would be done on a rotation basis, or when a face/movement is detected. The addon should allow a user to look at and listen to this stream live, which might be done through the existing support, if it's easy to then quickly press an "unlock the door" button.

In the case of the doorbell, it should ideally also return speech to a tiny speaker (bluetooth? ;-) ), or otherwise allow the user to trigger pre-recorded audio ("please wait a second", "not interested"). Does the current UI allow for streaming audio back from the Gateway UI to an endpoint?

Privacy is once again the USP here so:

• The addon should allow users to delete footage easily.
• The addon should allow for time limits on when recording should be active.
• The addon should have to option to only record when nobody is home.
• The addon should be able to NOT record footage if certain conditions are met. For example, if a face is detected, but a family member's phone has also just appeared on the network (according to the network presence detection addon), then it's probably that family member arriving home, and then there should be an option to not record that.

To complete the picture: the idea is to also have the Raspberry Pi create its own wifi accesspoint, so as to enforce that these devices are on a separate WiFi network from the home network. Especially because a camera doorbell literally hangs outside your house. It's important that if that device gets stolen, only that secondary wifi network is compromised (if its wifi password is extracted). In this doom scenario the user would have to generate a new network password and reflash the devices that were connecting to it so that they all have the new password again. Not having smart home devices communicate over the local network wherever possible is another one of Candle's principles.

Sumarily put, the question this project asks is the same that all Candle experiments have asked: can we have some level of convenience (and security in this case) without blindly creating a surveillance culture in and around the home?

// end of plan / pitch :-)

[benfrancis]

Here's an early prototype:
https://www.candlesmarthome.com/exploring-a-privacy-friendly-smart-doorbell

If you're building your own device, why not make it a native web thing which exposes the VideoCamera capability directly and therefore doesn't need an adapter? You could look at what the ONVIF adapter does to expose a web-viewable video stream via a Thing Description and ideally natively output a web-viewable video stream so that the gateway doesn't need to transcode it (which is very resource intensive).

Does the current UI allow for streaming audio back from the Gateway UI to an endpoint?

No I don't think so, but the proposed MediaPlayer capability might be able to handle that.

[flatsiedatsie]

That's a good idea, thanks, I'll look into it.

[flatsiedatsie]

This looks interesting for a return audio channel: recording audio in JS.

[flatsiedatsie]

I'm getting an error that I'm not sure how to fix:

2021-02-23 16:05:07.201 ERROR  : Invalid message received: {
  "messageType": 8192,
  "data": {
    "adapterId": "candlecam",
    "device": {
      "id": "candlecam",
      "title": "Candlecam",
      "@context": "https://webthings.io/schemas",
      "@type": [
        "VideoCamera"
      ],
      "description": "A privacy friendly smart doorbell or security camera",
      "properties": {
        "stream": {
          "name": "stream",
          "value": null,
          "visible": true,
          "type": null,
          "@type": "VideoProperty",
          "readOnly": true,
          "links": [
            {
              "rel": "alternative",
              "href": "http://192.168.2.167:8000/stream.mjpg",
              "mediaType": "x-motion-jpeg"
            }
          ]
        },
        "snapshot": {
          "name": "snapshot",
          "value": null,
          "visible": true,
          "type": null,
          "@type": "ImageProperty",
          "readOnly": true,
          "links": [
            {
              "rel": "alternative",
              "href": "/extensions/candlecam/photos/latest.jpg",
              "mediaType": "image/jpeg"
            }
          ]
        }
      },
      "actions": {},
      "events": {},
      "links": [],
      "baseHref": "",
      "pin": {
        "required": false,
        "pattern": ""
      },
      "credentialsRequired": false
    },
    "pluginId": "candlecam"
  }
}
2021-02-23 16:05:07.202 ERROR  : Validation error: [
  {
    "keyword": "type",
    "dataPath": ".data.device.properties['stream'].type",
    "schemaPath": "#/properties/type/type",
    "params": {
      "type": "string"
    },
    "message": "should be string"
  }
]

But shouldn't type be None if it's a video stream?

[flatsiedatsie]

Setting it to a string of 'null' solved the warning. I still don't see the media show up, but that's another issue.

[flatsiedatsie]

I've managed to get an image to show.

• Do videostreams have to be served up from the local /media folder somehow?
• Is there a way for an addon to be notified if the user clicks on the refresh snapshot button?

[atirage]

@flatsiedatsie, for the video streams you need to set up a Tornado handler to serve that URL, i.e. the stream is not a static resource like the snapshot.
Have a look here for an example:
https://github.com/atirage/picamera-webthing/blob/aa782fcc7723d636a2cab0c296aa3babfaa37a8a/picamera-webthing.py#L16

[flatsiedatsie]

Thanks @atirage. Your work has actually been the inspiration for what I tried to do, after it was recommended to me earlier in this thread.

I have used this example from picamera to create an mjpg server. It works great if I open the link directly. So my hope was that the Gateway would just set the mjpg stream URL as the src of the image object. But it seems that's not the case.
https://picamera.readthedocs.io/en/release-1.13/recipes2.html#web-streaming

[flatsiedatsie]

I guess it would require the gateway to use an img element instead of a video element, since it seems the video element can't handle motion jpeg. Weird.

If I hack in an image element and set the mjpeg url as it's source...

So close :-)

[flatsiedatsie]

I've tried to create a hybrid ffmpeg command based on the settings in the onvif-adapter.

ffmpeg -input_format yuyv422 -fflags nobuffer -vsync 0 -f video4linux2 -s 1280x720 -r 10 -i /dev/video0 -f alsa -ac 1 -ar 44100 -i hw:1,0 -map 0:0 -map 1:0 -c:a aac -b:a 96k -c:v h264_omx -r 10 -b:v 800k -copyts -probesize 200000 -window_size 5 -extra_window_size 10 -use_timeline 1 -use_template 1 -hls_playlist 1 -format_options movflags=empty_moov+omit_tfhd_offset+frag_keyframe+default_base_moof -seg_duration 1 -dash_segment_type mp4 -f dash ~/.webthings/media/candlecam/stream/index.mpd -remove_at_exit 1 -loglevel quiet

This leads to the video element having some source, although something is clearly still wrong.

FFMPEG seems to be chugging along:

2021-02-23 22:55:27.331 ERROR  : candlecam: frame= 6126 fps= 10 q=-0.0 size=N/A time=448365:45:14.77 bitrate=N/A speed=2.64e+06x    
2021-02-23 22:55:27.837 ERROR  : candlecam: frame= 6131 fps= 10 q=-0.0 size=N/A time=448365:45:14.77 bitrate=N/A speed=2.63e+06x    
2021-02-23 22:55:28.238 ERROR  : candlecam: [dash @ 0x165c2a0] Opening '/home/pi/.webthings/media/candlecam/stream/index.mpd.tmp' for writing
2021-02-23 22:55:28.240 ERROR  : candlecam: [dash @ 0x165c2a0] Opening '/home/pi/.webthings/media/candlecam/stream/media_0.m3u8.tmp' for writing
2021-02-23 22:55:28.240 ERROR  : candlecam: [dash @ 0x165c2a0] Opening '/home/pi/.webthings/media/candlecam/stream/chunk-stream0-00512.m4s.tmp' for writing

The MPD file looks like this:

<?xml version="1.0" encoding="utf-8"?>
<MPD xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xmlns="urn:mpeg:dash:schema:mpd:2011"
	xmlns:xlink="http://www.w3.org/1999/xlink"
	xsi:schemaLocation="urn:mpeg:DASH:schema:MPD:2011 http://standards.iso.org/ittf/PubliclyAvailableStandards/MPEG-DASH_schema_files/DASH-MPD.xsd"
	profiles="urn:mpeg:dash:profile:isoff-live:2011"
	type="dynamic"
	minimumUpdatePeriod="PT1S"
	suggestedPresentationDelay="PT1S"
	availabilityStartTime="2021-02-23T21:45:14Z"
	publishTime="2021-02-23T21:47:05Z"
	timeShiftBufferDepth="PT6.0S"
	minBufferTime="PT2.4S">
	<ProgramInformation>
	</ProgramInformation>
	<Period id="0" start="PT0.0S">
		<AdaptationSet id="0" contentType="video" segmentAlignment="true" bitstreamSwitching="true">
			<Representation id="0" mimeType="video/mp4" codecs="avc1.640028" bandwidth="2000000" width="1280" height="720" frameRate="10/1">
				<SegmentTemplate timescale="10240" initialization="init-stream$RepresentationID$.m4s" media="chunk-stream$RepresentationID$-$Number%05d$.m4s" startNumber="88">
					<SegmentTimeline>
						<S t="1069056" d="12288" r="4" />
					</SegmentTimeline>
				</SegmentTemplate>
			</Representation>
		</AdaptationSet>
		<AdaptationSet id="1" contentType="audio" segmentAlignment="true" bitstreamSwitching="true">
			<Representation id="1" mimeType="audio/mp4" codecs="mp4a.40.2" bandwidth="96000" audioSamplingRate="48000">
				<AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="1" />
				<SegmentTemplate timescale="48000" initialization="init-stream$RepresentationID$.m4s" media="chunk-stream$RepresentationID$-$Number%05d$.m4s" startNumber="1">
					<SegmentTimeline>
					</SegmentTimeline>
				</SegmentTemplate>
			</Representation>
		</AdaptationSet>
	</Period>
</MPD>

[flatsiedatsie]

I've managed to get it working. Kind of.

ffmpeg -y -f alsa -ac 1 -ar 44100 -i hw:1,0  -f v4l2 -fflags nobuffer -vsync 0 -video_size 640x480 -framerate 10 -i /dev/video0 -muxdelay 0 -vcodec h264_omx -keyint_min 0 -g 10 -map 0:a -c:a aac -b:a 96k  -map 1:v -b:v 400k -video_track_timescale 9000 -f dash -seg_duration 1 -use_template 1 -use_timeline 1 -remove_at_exit 1 -window_size 6 -extra_window_size 10 /home/pi/.webthings/addons/candlecam/stream/index.mpd

I can see that stream in Shaka player, which I'm using in a UI extension. My goal was to have the stream work both in the UI extension and in the thing itself. But it still doesn't work if I click on the stream property on the thing itself. It still stuck on the loading animation. Using the /media/candlecam/stream/index.mpd location didn't solve that.

• It seems that the Gateway also uses the Shaka player, so I was hoping I could re-use the library. I wasn't able to manage that though. From that I can gleam here, it seems that webpack has to be told that re-use of the library is permitted. Is that something to explore? Or does that not really have any memory benefits, and should the addon just load its own copy?
• Are there limitations to where the index.mpd file may be placed in the file system? I needed to have access to the stream in the GUI of the addon, so using /home/pi/.webthings/addons/candlecam/stream/index.mpd allowed that file to be accessible. But I suspect I'll have to follow the blob methodology already used by the current video player to make the stream accessible through the tunnel?

The lowest delay I've been able to get is between 5 and 6 seconds (including telling Shaka player to buffer as little as possible).

• Is this also your experience with the onvif adapter?

• Could somewhat more 'realtime' video technically be possible? E.g. by igoring the tunneling infrastructure on the local network? Or through WebRTC somehow? This baby monitor project uses that, and it's very close to what I'm aiming for.

[flatsiedatsie]

It's working! Man this was a tough one.

To answer one of my own questions for future reference:

Are there limitations to where the index.mpd file may be placed in the file system

No, but the limitation is that href in the thing's json can't be set to an absolute url. the first part of these URL's is always formed by the somewhat mysterious base variable, which provides the first part of the URL. Because of that, the thing json and the media files are expected to come from the same port. So you can't have a thing providing json on port 8888, and then refer to a stream file on port 8080. It's a package deal.

For me, the trick was to first stop using the addon route to create things, and instead use the thing url adapter route. Secondly I had to make the index.pdb file available on port 8888 as well. For that I had to add some additional routes to the python webthing's webserver. Luckily, this was easy to do.

        more_routes = [
            (r"/media/candlecam/(.*)", tornado.web.StaticFileHandler, {"path": "/home/pi/.webthings/media/candlecam"})
        ]
        
        thing_server = WebThingServer(SingleThing(thing), port=8888, additional_routes=more_routes)

Phew