Static sensor identifier for reporter configuration (thinkspeak for example)

[JavierAder]

Description

Hi. Several modifications that I am proposing allow creating new sensors in runtime; now, code associated with reports (thinkspeak for example), associates the reported values ​​to the magnitude index. This index depends on the number of sensors (and slots) in the system and changes from execution to execution if the number of sensors changes.
My idea is to be able to associate each instance of a sensor with a unique static identifier (which is maintained from execution to execution), and which, together with the slot being reported, can be univocally used by the code that makes the reports.
This would allow adding or removing sensors, without modifying the configuration of the reporter.
In the specific case of thinkspeak, I think it could be done modifying

espurna/code/espurna/thingspeak.cpp

Line 107 in 1366713

size_t magnitude(size_t index) {

#if SENSOR_SUPPORT
size_t magnitude(size_t index) {
    return getSetting({FPSTR(keys::Magnitude), index}, build::Unset);
}
#endif

to something like this

#if SENSOR_SUPPORT
size_t magnitude(size_t index) {
    auto& magnitude = magnitude::get(index);
   int instanceSensorId = magnitude.sensor->staticInstId();
   if (instanceSensorId <=0)
    return getSetting({FPSTR(keys::Magnitude), index}, build::Unset);
   String prefixWithId = FPSTR(keys::Magnitude) + "_"+ instanceSensorId + "_";
   return getSetting({prefixWithId, magnitude.slot}, build::Unset);
}
#endif

In this way, the association of thingspeak fields with sensors that support static ID would be stored in configuration under a key with the form
tspkMagnitude_ID_slot

In this specific case I don't know what other code would have to be modified; I don't know where these keys are written (do I have to modify the code associated with the web?)
Beyond this, I don't know if there is perhaps a more general and elegant solution that I'm not seeing (perhaps associating the static id with the magnitudes instead of the sensors?).
Any suggestions?

Solution

No response

Alternatives

No response

Additional context

No response

[mcspr]

In this way, the association of thingspeak fields with sensors that support static ID would be stored in configuration under a key with the form
tspkMagnitude_ID_slot

In current code, keys are always static. Either as a plain string, or string plus integer index. Index is generated on demand. Key contents are never parsed or split to parse certain parts separately.

I think you are also mixing up the indexing used for and between modules. Thingspeak as-is iterates over available magnitudes, since it is the only index it has available.

Would it be appropriate to replace explicit indexing with a separate 'entities' inside of Thingspeak module itself, not basing it on magnitudes count?

tspkMagnitude0 => 1 // current version

tspkIndex0 => 1 // first tspk outputs to index 1
tspkMagnitude0 => device/foo/bar // tspk data gathered from 'device/foo/bar' magnitude
tspkMagnitude0 => 0 // tspk data gathered from 0th magnitude
tspkMagnitude0 => volt0 // tspk data from 0th voltage magnitude (when available)

My idea is to be able to associate each instance of a sensor with a unique static identifier (which is maintained from execution to execution), and which, together with the slot being reported, can be univocally used by the code that makes the reports.
This would allow adding or removing sensors, without modifying the configuration of the reporter.
...
In this specific case I don't know what other code would have to be modified; I don't know where these keys are written (do I have to modify the code associated with the web?)
Beyond this, I don't know if there is perhaps a more general and elegant solution that I'm not seeing (perhaps associating the static id with the magnitudes instead of the sensors?).
Any suggestions?

Meaning, modules stop tracking indexes and track some kind of addressable magnitude; either as index, or some unique string.

Another thing to keep in mind is multiple instances of the same sensor, which would still (at least based on current code) depend on ordering. Some sensors provide addresses (I2C, port-based), so it might help in some cases

[JavierAder]

In current code, keys are always static. Either as a plain string, or string plus integer index. Index is generated on demand. Key contents are never parsed or split to parse certain parts separately.

I think you are also mixing up the indexing used for and between modules. Thingspeak as-is iterates over available magnitudes, since it is the only index it has available.

I must have expressed myself wrong; by index I mean the one passed in this line of code

espurna/code/espurna/sensor.cpp

Line 4325 in 4c3122c

tspkEnqueueMagnitude(index, value.repr);

not the indexes of the keys in configuration. That index can clearly change from execution to execution if the number of sensors is modified.
Thinkspeak currently maps that index to the respective thinkspeak field (0 when the magnitude is not associated with a field).

Would it be appropriate to replace explicit indexing with a separate 'entities' inside of Thingspeak module itself, not basing it on magnitudes count?

tspkMagnitude0 => 1 // current version

tspkIndex0 => 1 // first tspk outputs to index 1
tspkMagnitude0 => device/foo/bar // tspk data gathered from 'device/foo/bar' magnitude
tspkMagnitude0 => 0 // tspk data gathered from 0th magnitude
tspkMagnitude0 => volt0 // tspk data from 0th voltage magnitude (when available)

I don't know if I understand your idea exactly, but I think that should be the way.
Actually, thinkspeak should store what it reports in each field; instead of storing the magnitude->field association, I find it more natural to store
field->"something identifying a magnitude or a sensor slot".
For example:
tskpField1 ->m0 //to report the magnitude with index 0 in field 1, for backward compatibility
or
tskpField1 ->s123s0 //to report slot 0 of the sensor with instance id 123, in case the sensor supports unique instance ids (the current sensor id is not useful since different instances of the same sensor will have the same id)
The biggest problem I see in any case is how to modify the web code to use a new way of updating keys.

[JavierAder]

The above assumes that the number of sensors can change, but the number of slots per sensor cannot; a more general solution I think would be in BaseSensor:

int staticInstId()
{return _staticInstId; //0 default}

//returns the static id associated with the slot
int staticSlotId(int slot)
{
//by default the slot id is its own index; subclasses must redefine this if they want another behavior
return slot;
}

and what thinkspeask would actually store would be
"s"+sensor->staticInstId()+"s"+sensor->staticSlotId(slot)

PS: "Static Instance Id" is possibly incorrect; "Persistent Instance Id" actually better reflects my idea.

[mcspr]

tskpField1 ->m0 //to report the magnitude with index 0 in field 1, for backward compatibility
or
tskpField1 ->s123s0 //to report slot 0 of the sensor with instance id 123, in case the sensor supports unique instance ids (the current sensor id is not useful since different instances of the same sensor will have the same id)
The biggest problem I see in any case is how to modify the web code to use a new way of updating keys.

If you are identifying magnitude, just pass along this unique key when initializing it in the web? Fields displayed on the home page associate via 'index', but could just as well be identified via this unique id.
But yes, this is the approach I meant above.

and what thinkspeask would actually store would be
"s"+sensor->staticInstId()+"s"+sensor->staticSlotId(slot)

PS: "Static Instance Id" is possibly incorrect; "Persistent Instance Id" actually better reflects my idea.

Right, but why API needs 2 more IDs? Any reason why address can not be used, or some other extra properties identifying the device? Slot is technically unique for the specific sensor, so not really sure what 'staticSlotId' supposed to do here. You also have to keep in mind a possibility of 'setAddress()' for the sensor, where it can be generated by the user.

Another approach is to pass identifying string to the sensor directly, i.e.

sensor::find_slot(match_string) -> unsigned char {
  ...
  returns slot_id or unknown;
}

magnitude::matches(match_string) -> bool

Where this could do whatever implementation wise.
'address=0xca'
`address=0xcb,type=i2c'
'version=0xfe'
'etc=...'

[mcspr]

re. 'Persistent'... this also reminds me of the udev naming schemes for network devices, which are picked up from other unique properties of the network device. e.g. port, MAC, hw path or explicitly labeled

https://www.freedesktop.org/software/systemd/man/latest/systemd.net-naming-scheme.html

Network interfaces names and MAC addresses may be generated based on certain stable interface attributes. This is possible when there is enough information about the device to generate those attributes and the use of this information is configured. This page describes interface naming, i.e. what possible names may be generated. Those names are generated by the systemd-udevd.service(8) builtin net_id and exported as udev(7) properties (ID_NET_NAME_ONBOARD=, ID_NET_LABEL_ONBOARD=, ID_NET_NAME_PATH=, ID_NET_NAME_SLOT=).

following up the source of net_get_persistent_name(sd_device *)
https://github.com/systemd/systemd/blob/1f0e4af32919aaac1bef6f7fff5884db2e155bc8/src/shared/netif-util.c#L54

[JavierAder]

Ok, first of all I don't think this functionality is very important (after all, if you add or remove sensors, it doesn't add much more work to modify the configuration of thinkspeak or another reporter)... But anyway, I'm interested in learning how the web-related code works for others features (configuring AnalogInputs and NTCs sensors in particular).
Back to the topic:
One way I see to implement this type of id persistence is that when a thinkspeak field is modified from the web and Save is pressed, the webSocket message is taken and instead of saving the "tspkMagnitudeX" keys, other ones are saved (maybe tspkFieldX or something similar).
The current thinkspeak.cpp never sets its keys by calling setSetting(....), that is being done by another code in a generic way (which I still can't figure out which one).
Inspecting the websocket messages that occur when the field associated with a magnitude is modified, I got something like this:

{"settings":{"set":{"tspkMagnitude0":4},"del":[]}}
That message is generated by javascript in the browser.
My question is: how could I, from thinkspeak.cpp, intercept that message and update the configuration in another way?
I suspect that it would have to be done along these lines

espurna/code/espurna/thingspeak.cpp

Line 721 in 4c3122c

void setup() {

and use a handler for onData(on_send_f), something like that:

void setup() {
    wsRegister()
        .onKeyCheck(onKeyCheck)
        .onVisible(onVisible)
        .onConnected(onConnected)
        .onData(onData);
}

void onData(JsonObject& root) {
//starting from root set tspkFieldX keys or something similar using setSetting(...)
}

I don't know if it would be correct; onData is called when the Save button is pressed?

[JavierAder]

Ah, no, the magic happens here (apparently):

espurna/code/espurna/ws.cpp

Line 553 in 4c3122c

void _wsParse(AsyncWebSocketClient* client, uint8_t* payload, size_t length) {

I guess there is no way to intercept the webSocket message generated by the save button.....
I think the correct way would be to modify the web module associated with thinkspeak so that it generates its own action to save its configuration (similar to homeassistant's "Publish").
Something like

void setup() {
    wsRegister()
        .onKeyCheck(onKeyCheck)
        .onVisible(onVisible)
        .onConnected(onConnected)
        .onAction(onAction);
}

void onAction(uint32_t, const char* action, JsonObject& data) {

  if ("tskp_save_fields" == action)
  {
   //use data to set tspkFieldX keys or something similar using setSetting(...)
  }
}

[mcspr]

I guess there is no way to intercept the webSocket message generated by the save button.....

Same file, just a few lines below. Not sure how that helps here, though

espurna/code/espurna/ws.cpp

Lines 646 to 647 in 4c3122c

	if (_wsCheckKey(key, kv.value)) {
	if (_wsStore(key, kv.value.as<String>())) {

Ok, first of all I don't think this functionality is very important (after all, if you add or remove sensors, it doesn't add much more work to modify the configuration of thinkspeak or another reporter)

Only apparent issue is index shift, and leftover keys that possibly do nothing or overwrite old behaviour? Does not matter much for an end device, that is true. But could, in theory, be useful to limit other reading / reporting outputs e.g. MQTT and / or modify output topics to be more unique and identifying the sensor device itself vs. identifying Nth output

[JavierAder]

One way I see to add unique IDs per sensor instance is the following:
BaseSensor.h

public:
....
   // Persistent, unique id per instance of sensor
    // Format: id of class * 100 + intance id in class
    uint16_t persistentInstanceId()
    {
        if (_subInstId <= 0)
            _subInstId = createSubIdForSensorClass();
        return id()*100 + _subInstId; //by 100 for easy identification by user
    }
 ...

protected:

   uint8_t _subInstId = 0;
    //(sub) id of sensor instance within its class, must be less 100; 
    //override if necesary (sensors created in runtime for example)
    virtual uint8_t createSubIdForSensorClass() const{
        return createDefaultSubId(id());
    }

    //Not realy persistent if sensors number changes
    static uint8_t createDefaultSubId(char sensorClassId)
    {
            if (_subInstIdDefaultMap.find(sensorClassId) == _subInstIdDefaultMap.end())
            {
                _subInstIdDefaultMap[sensorClassId] = 0;
                return 0;
            }
            _subInstIdDefaultMap[sensorClassId] = _subInstIdDefaultMap[sensorClassId] + 1;
            return _subInstIdDefaultMap[sensorClassId];
    }
 
    static std::map<char, uint8_t> _subInstIdDefaultMap;
};

std::map<char, uint8_t> BaseSensor::_subInstIdDefaultMap = {};

....

It is backwards compatible (current sensor code does not require modification).New sensors, if they wish, can redefine createSubIdForSensorClass() to achieve truly persistent IDs.

[JavierAder]

Regarding how to use this in reporters like Thinkspeak, I think the website should use a new module: it currently uses "magnitudes-module." I think that module should be, say "field-to-sns-slot-module," which generates configuration keys in the form
prefixModule+IdField->"id sensor instance:slot"

For example, for Thinkspeak, it would generate (websocket message) something like

{"settings":{"set":{"tspkField0":"1900:01"},"del":[]}}
to report in field 1 (thinkspeak fields start at 1) the second slot (by :01) of the first sensor instance in the class with ID 19.
What the user on the website would actually select is a magnitude (in a dropdown html for example) for each field, since a magnitude uniquely identifies a sensor instance and a slot.

[JavierAder]

For example, in the case of Thinkspeak

Thinkspeak.cpp

....
PROGMEM_STRING(Field, "tskpField");
....

#if SENSOR_SUPPORT
bool enqueueMagnitude(size_t index, const String& value) {
    if (internal::enabled) {
        auto& magnitude = magnitude::get(index);
        unit16_t instId = magnitude.sensor->persistentInstanceId();
        unsigned char slot = magnitude.slot;
        String v = instId+":"+slot;
        bool find = false;
        for (size i =0;i<std::size(internal::fields);i++)
        {
            if (v.equals(getSetting({FPSTR(keys::Field), index},"")))
            {
                internal::fields[i] = value;
                find = true;
            }
        }
    }
    if (find)
    {   
        schedule_flush();
        return true;
    }
    return false;
}
#endif

Well, the code to update the tspkField keys is missing, but as is it should be possible to use it by setting these keys manually from the console. As soon as I have some free time I will test it.

[mcspr]

Seems reasonable. Note that you already have access to magnitude value struct when reporting from the sensors side, just attach pre-generated id string there.
edit: and I am pretty sure there is no need for the map. sensor instance itself is an entry, static member on the class can count. either way there is uncertain inheritance order, though, not yet sure what to do about that. or, could manage it same as MAGNITUDE type counter - externally.

Regarding how to use this in reporters like Thinkspeak, I think the website should use a new module: it currently uses "magnitudes-module." I think that module should be, say "field-to-sns-slot-module," which generates configuration keys in the form
prefixModule+IdField->"id sensor instance:slot"

Is the intention to replace existing numeric field? Why separate inputs instead of using the same one as a text field w/ datalist helper instead of selector?
(ref. timezone text input)

[JavierAder]

Seems reasonable. Note that you already have access to magnitude value struct when reporting from the sensors side, just attach pre-generated id string there. edit: and I am pretty sure there is no need for the map. sensor instance itself is an entry, static member on the class can count. either way there is uncertain inheritance order, though, not yet sure what to do about that. or, could manage it same as MAGNITUDE type counter - externally.

Regarding how to use this in reporters like Thinkspeak, I think the website should use a new module: it currently uses "magnitudes-module." I think that module should be, say "field-to-sns-slot-module," which generates configuration keys in the form
prefixModule+IdField->"id sensor instance:slot"

Is the intention to replace existing numeric field? Why separate inputs instead of using the same one as a text field w/ datalist helper instead of selector? (ref. timezone text input)

Are you referring to thinkspeak.cpp that generates something similar to the following HTML code?(Instead of the current code with lists of magnitudes)

<p>Select sensor for Thinkspeak fields:</p>
<label >Field 1</label>
<input list="sns-slot-list" id="tspkField1" name="tspkField1" />
<label >Field 2</label>
<input list="sns-slot-list" id="tspkField2" name="tspkField2" />
<label >Field 3</label>
<input list="sns-slot-list" id="tspkField3" name="tspkField3" />
//to fiel 8

...
<datalist id="sns-slot-list">
  <option value="" label="None></option>
  <option value="1900:00" label = "Description of slot 0 of first sensor wiht type 19"></option>
 <option value="1900:01" label = "Description of slot 1 of first sensor wiht type 19"></option>
  <option value="2000:00" label = "Description of slot 0 of first sensor wiht type 20"></option>
</datalist>

[JavierAder]

edit: and I am pretty sure there is no need for the map. sensor instance itself is an entry, static member on the class can count. either way there is uncertain inheritance order, though, not yet sure what to do about that. or, could manage it same as MAGNITUDE type counter - externally

do you meen that?
https://github.com/xoseperez/espurna/blob/dev/code/espurna/sensors/BaseSensor.h#L43
Kind is a static member and what I need a unique id per instance