Add correction for padding multiplier in "Verteilung TRL"

[daimpi]

The plots in "Verteilung Transmission Risk Level (TRL) in Diagnoseschlüsseln" currently use the number of keys transmitted including the padded fake keys afaiu. As long as the padding factor stays the same this shouldn't be a problem. But this factor will change from tomorrow on (the plan is to bring it down to 1 eventually). The changes in the padding multiplier will cause some distortion in those graphs as new data will receive less weight.

My suggestion would be to use the data which has been corrected for this multiplier like in the "Geteilte Diagnoseschlüssel von positiv getesteten Personen" section.
@mh- has introduced an automatic detection for the multiplier used in the data set in his parsing tool: corona-warn-app/cwa-server#620 (comment)

[cfritzsche]

Is the tool by mh- able to see if the padding change really applied for the whole daily package? The upload numbers for today seem to be quite high. If that is a real increase I am happy ;-)

[mh-]

Is the tool by mh- able to see if the padding change really applied for the whole daily package? The upload numbers for today seem to be quite high. If that is a real increase I am happy ;-)

No, the new multiplier 5 was applied during the day, so for more correct values you would have to use the hourly key packages. 2 or 3 of these still used 10.

[micb25]

No, the new multiplier 5 was applied during the day, so for more correct values you would have to use the hourly key packages. 2 or 3 of these still used 10.

It was changed at 11 AM CEST (9 AM UTC; package 9; vide infra)

Is the tool by mh- able to see if the padding change really applied for the whole daily package? The upload numbers for today seem to be quite high. If that is a real increase I am happy ;-)

Good news! I checked this twice and also uploaded the hourly packages. The number of users seems to be correct:

sum of hourly packages:
3+2+5+8+7+4+3+5 = 37 users

---

daily package:

37 user(s) found.
They submitted these numbers of keys:
4 user(s): 1 Diagnosis Key(s)
3 user(s): 4 Diagnosis Key(s)
1 user(s): 5 Diagnosis Key(s)
1 user(s): 6 Diagnosis Key(s)
1 user(s): 7 Diagnosis Key(s)
1 user(s): 8 Diagnosis Key(s)
1 user(s): 9 Diagnosis Key(s)
25 user(s): 13 Diagnosis Key(s)
80 keys not parsed (16 without padding).
37 / 4*1, 3*4, 1*5, 1*6, 1*7, 1*8, 1*9, 25*13

---

hourly package 6:

Length: 390 keys
Padding Multiplier detected: 10
3 user(s) found.
They submitted these numbers of keys:
3 user(s): 13 Diagnosis Key(s)
0 keys not parsed (0 without padding).
3 / 3*13

---

hourly package 7:

Length: 260 keys
Padding Multiplier detected: 10
2 user(s) found.
They submitted these numbers of keys:
2 user(s): 13 Diagnosis Key(s)
0 keys not parsed (0 without padding).
2 / 2*13

---

hourly package 9:

Length: 220 keys
Padding Multiplier detected: 5
5 user(s) found.
They submitted these numbers of keys:
1 user(s): 1 Diagnosis Key(s)
1 user(s): 4 Diagnosis Key(s)
3 user(s): 13 Diagnosis Key(s)
0 keys not parsed (0 without padding).
5 / 1*1, 1*4, 3*13

---

hourly package 11:

Length: 200 keys
Padding Multiplier detected: 5
8 user(s) found.
They submitted these numbers of keys:
1 user(s): Invalid Transmission Risk Profile
2 user(s): 1 Diagnosis Key(s)
1 user(s): 3 Diagnosis Key(s)
2 user(s): 4 Diagnosis Key(s)
1 user(s): 8 Diagnosis Key(s)
1 user(s): 13 Diagnosis Key(s)
Old Android app used by 1 user(s).
30 keys not parsed (6 without padding).
8 / 2*1, 1*3, 2*4, 1*8, 1*13 (1 old Android app(s))

---

hourly package 14:

Length: 345 keys
Padding Multiplier detected: 5
7 user(s) found.
They submitted these numbers of keys:
1 user(s): 1 Diagnosis Key(s)
1 user(s): 7 Diagnosis Key(s)
1 user(s): 9 Diagnosis Key(s)
4 user(s): 13 Diagnosis Key(s)
0 keys not parsed (0 without padding).
7 / 1*1, 1*7, 1*9, 4*13

---

hourly package 15:

Length: 195 keys
Padding Multiplier detected: 5
4 user(s) found.
They submitted these numbers of keys:
1 user(s): 1 Diagnosis Key(s)
1 user(s): 8 Diagnosis Key(s)
2 user(s): 13 Diagnosis Key(s)
20 keys not parsed (4 without padding).
4 / 1*1, 1*8, 2*13

---

hourly package 16:

Length: 195 keys
Padding Multiplier detected: 5
3 user(s) found.
They submitted these numbers of keys:
3 user(s): 13 Diagnosis Key(s)
0 keys not parsed (0 without padding).
3 / 3*13

---

hourly package 19:

Length: 155 keys
Padding Multiplier detected: 5
5 user(s) found.
They submitted these numbers of keys:
1 user(s): Invalid Transmission Risk Profile
1 user(s): 1 Diagnosis Key(s)
1 user(s): 5 Diagnosis Key(s)
1 user(s): 7 Diagnosis Key(s)
1 user(s): 13 Diagnosis Key(s)
Old Android app used by 1 user(s).
25 keys not parsed (5 without padding).
5 / 1*1, 1*5, 1*7, 1*13 (1 old Android app(s))

[Tho-Mat]

I assume the package at 19: has only 3 users.
Old android should no longer be possible after pushing server version 1.0.9 online
corona-warn-app/cwa-server#640

one user 13 keys (1.7-19.6), 1 user 12 keys (1.7-19.6; has no key for 24.6), and 1 user 6 keys (1.7-26.6)

or 4 Users if no hole is allowed:
one user 13 keys (1.7-19.6), 1 user 7 keys (1.7- 25.6), 1 user 6 keys (1.7-26.6), and 1 user 5 keys (23.6-19.6)

[micb25]

I assume the package at 19: has only 3 users.
Old android should no longer be possible after pushing server version 1.0.9 online
corona-warn-app/cwa-server#640

This also affects package 11: one user with an "Invalid Transmission Risk Profile". So, it might be 2 users less for yesterday.

[Tho-Mat]

It was changed at 11 AM CEST (9 AM UTC; package 9; vide infra)

I think you are right, but:
how do you know that? 2*5 = 10, so the package at 06: and 07: could also have a multipler of 5.

[micb25]

how do you know that? 2*5 = 10, so the package at 06: and 07: could also have a multipler of 5.

You are absolutely right. My claim was just based on the inspection of the hourly packages. I don't see any way to improve the estimated numbers for yesterday. Hopefully, we do not see these multiplier changes too frequently.

[Tho-Mat]

I assume the package at 19: has only 3 users.
Old android should no longer be possible after pushing server version 1.0.9 online
corona-warn-app/cwa-server#640

This also affects package 11: one user with an "Invalid Transmission Risk Profile". So, it might be 2 users less for yesterday.

for package 11 i get 7 users with hole and 8 users without hole.

[kai-truempler]

I am not sure if this is the correct place here, but you may have seen the Spiegel interview with Mr. Spahn (here (paywall). He says:

SPIEGEL: Wie viele Infektionen wurden inzwischen in der App eingetragen?
Spahn: Wir gehen von rund 300 Infektionen aus, die bislang per App gemeldet wurden. Das ist die Zahl der Verschlüsselungs-Codes, die von der Hotline ausgegeben wurden, um andere zu warnen. Mehr wissen wir aus Datenschutzgründen nicht.

Do you think people would go through the trouble of calling the hotline and then not submit, or is there an issue with the padding factor calculation that leads to a result that is off by a factor of two?

[janpf]

my guess is that it's in the first day, since noone knows how the packet from "2020-06-23" is actually padded

[micb25]

Do you think people would go through the trouble of calling the hotline and then not submit, or is there an issue with the padding factor calculation that leads to a result that is off by a factor of two?

@kai-truempler: Thanks for sharing this. I totally agree and I would rather expect people not to call the hotline in case of a positive test (stigma, time, effort, etc.).

my guess is that it's in the first day, since noone knows how the packet from "2020-06-23" is actually padded

@janpf: This might be an issue, however, I want to point out that every day there's a significant number of keys which get not parsed (vide infra).

Thus, I would expect that the estimates by diagnosis-keys from @mh- are rather conservative (which I personally prefer). I may add a chart with these unparsed key numbers. At the end, it would be beneficial, if these number would be published by an official institution such as the RKI on a daily base (in addition of publishing the daily download counts).

2020-06-23.dat:89 keys not parsed (8 without padding).
2020-06-24.dat:30 keys not parsed (3 without padding).
2020-06-25.dat:50 keys not parsed (5 without padding).
2020-06-26.dat:150 keys not parsed (15 without padding).
2020-06-27.dat:250 keys not parsed (25 without padding).
2020-06-28.dat:40 keys not parsed (4 without padding).
2020-06-29.dat:100 keys not parsed (10 without padding).
2020-06-30.dat:160 keys not parsed (16 without padding).
2020-07-01.dat:290 keys not parsed (29 without padding).
2020-07-02.dat:80 keys not parsed (16 without padding).

[janpf]

Oh absolutely true, I forgot about those "keys not parsed"

What might be beneficial: on my dashboard I just changed to an hourly analysis, as suggested above by @mh-.
This means I check every hourly package and calculate the padding, number of keys, number of users etc. individually and then sum things up.

This way I'm currently at a total of 218 users and thereby off by a factor of 1.37 @kai-truempler ;)
And if we now consider "keys not parsed" and Mr. Spahn maybe rounding numbers a bit I think it's very hard to get closer to the real number.

At the end, it would be beneficial, if these number would be published by an official institution such as the RKI on a daily base (in addition of publishing the daily download counts).

Absolutely.

[Tho-Mat]

With parsing all keys you can get a minimum number of infected persons.
Theoretically each single key could belong to one person (the maximum).

If i count the minimum users that submit keys i get round about 250. (23.6. - 02.07.)
So there may be 250 users => 300=round(250;-2)