RS-19602: Remove characters from variable name if not supported by either Displayr or haven #93
Conversation
| # \\p{Sc} = any kind of currency symbol | ||
| # The special characters \ . _ $ # @ | ||
| # | ||
| # This is stricter than either haven or Displayr, because their set of allowed characters are different. |
There was a problem hiding this comment.
Are the allowed characters for haven or Displayr documented? Maybe you could add a link for them? For this bug, what was causing the error?
There was a problem hiding this comment.
Not documented explicitly as far as I know. The official rules from SPSS are vague:
Subsequent characters can be any combination of letters, numbers, nonpunctuation characters, and a period (.)
https://www.ibm.com/docs/en/spss-statistics/cd?topic=view-variable-names
Haven has the following regex https://github.com/tidyverse/haven/blob/190f7a9ababae53ea4bfb85e4183c21881644c29/R/haven-spss.R#L140
Whereas the readstat library (which haven uses) has this:
https://github.com/WizardMac/ReadStat/blob/dev/src/spss/readstat_sav_parse.rl#L20
Displayr has this https://github.com/Displayr/q/blob/master/QLib/DataModel/Variables/RawVariable.cs#L25-L58
All are different
There was a problem hiding this comment.
The regex is hard to read so I also manually tested both to derive the conclusions in the tests
There was a problem hiding this comment.
I guess if it was clearly documented, then we wouldn't be running into these bugs. But the regex doesn't look too bad now
There was a problem hiding this comment.
Yeah its good it ended up being quite simple. Better to have a bias towards removing characters, which should mean bugs like this won't happen or will be rare.
When I fixed the previous bug, I must have assumed that the input data files would never have invalid characters. This bug wouldn't happen if there were explicit rules and everyone followed them. The emdash in the bug was allowed by Displayr, but not haven.
No description provided.