Relax variant-name grammar

[stasm]

variant-name is currently defined as one or more words.

/* exclude whitespace and [ \ ] { } */
word ::= (((char - line-break) - inline-space) - [#x5b#x5c#x5d#x7b#x7d])+
variant-key ::= number | word (_ word)*

We don't use words anywhere else and I find it weird that they forbid so many arbitrary characters. Should we relax it to only forbid ] and maybe [? Should we allow escaping with \]? As in: [variant name with a closing bracket \] looks like this].

Additionally, I don't see why we have to force variant-name to be non-blank. [] should parse as a variant name of "" (a zero-width string). It's nor terribly useful, but consistent.

[zbraniecki]

My concern here is that it opens up room for a lot of edge cases that may be confusing and potentially hard to handle. In particular:

• Empty/accidental variant names with copy pasted invisible characters that are not matched later
• Characters that are unlikely to be useful but since we allowed them will block us from using those characters anywhere else

I'm not completely against pursuing this direction (I know stas was in favor of it from the very beginning), but I'd like to see use cases and/or examples of users wanting to use this feature. I'm concerned about limiting our future options by relaxing the syntax with noone asking for it.

[Pike]

I think there's value in variant-key being strict. This is mostly about the readability of the message. At the point where we have variants, that already goes down a fair amount. Having complexity in there would further degrade that.

Side question, is 15 ducks an OK variant-key?

[zbraniecki]

15 ducks is not an OK variant-key because it starts with a digit (1) so we expect it to be a number type value.

[stasm]

Use-cases

There are two use-cases which would be well-served by relaxing the grammar of VariantName:

1. Term attribute values in the native language.
2. Free-form variable values.

Native term attribute values

Term attributes are private to the current language. They may represent grammatical concepts which are foreign to the English grammar. Consequently, they may be hard to give English names to. To fully embrace the asymmetric design of terms and their attributes, I'd like to make it possible for attribute values to be written in the target language of the localization.

-brand-name = Firefox
    .Genus = männlich 

(Ideally, the attribute's name could also contain non-ASCII characters. I'll get back to this in a second.)

Variable values

There might be cases where the localizer might want to provide a specialized translation for one of the possible variable values.

# The ʻ character is U+02BB, outside of the Basic Latin block.
welcome = { $state ->
    [Hawaiʻi] Aloha!
   *[other] Welcome!
}

In most cases, dynamic references (#80) will be the preferred way of implementing such UIs, but they do have the drawback of requiring the developer's intervention. For lightweight one-off customizations simple SelectExpressions like the one above might still be good and cheap solutions.

Proposed design

I suggest that make the [name] syntax restricted to just identifiers. This is actually more restrictive than today because today's grammar also allows whitespace.

For names containing any other characters, including whitespace, ] or any non-Latin characters, I propose a new syntax: ["name"]. The standard StringExpression grammar would apply here.

This design has the advantage of being explicit about digits: [15 ducks] (illegal) vs. ["15 ducks"], as well as about whitespace: [ trim whitespace ] vs. [" keep whitespace "].

Lastly, although this might be best discussed in a separate issue, we could consider allowing attributes names to contain arbitrary characters using a new ."attr name" = Value syntax. (Moved to #117.)

[zbraniecki]

If I'm not mistaken this is something that we did not encounter anyone requesting yet. Can it wait for post-1.0 and any user request?

[stasm]

The sooner we relax, the less trouble we'll encounter due to cross-channel.

We haven't seen much use of custom term attributes yet, which is why I think we're not seeing these requests. I'd like to default to Unicode wherever possible. It's 2018.

[zbraniecki]

As I stated before, the risk of Unicode here is that it opens edge cases to be legitimate unless you blacklist them. Things like \n, \t, many forms of whitespaces etc. all become proper variant name characters.
It would be nice to research the concept of visible/non-breaking/non-whitespace unicode characters and maybe use this notion? Maybe we can use UAX31? My worry is that I don't feel I know enough about traps and limitations coming from allowing Unicode in syntax and I've seen how much time languages that wanted to introduce them spent trying.
Do you think we're somehow immune to such issues? Is there any programming language that successfully did this not via UAX31?

[stasm]

I'm trying to make a point that this is about matching strings, not identifiers. However, in context of identifiers, UAX31 could help us, yes.

In my proposal, ["text"] would follow the same semantics and parsing logic as StringExpressions do. In fact, I'm proposing that we make the current syntax ([text]) even more restricted than it is right now, so your concern is mitigated.

[zbraniecki]

key = { key2["
"] }

Would that make such code a valid FTL?

[stasm]

No, because StringExpressions may not contain newlines.

[zbraniecki]

Oh! Thanks for explaining it to me! No objections.