Feature: True Unicode Support #4
Description
currently, all non-ascii chars are identified as part-of-token. this is sufficient for many scenarios.
True unicode support would bring:
- equality / inequality comparison of filters would apply normalization
- "noCase" comparison would consider locale-specific rules for letter case
True unicode support would also mean the following, which I don't see as valuable features at all:
- we could identify unicode punctuation as such, and it would not be a part of an unquoted token
(this would probably be the most expensive change with the least benefit... but I'd provice
a less-surprising consistency) - non-ascii based numeric indices.