Allowing other characters in a token

[carpie]

When using UUIDs in tokens, the tokens are rejected because of the - character in them. I can subclass BooleanAlgebra and override tokenize but it is a lot of duplication for allowing an additional character in the token. It would be nice if one could specify the allowable character set.

[pombredanne]

Thanks for this and sorry for the late reply and review. It kinda makes sense... the rationale for only allowing certain characters is that tokens could then be used as Python-level identifiers and to avid possibly collision with short-form operators (~+| ... etc). In practice this is not big requirement IMHO. In fact in https://github.com/nexB/license-expression/ we accept any characters in tokens and have implemented a few custom tokenizers too.