Actually modern terminals support 24bit colors using the following sequences.
38;2;<r>;<g>;<b> - foreground
48;2;<r>;<g>;<b> - background
I'm net sure yet if it's worth to support the web format, like #ffaabb. Because then you need to come with a way to encode foreground and background.