Interpolation type map for tooling

[ianjosephwilson]

This is just a draft / concept of a way we could specify supported types that tooling could use to give feedback to users about what they are plugging into TDOM templates. This is sort of what I felt like we have been working towards:

• what is the structure of the templates -- TNode, TAttribute, ...
• what values can be interpolated in the templates -- default_template_type_map (this thing)

Obviously having object all over the place isn't great so I tried to take a stab at a more strict set of types that would require other types to be preprocessed either before involving Template at all or with a conversion or format spec.

I can imagine cases where you'd want to define some common types you are putting into templates and converting to strings yourself so there would probably be someway to extend this.

[davepeck]

Obviously having object all over the place isn't great

Hah, yeah. The non-strict types, all of which are basically Union[object, ...], while completely reasonable, don't seem super helpful from a tooling perspective.

take a stab at a more strict set of types

Primarily the intent of these types would be documentation for tool developers. If, for instance, Astral decided to support type checking of t-string HTML content in ty (not gonna happen!) they'd probably read it over?

That said, I see you did use some non-strict types in processor.py. What would happen if we used the strict variants?

[ianjosephwilson]

Primarily the intent of these types would be documentation for tool developers. If, for instance, Astral decided to support type checking of t-string HTML content in ty (not gonna happen!) they'd probably read it over?

My rough idea was that a tool would consume the "type map" and use that type map together with a TDOM-aware parser to type check each appropriate Template:

[tool.lintingtool.exts.tdom]
path_to_type_map = "src/myapp/tdom_config.py"
version = 0.1
annotated_marker="html+tdom"

type TDOM = Annotated[Template, "html+tdom"]
invalid = b'abc' #bytes
check_t: TDOM = t"<div>{invalid}</div>"

That said, I see you did use some non-strict types in processor.py. What would happen if we used the strict variants?

Since the Templates all just track object all we can do is runtime checks at this point AFAIK. Using cast can kind of help detect bugs but since fundamentally object is coming in we are kind of limited. I was thinking we could put in a upfront type checker that went as far as it could go doing runtime checks and that could be used in testing but it would be a separate flow as much as possible so it didn't interfere with the "integral" checks that make sure things are escaped properly and work correctly. I started on a prototype but it might be hard to check things without interfering with them so we'll see how it goes.

[davepeck]

I think it's a good possible approach to keep in mind for down the road.

Thinking specifically about extending ty to support tdom: it can probably load a type map like this. What's missing is some notion of grammar: for instance, where in the t-string can an AriaAttributeValue appear? Only then can we use a type map to decide that the interpolation should conform to (say) StrictAriaAttributeValue. Wildly hand-waving here, but in the future perhaps ty could support a general mechanism for third parties to express string grammars? Or perhaps it will only support fixed rust extensions, with tdom shipping its own? (At which point, type maps may be of limited value.) Today,ty has no extension mechanism that I'm aware of.

"General mechanism for third parties to express string grammars + type maps" sort of feels like a PEP in the making but it is fraught with peril.

[davepeck]

So my current hand-wave crazy idea: create a new library (string-grammar?) that introduces a large number of new typing special forms (someday in the future: typing.string.*) that let developers define a grammar... in Python. Specifically, static python; the forms have no runtime impact. The holes in the grammar can directly reference the expected types (AriaAttributeValue, etc.)

Then build an CLI (sgty) type checker for this stuff and, ultimately, an LSP. Might be a fun thing to sprint on at PyCon in a few weeks, just to see how hard it is to break the spine of it and where the gremlins are. If it works, it could be a legitimate extension point for mypy and friends.

[pauleveritt]

Your crazy idea would feel exceptionally well in my crazy idea: an agent that runs locally in a small model by moving more of the work out of the LLM. I'm pretty far along on it and have some machinery in a similar (but more primitive) ballpark.

[ianjosephwilson]

So my current hand-wave crazy idea: create a new library (string-grammar?) that introduces a large number of new typing special forms (someday in the future: typing.string.*) that let developers define a grammar... in Python. Specifically, static python; the forms have no runtime impact. The holes in the grammar can directly reference the expected types (AriaAttributeValue, etc.)

I just figured each format would need its own "tailored" "tnode-ifier" that parses the content and identifies those "slots". I think that is what tslinter was trying to do. Wouldn't the grammar have to be a spec of HTML among the others?

[davepeck]

I just figured each format would need its own "tailored" "tnode-ifier" that parses the content and identifies those "slots"

Yeah, that's the right day zero approach. My "crazy idea" is more like day 1,000.

In any case, I'm still not sure I understand the value of an "interpolation type map" in the "tailored" world. Tailored tools need to know the names of the possible slots; at that point, why not also know their expected types?

Wouldn't the grammar have to be a spec of HTML among the others?

Yes, that's the idea. There are obvious limitations here (can't help you much when there's a <script> tag; the real complexity of defining these grammars may be far too high; etc.) To me the open question is: on balance, could it be a win? Maybe there's a small exploration that would help clarify this...

[ianjosephwilson]

I just figured each format would need its own "tailored" "tnode-ifier" that parses the content and identifies those "slots"

Yeah, that's the right day zero approach. My "crazy idea" is more like day 1,000.

In any case, I'm still not sure I understand the value of an "interpolation type map" in the "tailored" world. Tailored tools need to know the names of the possible slots; at that point, why not also know their expected types?

case 1: You still want to catch type errors but you want to allow other types

You have a custom handling for datetime values and you want them all processed "auto-magically" via some sort of hook. Normally with the "strict" mapping those would fail but you can change the types you expect at that slot from the strict types to the strict types + datetime.

case 2: You want to use a completely different type for an extension

Another example might be that the components are not callables at all (ie. the fact that protocols satisfy that seems like a total accident), like maybe you want to just use a fixed marker.

#.... some sort of component marker for a custom component processor to handle
@dataclass(frozen=True, slots=True)
class ComponentMarker:
    name: str

Header = ComponentMarker('Header')

# some sort of configuration a tailored tool reads in 
from tdom.types import default_template_strict_type_map, ComponentInterpolationValue
# Here we use ComponentInterpolationValue as a key so the general "slots" are well defined but the type has changed
template_type_map = default_template_strict_type_map | {ComponentInterpolationValue: ComponentMarker}

# using it in a template doesn't throw redlines/yellowlines
example_t: Annotated[Template, "html+tdom"] = t"<{Header}>somsfjsakldjflkasdf</{Header}>"

Wouldn't the grammar have to be a spec of HTML among the others?

Yes, that's the idea. There are obvious limitations here (can't help you much when there's a <script> tag; the real complexity of defining these grammars may be far too high; etc.) To me the open question is: on balance, could it be a win? Maybe there's a small exploration that would help clarify this...

I did start on a prototype but it is more of a "script" at this point: #122

[davepeck]

Perhaps we should close this and move to discussion!

[pauleveritt]

@ianjosephwilson Sorry for using GitHub as a DM service. We booked the open space:

t-string-apalooza S-3 - Saturday May 16, 2:00 PM PDT

[ianjosephwilson]

Perfect, thanks @pauleveritt