Allow for caching of the jsonschema RefResolver

[jhosteny]

Hi @jbwyme, this package is working really nicely for me. One problem I've had is that the jsonschema RefResolver is recreated for each validation, and thus it pulls external $ref's when they could otherwise be cached.

I have a change that fixes this by creating and caching one when the schema is looked up. For example, in Resolver.lookup_schema:

ref_resolver = jsonschema.RefResolver.from_schema(lookup_result)
self.cache[schema_path] = [lookup_result, ref_resolver, store_time]
return lookup_result, ref_resolver

Then that is passed to validation like this:

schema, ref_resolver = resolver.lookup_schema(self.schema_key)
jsonschema.validate(
        instance=self.data,
        schema=schema,
        resolver=ref_resolver,
        format_checker=jsonschema.FormatChecker(),
)

I'm not sure if this is the proper way to do this, so I didn't submit a PR yet. Is there perhaps an easier way to do this that you are aware of?

[jbwyme]

Hey @jhosteny! There's already schema caching built in to the resolver (https://github.com/jbwyme/iglu-python-client/blob/master/iglu_client/resolver.py#L30). Can you help me understand your usage such that this cache isn't being hit?

[jhosteny]

Hi @jbwyme. Yup, I should have explained it a bit better. Take this schema from Snowplow as an example:

{
    "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
    "description": "Schema for a Sauna command",
    "self": {
        "vendor": "com.snowplowanalytics.sauna.commands",
        "name": "command",
        "format": "jsonschema",
        "version": "1-0-0"
    },
    "type": "object",
    "properties": {
        "envelope": {
            "$ref": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/instance-iglu-only/jsonschema/1-0-0#"
        },
        "command": {
            "$ref": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/instance-iglu-only/jsonschema/1-0-0#"
        }
    },
    "required": [
        "envelope",
        "command"
    ],
    "additionalProperties": false
}

Both the envelope and the event are self-describing. Now look at this schema, which is close to what we want to use for some set of schemas:

{
    "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
    "description": "Schema for MQTT telemetry event",
    "title": "event",
    "self": {
        "vendor": "com.acme.telem",
        "name": "event",
        "format": "jsonschema",
        "version": "1-0-0"
    },

    "type": "object",
    "properties": {
        "envelope": {
            "$ref": "https://schemas.acme.com/schemas/com.acme.telem/envelope/jsonschema/1-0-0#"
        },

        "event": {
            "$ref": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/instance-iglu-only/jsonschema/1-0-0#"
        }
    },
    "required": [
        "envelope",
        "event"
    ],
    "additionalProperties": false
}

In this case, the envelope is not used as a self-describing schema. If you were to generate a message that would properly validate, if would have the envelope schema's properties right underneath the envelope property here, rather than under envelope -> data (with a sibling schema field).

There are certain case where this is pretty useful for us. For example, I might have a schema that I want to use sometimes as self-describing, but at other times, I use the concrete version. Of course, this breaks some of the utility of them, but it can still be useful. I may have a specific event that as an array of items, which is perhaps quite large, and I don't want to include the schema field for each one, even if a change to that subfield will propagate a version change to the parent event.

So, in the example above, validation will still work - iglu_client will pull and cache (or get from the local repo), iglu:com.acme.telem/event/jsonschema/1-0-0, but inside that schema, resolving https://schemas.acme.com/schemas/com.acme.telem/envelope/jsonschema/1-0-0 is then handled by jsonschema. The crucial part is that when jsonschema.validate is called, it creates a new RefResolver every time, and thus loses the cached results.

I hope that description makes sense. I wasn't sure if there's another way to accomplish this without the PR, or if the way I did it would even make sense if not, which is why I didn't open one yet.

[jbwyme]

Ah ok I understand. I see there was some conversation around this here: python-jsonschema/jsonschema#135. It looks like perhaps we could use the store param in RefResolver to keep a cache instead of having to cache a bunch of RefResolvers. I haven't tried it but I think something like this might work:

store = {} # global
....
jsonschema.validate(
        instance=self.data,
        schema=schema,
        store=store
)

[jhosteny]

It looks like that is a mapping of URI to actual schemas, and is used to update the store instance var in the resolver. See here. So that could work, except that you would have to supply the raw schema contents to start with, rather than let it fetch the URL and populate.