Should the MIVOT block be enclosed in a RESOURCE?

[lmichel]

This issue comes from the discussion attached to the PR #174

In the current proposal, the MIVOT block (<VODML>....</VODML>) must be enclosed in a RESOURCE@type=meta that can be located anywhere in the host resource. The host resource is the resource that contains the data to be mapped.

Question 1: Do we have to maintain the requirement for the MIVOT resource to be of meta type?

Rationale

The content of RESOURCE@type=meta is meant to be descriptive only (section 3.4).

Should we consider the MIVOT block as being descriptive only? I would say yes since it provides a model view on the data. This is certainly questionable however.

Proposal

• option 1: Keep the meta requirement in place
• option 2: drop the type=meta requirement if this bothers the implementer job.

Question 2: Do we need to enclose the MIVOT block into a specific RESOURCE?

Rationale

The suggestion is to suppress the requirement for the MIVOT block to be into a specific RESOURCE.
This is feasible but one argument for this MR #174 was not to force the MIVOT block to be in a mandatory location (the head of the host resource). If we drop the encompassing RESOURCE for MIVOT, we require the mapping block to be at the bottom of the host resource: Its location is forced again and therefore, we lose the flexibility we got from the actual MR.

Proposal

• Option 1: Let the proposal as it is with the MIVOT block located into a specific RESOURCE

RESOURCE
    RESOURCE
        MIVOT block
    DATA
or
RESOURCE
    DATA
    RESOURCE
        MIVOT block

• Option 2: Put the MIVOT block at the bottom of the resource containing the data to be mapped.

RESOURCE
    DATA
    MIVOT block

[mbtaylor]

There is an advantage in having the MIVOT block before the data, especially if consumer software can rely on this being the case.

If the MIVOT block comes first, then the consumer software can prepare for decoding the data before encountering it, which means that described objects can be streamed. In practice that would make it possible for library code to offer something like a stream/iterator of MIVOT-described objects as it reads the data; otherwise it can only provide a stream/iterator of table rows, which can only be turned into modelled objects after the end of the data stream has been reached.

If the consumer software reads all of the metadata+data into memory before starting to do processing, this isn't an issue, but for large files that may not be optimal.

Concerning the presence or value of the RESOURCE/@type attribute: it probably makes sense for MIVOT blocks to be type="meta", but I don't see much benefit in making that a requirement. In practice, with my implementor hat on I would likely just look for MIVOT blocks and ignore the value of any type attribute; my guess is that only validators would notice/care whether any such requirement was followed or not, making it an additional (small) compliance burden without any real benefit.

[lmichel]

Let's try to summarize the arguments for different MIVOT resource locations.
For the sack of clarity, we suppose that the MIVOT block is enclosed in a RESOURCE. This point could be dropped according the consensus on the MIVOT location

Option 1

• A resource just after the host resource

RESOURCE
    MIVOT RESOURCE
    VOTABLE stuff

• PRO:
  • the client knows exactly where the annotations are
  • Allows to process data in streaming mode (see MT comment above)
• CON:
  • this breaks the DALI rules (see MD's mail in the list thread)
  • Breaks some Dachs logic.

Option 2

• A resource just before the first data table

RESOURCE
    Global elements required by DALI (INFO, DESCRIPTION..)
    MIVOT RESOURCE
    TABLE

• PRO:
  • Allows to process data in streaming mode (see MT comment above)
  • does not break any DALI rule
• CON:
  • Breaks some Dachs logic.

Option 3

• A resource at the bottom of the host RESOURCE

RESOURCE
    VOTABLE stuff
    MIVOT RESOURCE

• PRO:
  • Fit well with the Dachs logic.
  • does not break any DALI rule
• CON:
  • Does not allow to process data in streaming mode (see MT comment above)

I would lean toward adopting option 2 to preserve the possibility of applying the annotation on data read on the fly without breaking anything in DALI.
Is there any other comments?

[mbtaylor]

I'd go for something like Option 2, but I wouldn't object to making the positioning a SHOULD rather than MUST, e.g.

The MIVOT RESOURCE MUST appear in the same parent RESOURCE as the TABLE it describes, and SHOULD appear before the TABLE to enable streaming of MIVOT-described records.

[lmichel]

I agree with the @mbtaylor statement.
If there is no veto, I'll make a PR very soon.

[lmichel]

fixed by MR #179