Type tweaks

[mwiebe]

This is an update corresponding to the type_tweaks PR on libdynd.

libdynd/libdynd#30

[mrocklin]

We're not using this syntax in DataShape. Will this cause a mismatch?

[mwiebe]

My idea presently is to have dynd's syntax be a strict superset of datashape. In this particular case, "{x: int32, y: int32}" is a dynd type with flexible field layout in memory, while "c{x: int32, y: int32}" has a fixed field layout in memory. This is one reason why there is both "nd.type_of" and "nd.dshape_of".

[mrocklin]

I guess the question is does the DyND type system implement datashape. I think I've asked before and the "answer is almost but not quite." Follow on is "should it?" and "is this important?"

[mwiebe]

One way this PR moves dynd closer to datashape conformance is with respect to the syntax sugar definitions. "{..}" is defined to be syntax sugar for the "struct" type constructor, and "3" as a dimension is defined to be syntax sugar for "fixed[3]". We might want to add an "extensions to datashape" document into the datashape project, which lays out the particular ways dynd extends the syntax, both to make sure it's good and to provide a blueprint if others want to do this too.

[mrocklin]

Is the layout bit linearly dependent with what you've recently proposed on data locators? blaze/datashape#45

[mrocklin]

I think of {x: int, y: int} as being ordered logically. I guess I didn't think that this might differ from how it is stored physically.

[mwiebe]

Yeah, the data locators proposal is basically an attempt to abstract some of these layout ideas from the specifics of dynd's array metadata.

Here's an example of the field ordering in action:

In [1]: from dynd import nd, ndt

In [2]: a = nd.array((3, 5), type='{y: int, x: int}', access='rw')

In [3]: b = a[::-1]

In [4]: a
Out[4]: nd.array([3, 5], type="{y : int32, x : int32}")

In [5]: b
Out[5]: nd.array([5, 3], type="{x : int32, y : int32}")

In [6]: b.y = 100

In [7]: a
Out[7]: nd.array([100, 5], type="{y : int32, x : int32}")

In [8]: b
Out[8]: nd.array([5, 100], type="{x : int32, y : int32}")

[mwiebe]

In [9]: nd.debug_repr(b)
Out[9]: 
------ array
 address: 00000000003BAB40
 refcount: 1
 type:
  pointer: 00000000003922D0
  type: {x : int32, y : int32}
 metadata:
  flags: 3 (read_access write_access )
  type-specific metadata:
   struct metadata
    field offsets: 4, 0
 data:
   pointer: 00000000003BAB30
   reference: 00000000003BAAF0
    ------ memory_block at 00000000003BAAF0
     reference count: 2
     type: array
     type: {y : int32, x : int32}
    ------
In [10]: nd.debug_repr(a)
Out[10]: 
------ array
 address: 00000000003BAAF0
 refcount: 2
 type:
  pointer: 0000000000392190
  type: {y : int32, x : int32}
 metadata:
  flags: 3 (read_access write_access )
  type-specific metadata:
   struct metadata
    field offsets: 0, 4
 data:
   pointer: 00000000003BAB30
   reference: 0000000000000000 (embedded in array memory)
------

[mwiebe]

Another more sophisticated example:

In [11]: a = nd.array((3, 5, 2, "test"), type='{y: int, z: real, x: int, desc: string}', access='rw')

In [12]: a
Out[12]: nd.array([3, 5, 2, "test"], type="{y : int32, z : float64, x : int32, desc : string}")

In [13]: b = nd.fields(a, "x", "y")

In [14]: b
Out[14]: nd.array([2, 3], type="{x : int32, y : int32}")

In [15]: nd.debug_repr(b)
Out[15]: 
------ array
 address: 00000000003BAD20
 refcount: 1
 type:
  pointer: 00000000003924B0
  type: {x : int32, y : int32}
 metadata:
  flags: 3 (read_access write_access )
  type-specific metadata:
   struct metadata
    field offsets: 16, 0
 data:
   pointer: 000000000038D788
   reference: 000000000038D730
    ------ memory_block at 000000000038D730
     reference count: 2
     type: array
     type: {y : int32, z : float64, x : int32, desc : string}
    ------
------