Support more of abstract array interface

[MilesCranmer]

Currently similar will return an array of a different type:

from juliacall import Main as jl
import numpy as np

x = np.random.randn(5)
y = jl.similar(x)

print(jl.typeof(x))
# PyArray{Float64, 1, true, true, Float64}
print(jl.typeof(y))
# Vector{Float64}

This can introduce some type instabilities in libraries due to the assumption that container type is preserved by similar.

I guess we just need the "optional methods" from here: https://docs.julialang.org/en/v1/manual/interfaces/#man-interface-array

[marcosdiezgarcia]

julia> using PythonCall

julia> Py(rand(10)).to_numpy().dtype
Python: dtype('float64')

julia> Py([[1,2,3], [4,5,6]]).to_numpy().dtype
Python: dtype('O')

I would expect the latter to be dtype('int64') to match Python:

>>> import numpy
>>> a = numpy.array([[1,2,3], [4,5,6]])
>>> a.dtype
dtype('int64')

I am using the following:

• julia 1.12.1
• PythonCall 0.9.28
• python 3.13.9
• numpy 2.3.4

[MilesCranmer]

dtype('O') simply means "object". I'm pretty this mean that this difference is due to [[1]] being a Vector{Vector{Int}} in Julia whereas it's a single array object in numpy.

Basically, numpy is autoconverting to a single array object (presumably because there's no alternative way to write a matrix). Julia does not do this, because for a matrix, you would write [1 2 3; 4 5 6] for example. So [[1,2,3]] is taken literally as a vector of vectors.

[marcosdiezgarcia]

In my previous example, the "conversion" you mention is done by the to_numpy function in array.jl, which just calls numpy.array to initialise a new numpy array.

There is no problem with numpy itself. Indeed, [[1]] is a single numpy.ndarray object. However, being a single array object does not imply its dtype must be a generic Python object per se; and, the dtype of [[1]] is in fact dtype('int64'). You can create multi-dimensional arrays with any supported dtype, not just the generic object.

I think implicit "conversions" to a more abstract type like object from concrete types like int64, when it is not really necessary, can lead to unexpected behaviour in user code. It would be great to have to_numpy function handle this and also be more explicit about such case in the documentation regarding Python-Julia conversion rules or the API reference (juliacall.ArrayValue).

[MilesCranmer]

Hi @marcosdiezgarcia,

There is no bug here, though. Sorry if I was unclear.

The answer is that the syntax you are trying to use, [[1]], is incorrect in Julia. It means something different in Julia than in NumPy. In Julia you should be using [1;;]. A [[1]] is a completely different object altogether.

This can be surprising if you are coming from Python, but I do not think juliacall should automatically clean up such mistakes, for the following reason. Because, were it to do so, this would introduce various inconsistent behavior. For example, if Vector{Vector{Int}} automatically converts to Matrix{Int}, then, what should a Vector{SparseVector{Int}} get converted to? Should it get converted at all? Any decision here leads to ambiguities. Especially because someone may want to access Julia arrays from Python, automatically normalizing to Matrix{Int} would create a copy whereas someone might want to have views into their original arrays (though I admit this often causes more problems).

My feeling is that NumPy does these sorts of nice automatic cleanups because the Python standard library has no way of expression a 2D array, other than with a list of lists. But Julia has a fully featured 2D array type, so it should be used.

What you should actually be using is the normal matrix syntax:

julia> Py([1 2 3; 4 5 6]).to_numpy().dtype
Python: dtype('int64')

To me this makes a lot of sense, once you look at the actual types being created:

julia> [1 2 3; 4 5 6] |> typeof
Matrix{Int64} (alias for Array{Int64, 2})

julia> [[1, 2, 3], [4, 5, 6]] |> typeof
Vector{Vector{Int64}} (alias for Array{Array{Int64, 1}, 1})

julia> Py([[1, 2, 3], [4, 5, 6]]).to_numpy().dtype
Python: dtype('O')  # Since it stores Vector{Int}

[marcosdiezgarcia]

Thanks for your feedback @MilesCranmer.

I understand the difference between [[1, 2, 3], [4, 5, 6]] and [1 2 3; 4 5 6] in Julia. When you say that [[1]] is syntactically incorrect in Julia, you assume that I was trying to use it as [1;;]. This not what I am trying to do, and I did not mention it in my example. I am not suggesting that julicall should automatically convert Vector{Vector{Int}} to Matrix{Int} nor suggesting ways about how you would convert Vector{SparseVector{Int}} or other types.

My concern is what I reported in my example. That is, the dtype('O') output by juliacall in the expression Py([[1,2,3], [4,5,6]]).to_numpy().dtype (including any other Vector{Vector{Int64}} instance given as argument). Is there any possible way to make the dtype be dtype('int64') instead of dtype('O')?

[MilesCranmer]

But the dtype is an object. This is correct behavior. It's not an int64. The same way that the eltype in Julia is a Vector{Int64}, in numpy, the dtype is also not an int64, but a more abstract "object" to refer to a Julia vector object.

[marcosdiezgarcia]

Yes, of course, the dtype itself is an object. Specifically, it is an Int64DType as shown below. But I was referring to the types of the elements contained in the numpy array when I asked

Is there any possible way to make the dtype be dtype('int64') instead of dtype('O')?

Regardless of how many levels of nesting, dtype('int64') is the value of the property dtype as shown below:

Whereas in Julia, using PythonCall to_numpy(), it is clear that dtype('int64') is only output for Vector{Int64} not Vector{Vector{Int64}} nor Vector{Vector{Vector{Int64}}}:

To me this shows that the value of the property .dtype in numpy is defined based on the innermost elements in the array, whereas in Julia the value of .dtype upon using to_numpy() is not based on the innermost elements (i.e. 1, 2, 3, etc) but on the whole structure containing them (i.e Vector{Int64} for the 2nd array, and Vector{Vector{Int64}} for the 3rd array). (Perhaps this is what you were trying to explain above.) So it is clear that the behaviour of accessing the property .dtype is different in numpy compared to what julicall outputs from to_numpy(). I was expecting same behaviour.

[MilesCranmer]

Again, this is because numpy automatically converts to a single 2D array, whereas Julia leaves it as a vector of vectors. There is no bug here.

[marcosdiezgarcia]

I found a simple fix to JlWrap/array.jl (line 370) in PythonCall 0.9.28:

try:
    import numpy
-   arr = numpy.array(arr, dtype=dtype)
+   arr = numpy.array(list(arr), dtype=dtype)
except ImportError:
    pass

Now I get what I was looking for:

julia> Py([1, 2, 3]).to_numpy().dtype
Python: dtype('int64')

julia> Py([[1, 2, 3], [4, 5, 6]]).to_numpy().dtype
Python: dtype('int64')

julia> Py([[[1, 2, 3], [4, 5, 6]]]).to_numpy().dtype
Python: dtype('int64')

This also works if the underlying elements are of other type like floats or strings, e.g.:

julia> Py([[1.0,2.0,3.0], [4.1,5.1,6.1]]).to_numpy()
Python:
array([[1. , 2. , 3. ],
       [4.1, 5.1, 6.1]])

julia> Py([[1.0,2.0,3.0], [4.1,5.1,6.1]]).to_numpy().dtype
Python: dtype('float64')

julia> Py([["a","b","c"], ["d","f","g"]]).to_numpy()
Python:
array([['a', 'b', 'c'],
       ['d', 'f', 'g']], dtype='<U1')

[MilesCranmer]

@marcosdiezgarcia please move this discussion elsewhere; maybe to the Julia discourse. This issue is about getting similar and other Base operations working on PyArray. (I should have pointed this out earlier)

[marcosdiezgarcia]

No worries. I'll raise a separate issue.