Custom enumerators

[Theadd]

This discussion is a continuation of two other discussions, Theadd/ds3x#27 and a great and very elaborated post on stackexchange about enumerating collections or arrays in VBA by @cristianbuse ^{following this comment from the other discussion}

Hi @cristianbuse,

Actually I've been dealing with all this stuff in the few spare times I can get lately.
Although it is not listed on the tB BETA 416's changelog, the way tB deals with enumerators since then significantly differs ^{1 2 3 4} to the way it does VBA and that made me play around with the Enumerator class again, making me realize that it could be made "generic", not only for Variant() types, while making it also reusable to other enumerables with no additional implementation needed, providing advanced enumeration capabilities having direct access to the enumerator instance public properties while iterating it. +nesting

... the entire memory copying then happens at the wrong address i.e. at the address of the actual Variant instead of the address of the first element

And that made me also run into this same problem, which I got resolved with this piece of code.

ArrayList/ArrayListLib/Sources/Modules/CommonModule.twin

Lines 56 to 83 in 16322cb

	#If Win64 Then
	Const VARIANT_VT_ZERO_MASK As LongPtr = &HFFFFFFFFFFFF0000^
	#Else
	Const VARIANT_VT_ZERO_MASK As LongPtr = &HFFFF0000&
	#End If
	Public Sub SafeArrayDescriptorAndVT(TargetArray As LongPtr, ByRef PTR As LongPtr, ByRef VT As Integer)
	GetMemPtr TargetArray, PTR
	If ((PTR And VARIANT_VT_ZERO_MASK) = 0) Then
	VT = CInt(PTR And &HFF&)
	Select Case (PTR And &HFF00&)
	Case vbArray
	GetMemPtr TargetArray + 8, PTR
	Case vbArray + VT_BYREF
	GetMemPtr TargetArray + 8, PTR
	GetMemPtr PTR, PTR
	Case Else
	PTR = vbNullPtr
	Exit Sub
	End Select
	Else
	GetMem2 PTR - 4, VT
	End If
	Select Case VT
	Case 2 To 14, 17, 20, 36: Exit Sub
	End Select
	PTR = vbNullPtr
	End Sub

Provided a VarPtr to any dynamic array or anything holding an array (an array in a Variant or a Variant with a reference to an array), it returns the a LongPtr to the SAFEARRAY structure ^{(the Not Not equivalent also working on variants)} and the vbVarType in which that array was declared/created, as out parameters.

Although I don't like the Not Not notation to get the SAFEARRAY address, it seems this is actually the only way to get the address for a param array

I did not take into account the ParamArray case but I just checked it with your small Sub dh() / Sub TestCloneParam() code sample at Theadd/ds3x#27 (comment) and it is also working.

Whoahh, ...!! WTF... SIDE NOTE: I just finally fucking figured out "the how" and the required var types and pointer addresses behind your remote array trick at exactly this point of the writing! :@!?!

Those recently added SAFEARRAY_FEATURES and SAFEARRAY_OFFSETS enums are coming in very handy as proper replacements of public constants, I'm missing the rgsabound0's lbounds tho.

The SafeArrayDescriptorAndVT procedure in the attached code from above is being used here but the lines 122 and 123 in the code below are the most significant ones as it just uses a simple VT_BYREF Variant with Array's VarType and a pointer to the actual value, whether it is a value type or a reference type, no IsObject()/Set required either.

But while it magically works in tB, when exported and then referenced in VBA, it doesn't work unless I reassign the variant value ByVal within the loop... I still have to take a deeper look on it once I get some proper sleep.

ArrayList/ArrayListLib/Sources/Enumerator.twin

Lines 116 to 133 in 16322cb

	Private Sub Next(ByVal celt As Long, ByRef rgvar As Variant, ByRef pceltFetched As Long) Implements IEnumerator.Next
	If VarPtr(rgvar) = 0 Or celt <> 1 Then
	Err.ReturnHResult = E_INVALIDARGS
	Else
	CIndex += NStep
	If CIndex < MaxValue And CIndex >= 0 Then
	PutMem2 VarPtr(rgvar), VT + VT_BYREF
	PutMemPtr VarPtr(rgvar) + 8, FirstElementAddress + (CIndex * SizeVT)
	NStep = CStep
	If VarPtr(pceltFetched) <> 0 Then pceltFetched = 1
	Err.ReturnHResult = S_OK
	Else
	If VarPtr(pceltFetched) <> 0 Then pceltFetched = 0
	IsEnumerating = False
	Err.ReturnHResult = S_FALSE
	End If
	End If
	End Sub

In relation to your post in stackexchange, I'll leave it for a second post as there's several things I'd like to talk about and test it by myself but It's almost 2 AM. In a short advance:

• I'm addressing to array ranges with a pointer to an item at a given index but I'm considering the "byref variant pointing at the array or it's SA" EDIT: or just store the SA address ^{(assuming it doesn't get reallocated along with pvData)}, and check for LongPtr value change on each iteration to and fix other pointers accordingly in order to prevent crashing when redim preserve gets the array reallocated.
• I don't like reusing the _NewEnum on a custom collection-like class but your Collection object in # 2 doesn't have to be populated for it to work, nice.
• In # 2, aren't the memory addresses of NextPtr at the same distance from one to another?

---

Footnotes

1. tB automatically calls to .Reset just before it starts to enumerate, VBA doesn't. ↩

2. In debug mode, for every step (VBA's F8/Shift+F8/F5) the debugger does, each enumerable gets fully enumerated and attached to the "Locals" window as-is, not even in a collapsible element so that tool window becomes kind of unusable. ↩

3. The returned Enumerable instance used by the debug mode needs to be declared as static to avoid crashing, probably related to the same type of crashes you mentioned in your stackexchange post. ↩

4. Which forced me to keep two static Enumerators and only provide the second when the first one is still enumerating to prevent the debugger from resetting and consuming the main one in every step, which obv won't pass the first loop. ↩

[cristianbuse]

Hi @Theadd ,

Provided a VarPtr to any dynamic array or anything holding an array (an array in a Variant or a Variant with a reference to an array), it returns the a LongPtr to the SAFEARRAY structure (the Not Not equivalent also working on variants) and the vbVarType in which that array was declared/created, as out parameters.

Would it make sense to return a small struct (Private Type) containing the 2 values (ptr and vt) instead of returning as out params?

Whoahh, ...!! WTF... SIDE NOTE: I just finally fucking figured out "the how" and the required var types and pointer addresses behind your remote array trick at exactly this point of the writing! :@!?!

😄 Not sure what you meant by 'remote array trick'. Sorry. Are you referring to a Variant pointing to a SAFEARRAY ByRef?

Those recently added SAFEARRAY_FEATURES and SAFEARRAY_OFFSETS enums are coming in very handy as proper replacements of public constants, I'm missing the rgsabound0's lbounds tho.

Fixed here. Thanks for pointing out.

But while it magically works in tB, when exported and then referenced in VBA, it doesn't work unless I reassign the variant value ByVal within the loop

It won't surprise you that this is among the very first things I tried - so that I can avoid copying elements one by one. This was my test (added to the CR question as well):

Option Explicit

Private Type TagVariant
    vt As Integer
    wReserved1 As Integer
    wReserved2 As Integer
    wReserved3 As Integer
    ptr1 As LongPtr
    ptr2 As LongPtr
End Type

Private Type nextItemAddr
    itemByRef As TagVariant
    dummy1 As LongPtr
    dummy2 As LongPtr
    nextPtr As LongPtr
End Type

#If Win64 Then
    Const sizeOfNext = 48
#Else
    Const sizeOfNext = 28
#End If

Sub TestByRef()
    Const size As Long = 10
    '
    Dim c As Collection:        Set c = New Collection
    Dim e As IEnumVARIANT:      Set e = c.[_NewEnum]
    Dim arr() As Variant:       ReDim arr(0 To size - 1)
    Dim arr2() As nextItemAddr: ReDim arr2(0 To size - 1)
    Dim h As New EnumHelper:    Set h.EnumVariant = e
    Dim i As Long
    Dim v As Variant
    Dim ptr1 As LongPtr
    Dim ptr2 As LongPtr
    Dim vt As Integer
    '
    ptr1 = VarPtr(arr(0))
    ptr2 = VarPtr(arr2(1))
    vt = vbVariant + VT_BYREF
    For i = 0 To UBound(arr2)
        arr(i) = i 'doesn't matter what values
        arr2(i).itemByRef.ptr1 = ptr1
        arr2(i).itemByRef.vt = vt 'could use the actual var type here but then the ptr needs updated with +8
        arr2(i).nextPtr = ptr2
        ptr1 = ptr1 + VARIANT_SIZE
        ptr2 = ptr2 + sizeOfNext
    Next i
    arr2(UBound(arr2)).nextPtr = 0
    '
    MemLongPtr(ObjPtr(e) + PTR_SIZE * 2) = VarPtr(arr2(0))
    '
    For Each v In h
        Debug.Print v
        v = Empty 'Required to avoid 'Type Mismatch' error that gets raised on the 'Next v' line
    Next v
End Sub

It is very unfortunate that this does not work without setting v to anything that does not have the VT_BYREF flag. I chose to use the TagVariant structure so that I can set the vt and ptr separately without copying memory but that is not important considering that the thing doesn't work well in VBA.

In relation to your post in stackexchange, I'll leave it for a second post as there's several things I'd like to talk about and test it by myself but It's almost 2 AM. In a short advance:

I'm addressing to array ranges with a pointer to an item at a given index but I'm considering [Edit] just store the SA address (assuming it doesn't get reallocated along with pvData), and check for LongPtr value change on each iteration to and fix other pointers accordingly in order to prevent crashing when redim preserve gets the array reallocated.

I also thought about how to make this safe and I think the answer lies in locking the array while the loop is running. After all this is what VBA does when you iterate an actual array:

Option Explicit

Sub TestArrayLoop()
    Dim arr() As Variant
    Dim v As Variant
    '
    ReDim arr(0 To 3)
    '
    For Each v In arr
        ReDim arr(0 To 4) 'This raises error '10' - 'The array is fixed or temporarily locked'
    Next v
End Sub

I don't like reusing the _NewEnum on a custom collection-like class but your Collection object in # 2 doesn't have to be populated for it to work, nice.

Well, an extra, unpopulated collection used just for this purpose doesn't seem to be a big resource investment considering that when it gets destroyed it does the cleaning for us. If the class already has a Collection that actually gets used, then that would work just fine and there would be no need for an extra one.

In # 2, aren't the memory addresses of NextPtr at the same distance from one to another?

Always. 48 bytes on x64 and 28 bytes on x32 (i.e. the sizeOfNext constant in my example above). This is just how the internal structure of a collection item happens to be. Here are the interesting VbCollectionHeader and VbCollectionItem structures in detail.