Improve generic pattern to support optional, union, array. Fixed regression.

[TIMONz1535]

I examined the original generic pattern code in more detail and realized that it incorrectly handles cases when it cannot parse further. We need to partially successfully parse, not cancel it by return nil.

• Fixed regression by previous pull request Improve generic pattern to support "T.*-" #3015 where parseCode was replaced with parseCodePattern

---@param a `TC`#Comment -- ok
---@param a `TC`? -- ok
---@param a `TC`|integer -- ok
---@param a `TC`[] -- ok

• Generic pattern now supports optional, union, array and comment without a space, like all other types.
  The signs <> are also correctly recognized, but the generics don't support it at all.

---@param a string#Comment -- ok
---@param a `TCP`2#Comment -- now ok
---@param a `TCP`2? -- now ok
---@param a `TCP`2|integer -- now ok
---@param a `TCP`2[] -- now ok

My complete test

---@param a string#Comment
---@return string
local function new(a) end
local obj = new("Vehicle")

---@generic T
---@param a T#Comment
---@return T
local function new(a) end
local obj = new("Vehicle")

---@generic TC
---@param a `TC`#Comment
---@return TC
local function new(a) end
local obj = new("Vehicle")

---@generic TCP
---@param a `TCP`2#Comment
---@return TCP
local function new(a) end
local obj = new("Vehicle")



---@param a string?Comment
---@return string?
local function new(a) end
local obj = new("Vehicle")

---@generic T
---@param a T?Comment
---@return T?
local function new(a) end
local obj = new("Vehicle")

---@generic TC
---@param a `TC`?Comment
---@return TC?
local function new(a) end
local obj = new("Vehicle")

---@generic TCP
---@param a `TCP`2?Comment
---@return TCP?
local function new(a) end
local obj = new("Vehicle")



---@param a string[]Comment
---@return string[]
local function new(a) end
local obj = new({"Vehicle"})

---@generic T
---@param a T[]Comment
---@return T[]
local function new(a) end
local obj = new({"Vehicle"})

---@generic TC
---@param a `TC`[]Comment
---@return TC[]
local function new(a) end
local obj = new({"Vehicle"})

---@generic TCP
---@param a `TCP`2[]Comment
---@return TCP[]
local function new(a) end
local obj = new({"Vehicle"})



---@param a string|integer #     the type
---@return string|integer
local function new(a) end
local obj = new("Vehicle")

---@generic T
---@param a T|integer #     the type
---@return T|integer
local function new(a) end
local obj = new("Vehicle")

---@generic TC
---@param a `TC`|integer #     the type
---@return TC|integer
local function new(a) end
local obj = new("Vehicle")

---@generic TCP
---@param a `TCP`2|integer #     the type
---@return TCP|integer
local function new(a) end
local obj = new("Vehicle")

---

• Generic return can be optional.

---@return T?

local some -- `unknown`
local k, v = next() -- `nil, nil` instead of `unknown, unknown`
local k, v = next({}) -- `nil, nil` instead of `unknown, unknown`
local k, v = next({1}) -- `integer?, integer?` instead of `integer, integer`
local k, v = next({some}) -- `integer?, nil` instead of `integer, unknown`

I'm checking the protoNode (self.proto) in generic:resolve. This node is created in server\script\vm\compiler.lua:1753 in : case 'function.return' where rtn (proto) has optional but vm.createGeneric is not.

lua-language-server/script/vm/compiler.lua

Lines 1751 to 1756 in cb964c6

	if hasGeneric then
	---@cast sign -?
	vm.setNode(source, vm.createGeneric(rtn, sign))
	else
	vm.setNode(source, vm.compileNode(rtn))
	end

Well, I think generic:resolve is the best place.

lua-language-server/script/vm/generic.lua

Lines 139 to 142 in 5f0efc4

	if protoNode:isOptional() then
	result:addOptional()
	end
	return result

But there is some change here. Previously the optional unresolved generic was unknown, but now it chooses between unknown|nil so becomes nil

local some -- `unknown`

-- its correct because generic is `nil`
local k, v = next() -- `nil, nil` instead of `unknown, unknown`

-- is it correct last `nil` with explicit generic `unknown`?
local k, v = next({some}) -- `integer?, nil` instead of `integer, unknown`

I think its OK, otherwise its different issue.

---

Minor changes

• Removed pattern mask m.R('__') from name parsing because it the same as m.S('_'), so nothing changed.
• Rename next variable to nextTp to avoid names conflict.
• NextComment takes 2nd argument boolean peek, not a string 'peek'.

[sumneko]

感谢你的工作！不过关于泛型的改动可能影响会比较大，你是否在大型工程中做过测试？
例如你可以就用LuaLS这个项目自己测试，关注一下 pairs 的结果是否正确？是否会出现错误的警告？

[Issues-translate-bot]

Sumneko Lua translate bot

---

Thank you for your work! However, changes to generics may have a greater impact. Have you tested them in large projects?
For example, you can use the LuaLS project to test it yourself and pay attention to whether the results of pairs are correct? Will there be false warnings?

[TIMONz1535]

My changes only apply to generic captures (aka "code" token parsing `T`) and optional generic return T?, only the next function had an optional return, not pairs. Therefore the behavior of "normal" generics or pairs should not have changed in any way.

This may be an issue

local k, v = next({unknownType}) -- `integer?, nil` instead of `integer?, unknown`
local k, v = next(unknownType) -- `nil, nil` instead of `unknown, unknown`

but for some reason, the first use of nil after checking for nil (which should have been) turns it into an unknown.

so there is no issue.

But...

yes, I wish it wasn't nil initially. I'm not sure I can change that.

upd. Ooooooh wait, in fact this applies to known types too, we do not know that the optionality of V depends on K value, so there will also require a nil check, and unknown just hid it. 🧐