GC-caused scheduler deadlock on RP2350

[herczegzsolt]

Hi!

If I run the following simple code on an RP2350 board, it'll lock-up after only a few (2 or 3) cycles:

package main

import (
	"runtime"
	"machine"


	"time"
)

var (
	uart = machine.Serial
)

func main() {
	uart.Configure(machine.UARTConfig{})
	time.Sleep(1000 * time.Millisecond)
	uart.Write([]byte("\n Start"))
	for {
		uart.Write([]byte("\n Gosched: "))
		runtime.Gosched()
		uart.Write([]byte(" GC: "))
		runtime.GC()
		uart.Write([]byte("done"))
	}
}

The code freezes on the runtime.Gosched() line. By adding println statements to the tinygo standard library, I was able to see that the cause of the stop is that the runtime.schedulerLock mutex is already locked when Gosched() is called. I don't know if anything is happening on the other core, but this locked-up state never resolves itself.

The issue does not seem to happen, if I don't call runtime.GC() - so the lockup is somehow related.

I understand that there's no need to call runtime.GC() in a tight loop like that, but this is only a very simple reproducer of the issue that I've found while doing some memory-heavy math.

Switching the scheduler to none -scheduler=none does workaround the issue, but it means I cannot use any goroutines.

[herczegzsolt]

I'd be happy to work on this and even potentially create a patch, but I'd need some pointers for how the tinygo internals supposed to work. The issue does not occur if I simply use tinygo run, so I'm stuck with on-device debugging for now.

[eliasnaur]

I don't know the answer to your questions, but -scheduler tasks should switch to the single core scheduler and possibly work around the issue while supporting goroutines.

[herczegzsolt]

Thanks! I've checked and -scheduler tasks is indeed a suitable workaround.

Seems like locks are working weirdly on RP2350 in general? #4988

[squidkeeper]

This is good to know, thanks. I find that the freeze just requires the GC to have to run; manipulating it is not necessary.
I have an RP2350 program that freezes after a few GC cycles using -scheduler=cores, but the more extreme test below hangs on the first cycle. -scheduler=tasks makes them happy. It looks like cores should not be the default for now. TinyGo 0.40.1.

package main

import (
	"runtime"
	"time"
)

var leaker *[]byte
var stats runtime.MemStats

func main() {
	for loops := 0; ; loops += 1 {
		foo := make([]byte, 50000)
		leaker = &foo

		runtime.ReadMemStats(&stats)
		println("heap=", stats.HeapInuse, "mallocs=", stats.Mallocs, "frees=", stats.Frees, "loops=", loops)
		time.Sleep(100 * time.Millisecond)
	}
}
// Happy: tinygo build -target=pico2-w -serial=uart -scheduler=tasks -o leaker.elf cmd/leaker/main.go
// Sad:   tinygo build -target=pico2-w -serial=uart -scheduler=cores -o leaker.elf cmd/leaker/main.go
// Happy keeps the GC busy but runs forever. Sad freezes before the GC has finished once.

[vejmarie]

Hello, I think I have patches which would work. At least all of your examples work with them and the cores scheduler. I am "new" to tinygo and its development process, but I work on a program for the rp2350 where I loved to enable multicore that is why I spent some time on it.
Some observations, to make it work on the RP2350, I am using pure software spin locks, I am also using a reset of the core1 before entering core1 payload and put it into a known state. I also enable extexcall, and last, I am splitting the interrupt handler per core like this

    intr := interrupt.New(sioIrqFifoProc0, func(intr interrupt.Interrupt) {
            status := rp.SIO.GetFIFO_ST_VLD()
            if status != 0 {
            switch rp.SIO.FIFO_RD.Get() {
            case 1:
                    if currentCPU() == 1 {
                            gcInterruptHandler(1)
                    } else {
                            gcInterruptHandler(0)
                    }
            }
            }
    })

As both core share the same interrupt number (but have an interrupt dedicated for them), but have a common interrupt vector (which means the same code is executed).

With that said, for compute pure application it works, but for thing which requires external interrupt it is a little bit more tricky as the M33 have their own interrupt controller and by default, we can't control the interrupt on core1, or have no option to "configure" core1 from application level, as register updates needs to be done through core1 only and the scheduler doesn't provide easy way to stick a task to a core. Is that something somebody is working on ?
I can upstream my patches, is that just a PR from my tinygo fork ?

[deadprogram]

This is a very intriguing conversation! @vejmarie I would be very interested in seeing your proposed changes.

Please see this page for info on how to submit PR:
https://tinygo.org/docs/guides/contributing/#how-to-use-our-github-repository

[deadprogram]

for thing which requires external interrupt

Can you be specific about that please? Which interrupts would those be?