Monaco EVK fails to boot with UFS issues in qcom-next 6.19.0-rc4-g65bb05266d47

[lool]

Hi

current qcom-next sometimes fails to boot on Monaco EVK with some UFS related errors:

[   10.232387] ufshcd-qcom 1d84000.ufs: UPIU[0] - Response UPIU phys@0xffffe0080
[   10.239713] UPIU RSP: 00000000: 00000000 00000000 00000000 00000000
[   10.246154] UPIU RSP: 00000010: 00000000 00000000 00000000 00000000
[   10.252593] UPIU RSP: 00000020: 00000000 00000000 00000000 00000000
[   10.259034] UPIU RSP: 00000030: 00000000
Begin: Running /scripts/local-block ... done.
[   10.263072] ufshcd-qcom 1d84000.ufs: UPIU[0] - PRDT - 0 entries  phys@0xffffe0100
Begin: Running /scripts/local-block ... done.
[   11.541599] ufshcd-qcom 1d84000.ufs: ufshcd_abort: Device abort task at tag 0
[   11.548962] ufshcd-qcom 1d84000.ufs: UPIU[0] - Transfer Request Descriptor phys@0xffffdf000
[   11.557539] UPIU TRD: 00000000: 11000000 00000000 0000000f 00000000
[   11.563973] UPIU TRD: 00000010: fffe0000 0000000f 00800080 01000000
[   11.570411] ufshcd-qcom 1d84000.ufs: UPIU[0] - Request UPIU phys@0xffffe0000
[   11.577652] UPIU REQ: 00000000: 00000000 00000000 00000000 00000000
[...]
[   22.366659] UPIU RSP: 00000030: 00000000
[   22.370727] ufshcd-qcom 1d84000.ufs: UPIU[0] - PRDT - 0 entries  phys@0xffffe0100
[   22.379677] ufshcd-qcom 1d84000.ufs: ufshcd_verify_dev_init: NOP OUT failed -5
[   22.392300] ufshcd-qcom 1d84000.ufs: error -EIO: Initialization failed with error -5
[   22.400420] ufshcd-qcom 1d84000.ufs: error -EIO: ufshcd_pltfrm_init() failed
[   22.407721] ufshcd-qcom 1d84000.ufs: probe with driver ufshcd-qcom failed with error -5

I can't provide a link to a CI device log as we don't have a Monaco EVK in the OSTT LAVA, but I do have CI for qcom-next builds; here's a flat image:
https://quic-yocto-fileserver-1029608027416.us-central1.run.app/qualcomm-linux/qcom-deb-images/20968824236-1/

I did noticed a thread about tcsrcc_sm8750 with similar error messages on linux-arm-msm@, and while it's present in the initramfs, it doesn't get loaded, not sure if that's relevant.

Attaching dmesg from a local boot on RB4:
dmesg-ufs-monaco-evk.txt

Sometimes, doing a warm reboot from initramfs works after that, sometimes cold booting works.

[loicpoulain]

Form the logs:

[    8.052820] ufshcd-qcom 1d84000.ufs: ufshcd_abort: Device abort task at tag 0
[    8.060266] host_regs: 00000000: 1587031f 00000000 00000300 00000000
[    8.066844] host_regs: 00000010: 01000000 00010217 00000000 00000000
[    8.073443] host_regs: 00000020: 00000000 00071ef5 00000000 00000000
[    8.080019] host_regs: 00000030: 0000000f 00000001 00000000 00000000
[    8.087086] host_regs: 00000040: 00000000 00000000 00000000 00000000
[    8.093723] host_regs: 00000050: fffdf000 0000000f 00000001 00000000
[    8.100309] host_regs: 00000060: 00000001 00000000 00000000 00000000
[    8.106897] host_regs: 00000070: fffde000 0000000f 00000000 00000000
[    8.113481] host_regs: 00000080: 00000001 00000000 00000000 00000000
[    8.120057] host_regs: 00000090: 00000016 00000000 00000000 0000000c

[sampnimm2629]

Hi @lool

Here we are observing NOP OUT Failed, do we have any working vs failing reference tags and what is the frequency?

[ 22.379677] ufshcd-qcom 1d84000.ufs: ufshcd_verify_dev_init: NOP OUT failed -
5
[ 22.392300] ufshcd-qcom 1d84000.ufs: error -EIO: Initialization failed with e
rror -5
[ 22.400420] ufshcd-qcom 1d84000.ufs: error -EIO: ufshcd_pltfrm_init() failed
[ 22.407721] ufshcd-qcom 1d84000.ufs: probe with driver ufshcd-qcom failed wit
h error -5

[loicpoulain]

I suspect the DTB you're using is not up to date (dtb_a/dtb_b partitions).

[lool]

I suspect the DTB you're using is not up to date (dtb_a/dtb_b partitions).

the dtb is part of the flat image, I never manage it manually; we've confirmed with @loicpoulain that it's correct

[lool]

Hi @lool

Here we are observing NOP OUT Failed, do we have any working vs failing reference tags and what is the frequency?

[ 22.379677] ufshcd-qcom 1d84000.ufs: ufshcd_verify_dev_init: NOP OUT failed - 5 [ 22.392300] ufshcd-qcom 1d84000.ufs: error -EIO: Initialization failed with e rror -5 [ 22.400420] ufshcd-qcom 1d84000.ufs: error -EIO: ufshcd_pltfrm_init() failed [ 22.407721] ufshcd-qcom 1d84000.ufs: probe with driver ufshcd-qcom failed wit h error -5

On failing boots, this is repeated many times until driver gives up; for me, this failed in my first few boot attempts (perhaps 2-3).

This morning, the same boot failure happened, but instead of turning the board off I did a hot reboot from the initramfs (reboot -f) and that booted, so it seems to be some hw init/raciness.

[lool]

Here we are observing NOP OUT Failed, do we have any working vs failing reference tags and what is the frequency?

I don't have previous working qcom-next tags to offer, I can only confirm that mainline and 6.18 Debian kernel + config tweaks seem to work in a stable fashion, including UFS, albeit with other features lacking.