Many __mptcp_init4_subsockets mptcp_add_sock failed ret: -1 kernel errors lately.

[ioogithub]

Hi,

I am using MPTCP in the openmptcprouter version v0.59.1-5.4 r0+16594-ce92de8c8c implementation.
Linux OpenMPTCProuter 5.4.194 #0 SMP Tue May 17 22:11:28 2022 x86_64 GNU/Linux

I have been getting a log of kernel errors in the system log in the past few days:

Mon Oct 16 15:55:15 2023 kern.err kernel: [ 6634.141479] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:15 2023 kern.err kernel: [ 6634.141940] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:15 2023 kern.err kernel: [ 6634.142388] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:15 2023 kern.err kernel: [ 6634.142822] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:15 2023 kern.err kernel: [ 6634.143450] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:15 2023 kern.err kernel: [ 6634.143692] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:15 2023 kern.err kernel: [ 6634.143877] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:15 2023 kern.err kernel: [ 6634.144350] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:15 2023 kern.err kernel: [ 6634.144755] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:15 2023 kern.err kernel: [ 6634.145150] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:30 2023 kern.warn kernel: [ 6649.499339] net_ratelimit: 5 callbacks suppressed
Mon Oct 16 15:55:30 2023 kern.err kernel: [ 6649.499341] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:30 2023 kern.err kernel: [ 6649.500079] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 15:55:30 2023 kern.err kernel: [ 6649.500423] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:05:16 2023 kern.err kernel: [ 7235.704584] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:05:16 2023 kern.err kernel: [ 7235.704930] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:12:57 2023 kern.err kernel: [ 7696.496316] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:12:57 2023 kern.err kernel: [ 7696.496645] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:12:57 2023 kern.err kernel: [ 7696.496950] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:12:57 2023 kern.err kernel: [ 7696.497257] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:12:57 2023 kern.err kernel: [ 7696.497546] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:12:57 2023 kern.err kernel: [ 7696.497812] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:12:57 2023 kern.err kernel: [ 7696.498066] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:12:57 2023 kern.err kernel: [ 7696.498318] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:12:57 2023 kern.err kernel: [ 7696.498548] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:12:57 2023 kern.err kernel: [ 7696.499893] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:13:10 2023 kern.warn kernel: [ 7709.300992] net_ratelimit: 24 callbacks suppressed
Mon Oct 16 16:13:10 2023 kern.err kernel: [ 7709.300993] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:13:10 2023 kern.err kernel: [ 7709.301439] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:13:10 2023 kern.err kernel: [ 7709.301627] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:29:20 2023 kern.err kernel: [ 8679.505203] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:29:20 2023 kern.err kernel: [ 8679.505398] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:29:20 2023 kern.err kernel: [ 8679.505588] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:29:20 2023 kern.err kernel: [ 8679.505801] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:29:20 2023 kern.err kernel: [ 8679.505984] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:29:20 2023 kern.err kernel: [ 8679.506165] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:29:20 2023 kern.err kernel: [ 8679.506406] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:29:20 2023 kern.err kernel: [ 8679.506592] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:29:20 2023 kern.err kernel: [ 8679.506779] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:29:20 2023 kern.err kernel: [ 8679.506963] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:38:46 2023 kern.warn kernel: [ 9245.244057] net_ratelimit: 18 callbacks suppressed
Mon Oct 16 16:44:26 2023 kern.err kernel: [ 9585.710471] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:44:26 2023 kern.err kernel: [ 9585.710661] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:44:26 2023 kern.err kernel: [ 9585.710856] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:44:26 2023 kern.err kernel: [ 9585.711040] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:44:26 2023 kern.err kernel: [ 9585.711225] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:44:26 2023 kern.err kernel: [ 9585.711404] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:44:26 2023 kern.err kernel: [ 9585.711588] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:44:39 2023 kern.err kernel: [ 9598.505907] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:44:39 2023 kern.err kernel: [ 9598.506096] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:44:39 2023 kern.err kernel: [ 9598.506274] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:50:32 2023 kern.err kernel: [ 9951.776239] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:50:32 2023 kern.err kernel: [ 9951.776436] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:50:32 2023 kern.err kernel: [ 9951.776617] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:50:32 2023 kern.err kernel: [ 9951.776787] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 16:50:32 2023 kern.err kernel: [ 9951.776952] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:10 2023 kern.err kernel: [10829.839483] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:10 2023 kern.err kernel: [10829.839668] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:10 2023 kern.err kernel: [10829.841189] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:10 2023 kern.err kernel: [10829.841363] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:10 2023 kern.err kernel: [10829.841524] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:10 2023 kern.err kernel: [10829.841736] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:10 2023 kern.err kernel: [10829.841920] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:10 2023 kern.err kernel: [10829.842077] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:10 2023 kern.err kernel: [10829.842235] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:10 2023 kern.err kernel: [10829.842392] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:26 2023 kern.warn kernel: [10845.189114] net_ratelimit: 6 callbacks suppressed
Mon Oct 16 17:05:26 2023 kern.err kernel: [10845.189124] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:26 2023 kern.err kernel: [10845.189499] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:05:26 2023 kern.err kernel: [10845.189658] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:06:25 2023 kern.err kernel: [10904.076000] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:06:25 2023 kern.err kernel: [10904.076173] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:06:25 2023 kern.err kernel: [10904.076333] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:06:25 2023 kern.err kernel: [10904.076490] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:06:25 2023 kern.err kernel: [10904.076653] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:06:25 2023 kern.err kernel: [10904.076826] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:06:25 2023 kern.err kernel: [10904.076996] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:06:25 2023 kern.err kernel: [10904.077149] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:06:25 2023 kern.err kernel: [10904.077309] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:06:25 2023 kern.err kernel: [10904.077469] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:07:26 2023 kern.err kernel: [10965.502737] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:07:26 2023 kern.err kernel: [10965.505369] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:07:26 2023 kern.err kernel: [10965.505538] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:07:26 2023 kern.err kernel: [10965.505695] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:07:26 2023 kern.err kernel: [10965.505847] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:08:27 2023 kern.err kernel: [11026.951586] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:08:27 2023 kern.err kernel: [11026.951751] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:08:27 2023 kern.err kernel: [11026.951901] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:08:27 2023 kern.err kernel: [11026.952047] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:16:08 2023 kern.err kernel: [11487.729882] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:16:08 2023 kern.err kernel: [11487.730051] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:16:08 2023 kern.err kernel: [11487.730668] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:16:08 2023 kern.err kernel: [11487.730831] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:16:08 2023 kern.err kernel: [11487.730983] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:16:08 2023 kern.err kernel: [11487.731170] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:16:08 2023 kern.err kernel: [11487.731314] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:16:08 2023 kern.err kernel: [11487.731453] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:16:08 2023 kern.err kernel: [11487.731576] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:16:08 2023 kern.err kernel: [11487.731700] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:26:38 2023 kern.err kernel: [12117.466211] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:26:38 2023 kern.err kernel: [12117.466530] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:26:38 2023 kern.err kernel: [12117.466689] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:26:38 2023 kern.err kernel: [12117.466855] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1
Mon Oct 16 17:27:11 2023 kern.err kernel: [12150.735869] __mptcp_init4_subsockets mptcp_add_sock failed ret: -1

I don't see any other errors in the log that would indicate issues.

Here are some statistics:

MPCapableSYNTX                  	12194
MPCapableSYNACKRX               	11464
MPCapableACKRX                  	4
MPCapableFallbackACK            	0
MPCapableFallbackSYNACK         	700
MPCapableRetransFallback        	0
MPTCPCsumEnabled                	0
MPTCPRetrans                    	4479
MPFailRX                        	0
MPCsumFail                      	0
MPFastcloseRX                   	931
MPFastcloseTX                   	2592
MPFallbackAckSub                	0
MPFallbackAckInit               	0
MPFallbackDataSub               	0
MPFallbackDataInit              	0
MPRemoveAddrSubDelete           	858
MPJoinNoTokenFound              	0
MPJoinAlreadyFallenback         	0
MPJoinSynTx                     	1268212
MPJoinSynRx                     	12
MPJoinSynAckRx                  	238691
MPJoinSynAckHMacFailure         	0
MPJoinAckRx                     	12
MPJoinAckHMacFailure            	0
MPJoinAckMissing                	0
MPJoinAckRTO                    	1
MPJoinAckRexmit                 	2560
NoDSSInWindow                   	0
DSSNotMatching                  	0
InfiniteMapRx                   	0
DSSNoMatchTCP                   	0
DSSTrimHead                     	0
DSSSplitTail                    	0
DSSPurgeOldSubSegs              	0
AddAddrRx                       	11718
AddAddrTx                       	11464
RemAddrRx                       	858
RemAddrTx                       	0
MPJoinAlternatePort             	0
MPCurrEstab                     	116

I saw in this issue: #245 that you support 32 subflows per connection. I am aggregating two wan connections on the client (openmptcprouter) each with 1 IP address and 1 IP on the server (VPS).

I have Fullmesh subflows for each pair of IP addresses set to a value of 3. What do these errors mean, why is the add_sock process failing and when it does what are the implications, does this mean these connections are not aggregating properly. How can I troubleshoot or fix the problem?

Thank you.

[matttbe]

Hello,

Linux OpenMPTCProuter 5.4.194 #0 SMP Tue May 17 22:11:28 2022 x86_64 GNU/Linux

This version is a bit old, maybe the issue has already been fixed.

If you have mptcp_add_sock failed ret: -1, it means No more space for more subflows. So it looks like you are trying to create too many subflows.

I have Fullmesh subflows for each pair of IP addresses set to a value of 3.

I'm not sure what it represents (best is to report this issue to OpenMPTCProuter devs I think) but I guess it will try to create more subflows than needed.