A query about the update of 'msk->sk_sndbuf'

[Dwyane-Yan]

Pre-requisites

• A similar question has not been reported before.
• mptcp.dev website does not cover my case.
• The wiki doesn't cover my case.
• This is not a question related to the current behavior, an issue or a feature requst: if it is, please use another template even if it is a question: we will need details about your system: kernel version, config, etc.

My question

Hi, @matttbe .
I've been looking into the update logic of 'msk->sk_sndbuf' recently, and I have a question for the community. When there is only one subflow, the sk_sndbuf of the newsk in 'mptcp_stream_accept' is not updated to msk->sk_sndbuf.

From my understanding, msk->sk_sndbuf should be updated in the following scenarios:

• in 'tcp_new_space()' through the callback to 'subflow_write_space';
• second, when the client connection is fully established, msk->first triggers an update via 'sk_rx_dst_set', which calls 'subflow_finish_connect' -> 'mptcp_propagate_state' -> update msk->sk_sndbuf.

However, it seems that the 'newsk' in 'mptcp_stream_accept' is overlooked. This means that when the connection is fully established and the server sends data to the client, there is a significant discrepancy between 'msk->sk_sndbuf' and 'msk->first->sk_sndbuf'. During my testing, I printed the information in 'mptcp_sendmsg' and observed something like this:

MPTCP: msk:00000000e55b09db, msk->sndbuf:20480, msk->first->sndbuf:2626560.

The reason for this discrepancy is that after the connection is established, the 'newsk->sk_sndbuf' from accept is updated in 'tcp_init_buffer_space' -> 'tcp_sndbuf_expand', but we haven't updated the server's msk->sk_sndbuf afterward. This leads to a high probability of encountering an '-EAGAIN' error when MSG_DONTWAIT is set."

With the patch below applied, the sk_sndbuf can be updated in time.

@@ -4198,6 +4200,7 @@ static int mptcp_stream_accept(struct socket *sock, struct socket *newsock,
 
                mptcp_graft_subflows(newsk);
                mptcp_rps_record_subflows(msk);
+               __mptcp_sync_sndbuf(newsk);

My question is, is this inconsistency in the sndbuf expected here? Or is this a bug in MPTCP that needs to be fixed?

[matttbe]

Hi @Dwyane-Yan,

Thank you for your question. (But no need to target it only to me, other people can answer too :) )

MPTCP: msk:00000000e55b09db, msk->sndbuf:20480, msk->first->sndbuf:2626560.

(...)

My question is, is this inconsistency in the sndbuf expected here? Or is this a bug in MPTCP that needs to be fixed?

Good catch, msk->sk_sndbuf should likely be synced at accept() time. It looks like a bug indeed. Please send a patch fixing it (with a Fixes tag, and targeting mptcp-net).

But before, just to be sure, did you experiment that on top of our latest export branch? I'm saying that because @pabeni did a few modifications around the buffers, but more around the rcvbuf and not there I think.

Also, please check if it is not better to call __mptcp_propagate_sndbuf() instead (maybe not, to be checked), and the appropriated locks are held.

[geliangtang]

I think using __mptcp_propagate_sndbuf(newsk, ssk) is much better than using __mptcp_sync_sndbuf(newsk) in mptcp_stream_accept().

[Dwyane-Yan]

Thank you for everyone's suggestions. This issue was discovered during the development of 'NVMe over MPTCP' by me and geliang. Perhaps I can add the 'reported-by' and 'suggested-by' tags in the new patch.

[Dwyane-Yan]

Hi, Paolo seems to have questions about this issue, so I've reopened it, and subsequent patches will include a 'Closes' tag.

[Dwyane-Yan]

Hi, @matttbe @pabeni ,
In 'MoM - 3rd of December 2025', this issue is commented as "might be due to WIP patches?"

So I make a easy test at the top of 'master' branch of kernel, and it can be reproduced:

Kernel patch:

diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c
index e212c1374bd0..490f529346ab 100644
--- a/net/mptcp/protocol.c
+++ b/net/mptcp/protocol.c
@@ -1899,6 +1899,8 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 
        pfrag = sk_page_frag(sk);
 
+       pr_err("msk:%p, msk->snd_buf:%d, first->sndbuf:%d", msk, sk->sk_sndbuf, msk->first->sk_sndbuf);
+       pr_err("Clear Log");
        while (msg_data_left(msg)) {
                int total_ts, frag_truesize = 0;
                struct mptcp_data_frag *dfrag;

Test cases:
I create a easy mptcp connection, and let server send messages first. The code is attached below:
1、mptcp_server.c:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <errno.h>

#define SERVER_PORT 8080
#define DATA_SIZE (2000 * 1024) // 2000 KB

int main() {
    int server_fd, client_fd;
    struct sockaddr_in server_addr, client_addr;
    socklen_t client_len = sizeof(client_addr);
    char *data_buffer;
    int ret;

    data_buffer = malloc(DATA_SIZE);
    if (!data_buffer) {
        perror("Failed to allocate data buffer");
        exit(EXIT_FAILURE);
    }
    memset(data_buffer, 'A', DATA_SIZE); 

    // 创建MPTCP Socket
    server_fd = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);
    if (server_fd < 0) {
        perror("Failed to create MPTCP socket");
        free(data_buffer);
        exit(EXIT_FAILURE);
    }

    memset(&server_addr, 0, sizeof(server_addr));
    server_addr.sin_family = AF_INET;
    server_addr.sin_addr.s_addr = INADDR_ANY;
    server_addr.sin_port = htons(SERVER_PORT);

    if (bind(server_fd, (struct sockaddr *)&server_addr, sizeof(server_addr)) < 0) {
        perror("Bind failed");
        close(server_fd);
        free(data_buffer);
        exit(EXIT_FAILURE);
    }

    if (listen(server_fd, 5) < 0) {
        perror("Listen failed");
        close(server_fd);
        free(data_buffer);
        exit(EXIT_FAILURE);
    }

    printf("MPTCP Server listening on port %d...\n", SERVER_PORT);

    client_fd = accept(server_fd, (struct sockaddr *)&client_addr, &client_len);
    if (client_fd < 0) {
        perror("Accept failed");
        close(server_fd);
        free(data_buffer);
        exit(EXIT_FAILURE);
    }
    printf("Client connected: %s:%d\n", inet_ntoa(client_addr.sin_addr), ntohs(client_addr.sin_port));

    printf("Sending %d KB data in one shot with MSG_DONTWAIT...\n", DATA_SIZE / 1024);

    ret = send(client_fd, data_buffer, DATA_SIZE, MSG_DONTWAIT);

    if (ret > 0) {
        printf("Successfully sent %d bytes (%d KB)\n", ret, ret / 1024);
    } else if (ret < 0) {
        if (errno == EAGAIN || errno == EWOULDBLOCK) {
            printf("Send would block (EAGAIN), only %d bytes sent\n", ret);
        } else {
            perror("Send failed");
        }
    } else {
        printf("No data sent (return 0)\n");
    }

    sleep(2);

    close(client_fd);
    close(server_fd);
    free(data_buffer);

    printf("Server exiting.\n");
    return 0;
}

2、mptcp_client.c

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>

#define SERVER_IP "127.0.0.1"
#define SERVER_PORT 8080
#define BUFFER_SIZE (2000 * 1024 * 10) // 2000 KB

int main() {
    int client_fd;
    struct sockaddr_in server_addr;
    char *recv_buffer;
    int ret;

    recv_buffer = malloc(BUFFER_SIZE);
    if (!recv_buffer) {
        perror("Failed to allocate receive buffer");
        exit(EXIT_FAILURE);
    }

    client_fd = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);
    if (client_fd < 0) {
        perror("Failed to create MPTCP socket");
        free(recv_buffer);
        exit(EXIT_FAILURE);
    }

    memset(&server_addr, 0, sizeof(server_addr));
    server_addr.sin_family = AF_INET;
    server_addr.sin_addr.s_addr = inet_addr(SERVER_IP);
    server_addr.sin_port = htons(SERVER_PORT);

    if (connect(client_fd, (struct sockaddr *)&server_addr, sizeof(server_addr)) < 0) {
        perror("Connect failed");
        close(client_fd);
        free(recv_buffer);
        exit(EXIT_FAILURE);
    }
    printf("Connected to server: %s:%d\n", SERVER_IP, SERVER_PORT);

    printf("Waiting to receive data...\n");

    ret = recv(client_fd, recv_buffer, BUFFER_SIZE, 0);
    if (ret > 0) {
        printf("Received %d bytes (%d KB) in one shot\n", ret, ret / 1024);

        if (ret > 0 && recv_buffer[0] == 'A') {
            printf("Data verification: first byte is 'A' (OK)\n");
        }
    } else if (ret == 0) {
        printf("Server closed connection\n");
    } else {
        perror("Receive failed");
    }

    close(client_fd);
    free(recv_buffer);

    printf("Client exiting.\n");
   return 0;
}

3、test_sndbuf.sh

gcc -o server mptcp_server.c
gcc -o client mptcp_client.c


./server &
sleep 3
./client

After running it in our mptcp-upstream-virtme-docer, the kernel log print:

[ 55.819975] MPTCP: msk:0000000039d925e3, msk->snd_buf:20480, first->sndbuf:2626560

So I think the root cause is shown in previous discussions:

However, it seems that the 'newsk' in 'mptcp_stream_accept' is overlooked. This means that when the connection is fully established and the server sends data to the client, there is a significant discrepancy between 'msk->sk_sndbuf' and 'msk->first->sk_sndbuf'. During my testing, I printed the information in 'mptcp_sendmsg' and observed something like this:

WDYT?