How can i run the fpart + cpio + GNU Parallel command with SSH

[Fettah]

The command under your point (5.9)
time parallel --will-cite -j ${threads} --pipepart --round-robin --delay .1 --block 1M -a /home/ec2-user/fpart-files-to-transfer.0 sudo "cpio -dpmL /mnt/efs/01/tutorial/parallelcpio/${instance_id}" &

How can I run it from an on premise server ? I am syncing data between a virtual machine and en ec2 that has EFS mounted. I do need to ssh here.

something like this:
time parallel --will-cite -j ${threads} --pipepart --round-robin --delay .1 --block 1M -a /home/ec2-user/fpart-files-to-transfer.0 "cpio -dpmL ec2-user@ec2-ip-address:/docker-efs/docker/parallelcpio"

[darrylsosborne]

@Fettah - I would recommend running the command from the on-prem virtual machine directly to EFS. If you have AWS Direct Connect or VPN connections between your on-prem environment on the VPC connected to your EFS file system, then you can issue the command as-is (pending any mountpoint changes for your environment). Mount the EFS file system from on-prem using an IP address of one of the EFS mount targets. That's the recommendation because it can still take advantage of the distributed data storage design of EFS.

[Fettah]

I am running a docker container that will do the sync for me, because of some restrictions on installing packages on a linux VM.
I was not able to mount EFS with tls directly from the docker container, so I spinn up an ec2 instance and I am syncing file from docker container using ssh and the EC2 instance.

[Fettah]

Thank you here. I have managed to mount the EFS directly from docker.

one question: what impact does "--block 1M" does have on the sync in your example ? if I am to sync very small files, (saying 100KB), will changing the value to "--block 100KB" make it faster ?