linux shell argument list too long rsync or cp

Jul 31, 2015 | Linux, shell

I needed to copy files generated by doxygen from one directory into another for a large opensource C++ project. Sadly there were too many files in the directory, so bash started complaining 🙁 cp and rsync died out with the error of argument list too long. initially I figured I could generate it all from scratch in new location but it was quicker and easier to use a for loop to rsync the files over 🙂

some info:

  • all files start with alphabetic characters.
  • there are no spaces in the names
  • all files are in single directory

I realized bash expansion would work here.

Using for loop

Notice I only stepped between A and B because I didn’t want to sit there for an hour while it listed all the files. this worked well, it listed all files and I was sure it would suite my purposes. now the real deal!

sometimes you might still get the error even for each letter, for example I still had too many files starting with D and Q. so I just changed where I globbed :

this allows me to further iterate a thru z but after starting the files with the letter d. Now what happens if you happen to have files starting with numbers? simply switch the letters for numbers.

You can use any other command you need in place of rsync. like mv cp mkdir or any custom commands.

Globbing

Now if you don’t want to use for loops you can glob them in a one liner like so :

and the actual command using cp and globbing

and again going a level deeper

Voila! argument list too long is now vanquished! do any of you have a better way of dealing with this? let me know!

More info about globbing.