Categories
Linux shell

linux shell argument list too long rsync or cp

I needed to copy files generated by doxygen from one directory into another for a large opensource C++ project. Sadly there were too many files in the directory, so bash started complaining 🙁 cp and rsync died out with the error of argument list too long. initially I figured I could generate it all from scratch in new location but it was quicker and easier to use a for loop to rsync the files over 🙂

some info:

  • all files start with alphabetic characters.
  • there are no spaces in the names
  • all files are in single directory

I realized bash expansion would work here.

Using for loop

for x in {a..b}
do
    echo $x*
done

Notice I only stepped between A and B because I didn’t want to sit there for an hour while it listed all the files. this worked well, it listed all files and I was sure it would suite my purposes. now the real deal!

for x in {a..z}
do
echo $x
rsync -az /backups/doxygen/$x* /home/user/current/directory/
done

sometimes you might still get the error even for each letter, for example I still had too many files starting with D and Q. so I just changed where I globbed :

for x in {a..z}
do
echo $x
rsync -az /backups/doxygen/d$x* /home/user/current/directory/
done

this allows me to further iterate a thru z but after starting the files with the letter d. Now what happens if you happen to have files starting with numbers? simply switch the letters for numbers.

for x in {0..9}
do
echo $x
rsync -az /backups/doxygen/$x* /home/user/current/directory/
done

You can use any other command you need in place of rsync. like mv cp mkdir or any custom commands.

for x in {a..z}
do
echo $x
mv /backups/doxygen/$x* /home/user/current/directory/
done

Globbing

Now if you don’t want to use for loops you can glob them in a one liner like so :

ls /backups/doxygen/[x-z]*

and the actual command using cp and globbing

cp -r /backups/doxygen/[a-z]* /home/user/current/directory/

and again going a level deeper

cp -r /backups/doxygen/d[a-z]* /home/user/current/directory/

Voila! argument list too long is now vanquished! do any of you have a better way of dealing with this? let me know!

More info about globbing.

Categories
Linux tuts

Debian package management speed ups

debian logo
Debian is a Linux distro that’s used by millions of machines all over the planet.

No one likes to sit around waiting for slow mirrors while updating multiple packages but its a fact of life usually. In debian it means typing apt-get update and sitting around for a while, then doing the actual install or upgrade and getting some coffee. what if you could speed the process along somewhat? well now just like we showed you how to speed up apt downloads for ubuntu you can speed up the apt speeds for Debian! This way you can focus more on clashing some clans or something…. whatever you do in your free time that is.

Httpredir

In comes httpredir, “It uses the geographic and network location of the user and the mirrors, the architecture of the requested files, IP address family, the availability and freshness of the mirrors, and a few other things” to find the closest and fastest mirror of data for you. This gives you the quickest way to get your files without resorting to new tools or alternate package managers. This is already there for Jessie ( debian 8 ) so no need to edit that, but for everyone else on older releases this will bring wonderful speedups.

setting up httpredir is simple :

edit your /etc/apt/soucres.list

deb http://ftp.us.debian.org/debian sid main
deb-src http://ftp.us.debian.org/debian sid main

replace with

deb http://httpredir.debian.org/debian sid main
deb-src http://httpredir.debian.org/debian sid main
Categories
Linux

Sample apt.sources file with

Remove the us. part if you arent on a US server.

change the word precise to whatever your version of ubuntu is.

 

 

deb mirror://mirrors.ubuntu.com/mirrors.txt precise main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/mirrors.txt precise-updates main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/mirrors.txt precise-backports main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/mirrors.txt precise-security main restricted universe multiverse

deb http://us.archive.ubuntu.com/ubuntu precise main restricted
deb-src http://us.archive.ubuntu.com/ubuntu precise main restricted
deb http://us.archive.ubuntu.com/ubuntu precise-updates main restricted
deb-src http://us.archive.ubuntu.com/ubuntu precise-updates main restricted
deb http://us.archive.ubuntu.com/ubuntu precise universe
deb-src http://us.archive.ubuntu.com/ubuntu precise universe
deb http://us.archive.ubuntu.com/ubuntu precise-updates universe
deb-src http://us.archive.ubuntu.com/ubuntu precise-updates universe
deb http://us.archive.ubuntu.com/ubuntu precise multiverse
deb-src http://us.archive.ubuntu.com/ubuntu precise multiverse
deb http://us.archive.ubuntu.com/ubuntu precise-updates multiverse

deb http://security.ubuntu.com/ubuntu precise-security main restricted universe multiverse
deb-src http://security.ubuntu.com/ubuntu precise-security main restricted universe multiverse

deb http://archive.canonical.com/ubuntu precise partner
deb-src http://archive.canonical.com/ubuntu precise partner
deb http://extras.ubuntu.com/ubuntu precise main
deb-src http://extras.ubuntu.com/ubuntu precise main

 

Categories
Linux shell tuts

Find out whats taking up all the hdd space

[et_pb_section admin_label=”Section” fullwidth=”on” specialty=”off”][et_pb_fullwidth_post_title admin_label=”Fullwidth Post Title” title=”on” meta=”on” author=”off” date=”on” categories=”on” comments=”off” featured_image=”on” featured_placement=”background” parallax_effect=”off” parallax_method=”on” text_orientation=”center” text_color=”dark” text_background=”on” text_bg_color=”rgba(255,255,255,0.9)” module_bg_color=”rgba(255,255,255,0)” title_all_caps=”off” use_border_color=”off” border_color=”#ffffff” border_style=”solid”]

[/et_pb_fullwidth_post_title][/et_pb_section][et_pb_section admin_label=”section”][et_pb_row admin_label=”row”][et_pb_column type=”4_4″][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” use_border_color=”off” border_color=”#ffffff” border_style=”solid”]

Running out of hdd space is pretty annoying. So if you are running out of space and need to find out whats taking up all that space type the following command to find out more :

du -h  |grep '[0-9]G'

 

[/et_pb_text][/et_pb_column][/et_pb_row][et_pb_row admin_label=”row”][et_pb_column type=”4_4″][et_pb_text admin_label=”bottom above footer” background_layout=”light” text_orientation=”left” use_border_color=”off” border_color=”#ffffff” border_style=”solid”]

[/et_pb_text][et_pb_divider admin_label=”Divider” color=”#ffffff” show_divider=”off” divider_style=”solid” divider_position=”top” hide_on_mobile=”on”]

[/et_pb_divider][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” use_border_color=”off” border_color=”#ffffff” border_style=”solid”]

[jetpack-related-posts]

[/et_pb_text][/et_pb_column][/et_pb_row][/et_pb_section]

Categories
Linux shell

Start Xvfb on boot on Centos Linux

XVFB

Xvfb or X virtual framebuffer

X virtual framebuffer is a display server using the X11 protocol. In contrast to other display servers, it performs all graphical operations in memory without showing any screen output. which makes it ideal for some surprising uses. e.g.

Xvfb :1 &
xv -display :1 &
import -display :1 -window root image.png

(above snippet via Wikipedia)

I had to run xvfb automatically on boot on a centos system, here’s the init script I used to carry out the task. add the following as a script to /etc/init.d , chmod +x the script then chkconfig xvfb on

* Note: see the :11 in the middle of the script? change that to another number to change the display port.

   #!/bin/bash
   # chkconfig: 345 95 50
   # description: Starts xvfb on display 11
   # why 11? dunno I just work here. :( -FB
   # 9/30/13 r1 - initial addition -FB

   if [ -z "$1" ]; then
   echo "`basename $0` {start|stop}"
       exit
   fi

   case "$1" in
   start)
       /usr/bin/Xvfb :11 &
   ;;

   stop)
       killall Xvfb
   ;;
   esac

[/et_pb_text][/et_pb_column][/et_pb_row][et_pb_row admin_label=”row”][et_pb_column type=”4_4″][et_pb_text admin_label=”bottom above footer” background_layout=”light” text_orientation=”left” use_border_color=”off” border_color=”#ffffff” border_style=”solid”]

[/et_pb_text][et_pb_divider admin_label=”Divider” color=”#ffffff” show_divider=”off” divider_style=”solid” divider_position=”top” hide_on_mobile=”on”] [/et_pb_divider][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” use_border_color=”off” border_color=”#ffffff” border_style=”solid”]

[jetpack-related-posts]

[/et_pb_text][/et_pb_column][/et_pb_row][/et_pb_section]

Categories
Linux shell

Bash for loops sequential counting

 

Bash for loops are very useful in everyday life. I am going to post some simple examples where I combine bash for loops with other simple techniques like bash stepping etc.

Want to count from 1 – 100 and list each number sequentially?

for i in {1..100}; do   echo $i; done

Same as above but add a timestamp?

for i in {1..100}; do   echo $(date +"%D %I:%M:%S") - $i; done

Again same as above but now we step every 3 numbers instead of every 1

for i in {1..3..100}; do   echo $(date +"%D %I:%M:%S") - $i; done

 

now for something slightly harder, we add each successful result to the next number after it.

for i in {1..100}; do s=$((s+i)); done; echo $(date +"%D %I:%M:%S") - Total - $s

How about if we echo every resultant number before the total?

for i in {1..100}; do s=$((s+i)); echo $(date +"%D %I:%M:%S") - $s; done; echo $(date +"%D %I:%M:%S") - Total - $s