linux shell argument list too long rsync or cp

I needed to copy files generated by doxygen from one directory into another for a large opensource C++ project. Sadly there were too many files in the directory, so bash started complaining 🙁 cp and rsync died out with the error of argument list too long. initially I figured I could generate it all from scratch in new location but it was quicker and easier to use a for loop to rsync the files over 🙂 some info: all files start with alphabetic characters. there are no spaces in the names all files are in single directory I realized bash expansion would work here. Using for loop for x in {a..b} do echo $x* done Notice I only stepped between A and B because I didn’t want to sit there for an hour while it listed all the files. this worked well, it listed all files and I was sure it would suite my purposes. now the real deal! for x in {a..z} do echo $x rsync -az /backups/doxygen/$x* /home/user/current/directory/ done sometimes you might still get the error even for each letter, for example I still had too many files starting with D and Q. so I just changed where I globbed : for x in {a..z} do echo $x rsync -az /backups/doxygen/d$x* /home/user/current/directory/ done this allows me to further iterate a thru z but after starting the files with the letter d. Now what happens if you happen to have files starting with numbers? simply switch the letters for numbers. for x in {0..9} […]

Debian package management speed ups

No one likes to sit around waiting for slow mirrors while updating multiple packages but its a fact of life usually. In debian it means typing apt-get update and sitting around for a while, then doing the actual install or upgrade and getting some coffee. what if you could speed the process along somewhat? well now just like we showed you how to speed up apt downloads for ubuntu you can speed up the apt speeds for Debian! This way you can focus more on clashing some clans or something…. whatever you do in your free time that is. Httpredir In comes httpredir, “It uses the geographic and network location of the user and the mirrors, the architecture of the requested files, IP address family, the availability and freshness of the mirrors, and a few other things” to find the closest and fastest mirror of data for you. This gives you the quickest way to get your files without resorting to new tools or alternate package managers. This is already there for Jessie ( debian 8 ) so no need to edit that, but for everyone else on older releases this will bring wonderful speedups. setting up httpredir is simple : edit your /etc/apt/soucres.list deb http://ftp.us.debian.org/debian sid main deb-src http://ftp.us.debian.org/debian sid main replace with deb http://httpredir.debian.org/debian sid main deb-src http://httpredir.debian.org/debian sid main

Sample apt.sources file with

Remove the us. part if you arent on a US server. change the word precise to whatever your version of ubuntu is.     deb mirror://mirrors.ubuntu.com/mirrors.txt precise main restricted universe multiverse deb mirror://mirrors.ubuntu.com/mirrors.txt precise-updates main restricted universe multiverse deb mirror://mirrors.ubuntu.com/mirrors.txt precise-backports main restricted universe multiverse deb mirror://mirrors.ubuntu.com/mirrors.txt precise-security main restricted universe multiverse deb http://us.archive.ubuntu.com/ubuntu precise main restricted deb-src http://us.archive.ubuntu.com/ubuntu precise main restricted deb http://us.archive.ubuntu.com/ubuntu precise-updates main restricted deb-src http://us.archive.ubuntu.com/ubuntu precise-updates main restricted deb http://us.archive.ubuntu.com/ubuntu precise universe deb-src http://us.archive.ubuntu.com/ubuntu precise universe deb http://us.archive.ubuntu.com/ubuntu precise-updates universe deb-src http://us.archive.ubuntu.com/ubuntu precise-updates universe deb http://us.archive.ubuntu.com/ubuntu precise multiverse deb-src http://us.archive.ubuntu.com/ubuntu precise multiverse deb http://us.archive.ubuntu.com/ubuntu precise-updates multiverse deb http://security.ubuntu.com/ubuntu precise-security main restricted universe multiverse deb-src http://security.ubuntu.com/ubuntu precise-security main restricted universe multiverse deb http://archive.canonical.com/ubuntu precise partner deb-src http://archive.canonical.com/ubuntu precise partner deb http://extras.ubuntu.com/ubuntu precise main deb-src http://extras.ubuntu.com/ubuntu precise main  

Find out whats taking up all the hdd space

[et_pb_section admin_label=”Section” fullwidth=”on” specialty=”off”][et_pb_fullwidth_post_title admin_label=”Fullwidth Post Title” title=”on” meta=”on” author=”off” date=”on” categories=”on” comments=”off” featured_image=”on” featured_placement=”background” parallax_effect=”off” parallax_method=”on” text_orientation=”center” text_color=”dark” text_background=”on” text_bg_color=”rgba(255,255,255,0.9)” module_bg_color=”rgba(255,255,255,0)” title_all_caps=”off” use_border_color=”off” border_color=”#ffffff” border_style=”solid”] [/et_pb_fullwidth_post_title][/et_pb_section][et_pb_section admin_label=”section”][et_pb_row admin_label=”row”][et_pb_column type=”4_4″][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” use_border_color=”off” border_color=”#ffffff” border_style=”solid”] Running out of hdd space is pretty annoying. So if you are running out of space and need to find out whats taking up all that space type the following command to find out more : du -h |grep ‘[0-9]G’   [/et_pb_text][/et_pb_column][/et_pb_row][et_pb_row admin_label=”row”][et_pb_column type=”4_4″][et_pb_text admin_label=”bottom above footer” background_layout=”light” text_orientation=”left” use_border_color=”off” border_color=”#ffffff” border_style=”solid”] [/et_pb_text][et_pb_divider admin_label=”Divider” color=”#ffffff” show_divider=”off” divider_style=”solid” divider_position=”top” hide_on_mobile=”on”] [/et_pb_divider][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” use_border_color=”off” border_color=”#ffffff” border_style=”solid”] [/et_pb_text][/et_pb_column][/et_pb_row][/et_pb_section]

Start Xvfb on boot on Centos Linux

Xvfb or X virtual framebuffer X virtual framebuffer is a display server using the X11 protocol. In contrast to other display servers, it performs all graphical operations in memory without showing any screen output. which makes it ideal for some surprising uses. e.g. Xvfb :1 & xv -display :1 & import -display :1 -window root image.png (above snippet via Wikipedia) I had to run xvfb automatically on boot on a centos system, here’s the init script I used to carry out the task. add the following as a script to /etc/init.d , chmod +x the script then chkconfig xvfb on * Note: see the :11 in the middle of the script? change that to another number to change the display port. #!/bin/bash # chkconfig: 345 95 50 # description: Starts xvfb on display 11 # why 11? dunno I just work here. 🙁 -FB # 9/30/13 r1 – initial addition -FB if [ -z “$1” ]; then echo “`basename $0` {start|stop}” exit fi case “$1″ in start) /usr/bin/Xvfb :11 & ;; stop) killall Xvfb ;; esac [/et_pb_text][/et_pb_column][/et_pb_row][et_pb_row admin_label=”row”][et_pb_column type=”4_4″][et_pb_text admin_label=”bottom above footer” background_layout=”light” text_orientation=”left” use_border_color=”off” border_color=”#ffffff” border_style=”solid”] [/et_pb_text][et_pb_divider admin_label=”Divider” color=”#ffffff” show_divider=”off” divider_style=”solid” divider_position=”top” hide_on_mobile=”on”] [/et_pb_divider][et_pb_text admin_label=”Text” background_layout=”light” text_orientation=”left” use_border_color=”off” border_color=”#ffffff” border_style=”solid”] [/et_pb_text][/et_pb_column][/et_pb_row][/et_pb_section]

Bash for loops sequential counting

  Bash for loops are very useful in everyday life. I am going to post some simple examples where I combine bash for loops with other simple techniques like bash stepping etc. Want to count from 1 – 100 and list each number sequentially? for i in {1..100}; do echo $i; done Same as above but add a timestamp? for i in {1..100}; do echo $(date +”%D %I:%M:%S”) – $i; done Again same as above but now we step every 3 numbers instead of every 1 for i in {1..3..100}; do echo $(date +”%D %I:%M:%S”) – $i; done   now for something slightly harder, we add each successful result to the next number after it. for i in {1..100}; do s=$((s+i)); done; echo $(date +”%D %I:%M:%S”) – Total – $s How about if we echo every resultant number before the total? for i in {1..100}; do s=$((s+i)); echo $(date +”%D %I:%M:%S”) – $s; done; echo $(date +”%D %I:%M:%S”) – Total – $s