Bash, sed, awk

How to clean bash history

history -cw

Copy retaining subfolder structure

find -name "*.svg*" -exec cp --parents --no-dereference {} Vector/ \;

Replace pattern in file names

for i in *.txt; do if $i =~ Tivoli ; then mv "$i" "$(echo $i | sed -n s/Changethis/Tothat/p)"; fi; done

How to go up and down shell history

Up is Ctrl-R, down is Ctrl-S but there is a conflict with kde's xon/xoff. You need to disable it but only for interactive shells, so scp etc is not affected.
Add the following to .bashrc, or .bash_profile (for interactive logins)
$- == *i* && stty -ixon

Mass Rename

for i in *.avi; do mv "$i" "${i%.avi}.dv"; done

How to remove non-ascii characters from a file

tr -cd '\11\12\15\40-\176' < infile > outfile

How to convert text to lower or upper case

cat instances-stage | tr [:lower:] [:upper:]

How to lowercase text with sed

Regexp's \L in front of a parameter is the lowercaser, \U is the uppercaser
sed -ri 's/(\W|^)somestuff(\w*)/\1somestuff\L\2/g' text.file

How to remove blank lines

sed '/^\s*$/d' File.in > File.out

How to use SED regexp with a non-greedy qualifier

sed does not have it. Use an exclusion format [^ ] to terminate match early like this

echo "http://dada.do/dadad" | sed -rne 's|\/[^/]*\/|\1|gp'

or use perl that supports it

echo "http://dada.do/dadad" | perl -pe 's|\/.*?\/|\1\|'

or awk using a split

string="http://www.dada.da/baba/16/"
echo $string | awk -F"/" '{print $1,$2,$3}' OFS="/"

How to watch for file change and run command

while true; do
inotifywait -r -e MODIFY dir/file && your-command
done

How to move single files to higher directory

for just the current folder

# captial A to avoid . and ..
for i in *; do if [[ -d $i && $(ls -Al "$i" | grep -v ^total | wc -l) -eq 1 ]]; then echo $(ls "$i"/*); mv "$i"/* . ; fi; done;

for all folders recursively

this also renames the file with the name of the folder it contains. May cause some files to be really long
find -type d -exec bash -c 'if grep -v ^d | grep -v ^total | wc -l) -eq 1 ; then fn=$(ls "$1/"); if -f "$1/$fn" ; then if $fn == $(basename "$1")* ; then nn=$(dirname "$1")/$fn; else nn="$1 - $fn"; fi; echo mv "$1/$fn" "$nn" ; fi; fi; ' _ {} \;

the first if checks that the number of files (not folders, total, . or .. (ls -A)) is 1.
Then checks that the single item in the folder is a file, not another folder
Then it checks if the file's name already starts with the parent folder name. If it does it will not prepend it, otherwise it adds the name of the parent folder to the file name
Then it moves up the file.
note that in a rare case that the file has exactly the same name as the parent folder, the move will fail.

after running the line above you will need to remove empty folders and re-run the recursive thingy again, in case there is one files in a folder in a folder in a folder etc.
To clean empty folders run:
find -type d -empty -delete

How extract multiple matches from a single line

Use perl because sed does not have a non-greedy modifier. To make sure only matching lines are printed and mimic sed behavior use perl -n (the opposite of -p) and "print if" function.
Here is an example of how to match, extract and create a downloader script from an html, to get all images linked from a page:
perl -ne 'print if s/.*?href="(.*?\.jpg)".*?alt="(.*?)".*?/curl -o "\2.jpg" "\1"\n/g' < Desktop\ backgrounds\ -\ Microsoft\ Windows.html > download.sh

How to find files that don't have the group read permission set

Use the negation combined with an "or" for file permission attributes
sudo find . -type f ! -perm /077 -ls
Would find all files that dont have a single bit set in either group or other permission attribute, regardless of what is in the users's attribute.

How to use xargs with quoting for file names with spaces

Use xarg substitution like this:
file * | egrep  "\bdata$" | awk -F ":" '{print $1}' | xargs -I{} rm -v {}
The command above kills all unknown files in the folder, last part functions similarly to find's {}

How to use bash parameter substitution and other functions inside find exec

Call bash -c with a parameter:
find . -name "*.mp4" -exec bash -c 'rm -v "${0%.mp4}.avi"' "{}" \;
In this case the command removes all .avi files that have a similarly named .mp4 file

How to join lines in a file or match patterns that span multiple lines

Joining every two lines. Change the subst pattern to also do some work on the joined lines
sed 'N;s/\n//' yourFile
Delete everything between "ONE" and "TWO" if they are on one or two consecutive lines:

sed '/ONE/ {
# append the next line
   N
# look for "ONE" followed by "TWO"
   /ONE.*TWO/ {
#  delete everything between
       s/ONE.*TWO/ONE TWO/
#  print
       P
#  then delete the first line
       D
   }
}' file

Inserting a new line. Don't use "\n" - instead insert a literal new line character:

(echo a;echo x;echo y) | sed 's:x:X\
:'

More details here

How to find all files in subdirectories containing a string

find . -name "*.jsp" | xargs grep "login b"

How to get a list of files with atypical owners

sudo ls -AlR / | grep -v -E 'root|total|www-data|/|^$'

How to get directory sizes in the current folder

ls -d \* | xargs -i du -kasm {}

How to get size of folders in the current folder

ls | xargs du -sh
or
du -sh `ls`
Note - does not work for folders with spaces in the name, escape with sed 's/\s/\\ /'

How to make a command survive term or putty or xterm termination

Just running it with ampersand wont work First run it with ampo
./command &
then disown it
disown [process number]
where process number is the process number from the previous command (or you can get it by doing ps -ef | grep [command])
disown is a bash command to detach the child-process from the parent and allow it to run "freely" (actually, it changes the parent to whatever is farther up the process chain).
Another handy way of doing this is the nohup command. This will even further detach the given command to "survive" even a full logout - which is not recommended for UI programs. It's used quite a bit in the system services at startup.

How to search for relevant man pages

apropos [substring]

How to perform a mass rename of files

Use rename regex in conjunction with find. For example to strip off a pattern:
find . -name "*pattern*" -exec rename -n -v 's|^(.*/)(.*) pattern (.*)\s*$|$1$2$3|g' {} \;

How to extract data with bash

Just an example of getting data with bash commands

df -h | awk '{print $5}' | grep % | grep -v Use | sort -n | tail -1 | cut -d "%" -f1 -

This will show the topmost utilization of a mounted linux partition.

How to iterate over lines in a file with bash

while read i; do touch "$i"; done < ../list

How to change permissions on all files that have a certain other set of permissions (or type or name)

file . -perm 600 | sed -n "s/$.*$/\"\1\"/gp" | xargs chmod 600

How to pipe gunzip into tar for on the fly unpacking

If your tar supports it run
tar zxvf file.tar.gz
otherwise do
gunzip -c file.tar.gz | tar -xvf -

How to set default shell to bash in Makefiles under Ubuntu

Add the following at the top of each file:
SHELL = /bin/bash

How to set a prompt to use current folder

export PS1="`whoami`@`hostname`"'($PWD)# '

How to set display variable automagically in .profile

D1=`who am i| awk '{print substr($6,2,(length($6)-2))}'`

if [ -z "$D1" ]
then
        D1=`hostname`
fi

D2=`echo $D1 | awk '{print index($1,":")}'`

if [ $D2 -gt 0 ]
  then
        export DISPLAY=$D1
  else
        #display=`host $D1 | awk '{print $3}'`
        display=$D1
        export DISPLAY=${display}:0.0
        echo "DISPLAY Set to: ${display}:0.0"
  fi

How to reinforce directory and file access limits

find . -perm /077 -type d -exec chmod -v 700 {} \;
find . -perm /077 -type f -exec chmod -v o-rwx,g-rwx {} \;

How to cp (copy) in a command line with a progress indicator

rsync -Pacz source destination

This will copy the metadata as well. If you want it to use the new date/time/owner, just like cp does, run the following. Note though that running this twice will copy the files twice unless you add the -c switch for checksumming files.

rsync -ghop --progress source destination

How to do a mass search and replace

Following is a handy little command to get the job done. Assuming you are in the directory where you want to effect this search/replace --

perl -pi -e 's/lookFor/replaceWith/' *.fileExtension
perl -pi -e 's/lookFor/replaceWith/g' *.fileExtension
perl -pi -e 's/lookFor/replaceWith/gi' *.fileExtension

NOTES: - lookFor is the word you wish to look for. - replaceWith is the word you wish to replace your original word with - g stands for global, it will basically replace all the occurences of lookFor with replaceWith in all your files in a directory - i stands for case insensitive
To load a series of replace expressions from a text file:

perl -pi -e /path/to/replace-patterns.txt *.fileExtension

where replace-patterns.txt looks like (don't forget the semicolon!):

s/lookForPatternOne/replaceWithOne/;
s/lookForPatTwo/replaceWithTwo/g;
s/lookForPatThree/replaceWithThree/gi;