Grep and Awk Tips and Examples

Remove duplicates and retain the order
awk ' !x[$0]++' 1.txt > 2.txt


Grep regex to find strings only contains alphabets 

grep "^[a-zA-Z]*$"

alphabets, numbers and + (this is typical for a query string)
 grep "^[a-zA-Z0-9+]*$"

Grep few characters after match - this case 15 characters after match - for getting the various calls to www.xyz.com

grep -E -o "www.xyz.com \"GET \/.{0,15}" httpd2-xyz.com.access-log.1370347200

-o - to show only the match
-E - extended regular expression


Grep - escape double quote
\"

-as in above example, before GET

Various calls goes to www.xyz.com from apache logs
grep -E -o "www.xyz.com \"GET \/.{0,15}" httpd2-xyz.com.access-log.1370347200 | awk '{ print $3}' | awk '{ if(match($0,"?")) { split($0,a,"?"); print a[1]}}' 



Grep for lines ending with q= - from the logs which has no query param
grep '.*q=$'