Useful Linux Commands

command line formatting (for reverse shells):

python -c 'import pty; pty.spawn("/bin/bash")'
# background process
stty raw -echo
# bring process back into foreground
fg

export TERM=xterm-256color
stty rows 45 cols 205
export PS1="\[\e[01;33m\]\u@\h\[\e[00m\]:\[\e[01;35m\]\w\[\e[00m\]\$ "

# with window title
export PS1="\[\e]0;\u@\h: \w\a\]\[\e[01;33m\]\u@\h\[\e[00m\]:\[\e[01;35m\]\w\[\e[00m\]\$ "

Get command to set LS colors:

/usr/bin/dircolors -b

Print man pages to the terminal

man -P cat ssh

Hexdump

# only the hex portion
hexdump -ve '/1 "%02x "' encrypted.bin | awk '{$1=$1};1'

# only hex, with frequency of each byte
hexdump -ve '/1 "%02x " "\n"' encrypted.bin | sort | uniq -c | sort -nr

Remove Null Bytes from file:

tr < file-with-nulls -d '\000' > file-without-nulls

Sort files in directory by line count

find /path/to/directory -type f -exec wc -l {} + | sort -rn

Loop through lines of a file

while read p; do
  echo $p
done <peptides.txt

recursively delete empty directories

# the GNU version of find supports the -empty test
# print all empty directories below your current directory
find . -type d -empty -print

# deletes the empty directories
find . -type d -empty -delete

Displaying permissions of file (as octal)

https://askubuntu.com/questions/152001

stat -c "%a %n" *

Compression / De-Compression

commands	file extensions	example
zip, unzip
gzip, gunzip, gzcat, zcat	.gz
bzip2, bunzip2, bzcat	.bz2, .bz, .tbz2 or .tbz
tar	.tar	tar -xzvf myfile.tar.gz

User & Group management

adduser username grouptoadd #add user to group
gpasswd -d user group       #remove user from group
useradd     #create user
userdel     #delete user
groupadd    #create group
groupdel    #delete Group

file stats (stat Command)

stat <filename>

Check if Linux is 32 or 64 bit:

arch # x86_64 = 64 bit and i686, i386, etc. = 32 bit (best way to determine the architecture is to run the arch command and google the output)

backgrounded jobs

# show backgrounded jobs with pid
jobs -l

# kill 1st backgrounded
kill %1

String manipulation

# reverse a string
echo "Hello there"|rev

grep for lines of a specific length

https://unix.stackexchange.com/questions/184519/how-to-grep-for-line-length-in-a-given-range

grep -x '.\{3,10\}'
where

-x match pattern to whole line
. any symbol
{3,10} quantify from 3 to 10 times previous symbol (in the case any ones)

exiftool - Read and write meta information in files

exiftool <file>

what is running on local ports (linux):

https://www.cyberciti.biz/faq/what-process-has-open-linux-port/

netstat -tulpn

ss -tulpn

remove line breaks from file

tr -d '\n' < yourfile.txt

symbolic links

ln -s file1 link1

Cron syntax

# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of the month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday;
# │ │ │ │ │                                   7 is also Sunday on some systems)
# │ │ │ │ │
# │ │ │ │ │
# * * * * * command to execute

Curl through proxy (for burp):

curl --proxy http://localhost:8080 <URL>

-x, --proxy <[protocol://][user:password@]proxyhost[:port]>

Remove password from ssh key https://stackoverflow.com/questions/112396

ssh-keygen -p [-P old_passphrase] [-N new_passphrase] [-f keyfile]

rlwrap (readline wrapper)

rlwrap runs the specified command, intercepting user input in order to provide readline's line editing, persistent history and completion.

https://stackoverflow.com/questions/28878995/check-if-a-field-is-an-integer-in-awk

cat full_nmap.nmap | awk -F"/" '$1 ~ /^[0-9]+$/ {print $1}'
cat full_nmap.nmap | awk -F"/" '$1 ~ /^[0-9]+$/ {print $1}'|tr '\n' ','
cat full_nmap.nmap | awk -F"/" '$1 ~ /^[0-9]+$/ {print $1}'|tr '\n' ','| rev | cut -c 2- | rev

get full path (useful in conjunction with find function):

root@kali:~/Documents/github# find . -name "*.pdf" -exec readlink -f {} \;
/root/Documents/github/CTF_notes/2019-KringleCon/Documents/LetterToElfUPersonnel.pdf

find path_to_files/* -type f -print0 |xargs -0 file -i|grep application/octet-stream|awk -F ":" '{ print $1 }'|xargs rm

Open file from terminal:

# generic
xdg-open filename.xxxx

# until Xenial (16.04):
gvfs-open file2open.xxx

# starting with Artful (17.10):
gio open file2open.xxx

"Colorize" output

# install pygments python package
pip install Pygments

alias ccat='pygmentize -O bg=dark,style=colorful'

Compare contents of two files

comm [-1] [-2] [-3] file1 file2
-1 Suppress the output column of lines unique to file1
-2 Suppress the output column of lines unique to file2
-3 Suppress the output column of lines duplicated in file1 and file2

Date formatting

date --date='TZ="America/New_York" 9:00'
date --date='TZ="America/New_York" 9:00 PM'
date +%m/%d/%Y\ %H:%M

Find symlinks:

find . -type l -user `whoami` 2>/dev/null

Text Processing

Return 2nd column from tab delimited file:

cat filename | cut -f2 -d$'\t'

Remove leading spaces:

echo "      this is a test" | sed -e "s/^[ \t]*//"

Find all files containing specific text:

grep -rnw 'path/to/somewhere' -e 'pattern'

padding within bash

for i in $(seq -f "%05g" 10 15)
do
  echo $i
done

Print 1st column:

cut -d' ' -f1  <<< "12/12/2013 14:32"
awk '{print $1}' <<< "12/12/2013 14:32"
sed 's/ .*//' <<< "12/12/2013 14:32"
grep -o "^\S\+"  <<< "12/12/2013 14:32"
perl -lane 'print $F[0]' <<< "12/12/2013 14:32"

find ./ -type d -empty -delete find ./ -type f -size 0 -delete

Bash - get chr and ord functions:

chr() {
  [ "$1" -lt 256 ] || return 1
  printf "\\$(printf '%03o' "$1")"
}

ord() {
  LC_CTYPE=C printf '%d' "'$1"
}

Example:

chr 65

Result: A

Curl - resolve hostname: curl --resolve 'yada.com:80:127.0.0.1' http://yada.com/something https://stackoverflow.com/questions/3390549/set-curl-to-use-local-virtual-hosts

Grep for line of a specific length: https://unix.stackexchange.com/questions/184519/how-to-grep-for-line-length-in-a-given-range

grep -x '.\{3,10\}'

     -x match pattern to whole line
     . any symbol
     {3,10} quantify from 3 to 10 times previous symbol

https://stackoverflow.com/questions/11967776/swap-two-columns-awk-sed-python-perl

awk ' { t = $1; $1 = $2; $2 = t; print; } ' input_file

awk: https://stackoverflow.com/questions/13046167/printing-the-last-column-of-a-line-in-a-file Print last column of a line containing "A1"

awk '/A1/ {print $NF}' file

Copy files using ssh identity file:

#send to remote host:
scp -i KEYFILE FILENAME USER@SERVER:/home/USER/FILENAME

#pull from remote host (save locally):
scp -i KEYFILE USER@SERVER:/home/USER/FILENAME /home/USER/FILENAME

Get Path of file/directory (Linux):

readlink -f filename

realpath filename

Open images from terminal

eog filename

display filename # from ImageMagick

Parse Markdown (e.g., README.md):

pandoc -t html README.md | lynx -stdin
pandoc -f gfm -t html5 README.md

Linux file permissions

$ ls -l
drwxr-xr-x 4 cliff user 1024 Jun 18 09:40 WAITRON_EARNINGS
-rw-r--r-- 1 cliff user 767392 Jun 6 14:28 scanlib.tar.gz
^ ^ ^  ^   ^ ^     ^    ^      ^     ^     ^
| | |  |   | |     |    |      |     |     |
| | |  |   | owner group size  date  time  name
| | |  |   number of links to file or directory contents
| | |  permissions for world
| | permissions for members of group
| permissions for owner of file: r = read, w = write, x = execute -=no permission
type of file: - = normal file, d=directory, l = symbolic link, and others...

ls -a List the current directory including hidden files. Hidden files start with "."
ls -ld * List all the file and directory names in the current directory using long format. Without the "d" option, ls would list the contents of any sub-directory of the current. With the "d" option, ls just lists them like regular files.

Integer Value	Permissions	Binary
7	Full	111
6	Read and write	110
5	Read and execute	101
4	Read only	100
3	Write and execute	011
2	Write only	010
1	Execute only	001
0	None	000

Finding files

find -maxdepth 1 -type f -writable # writable files
    -maxdepth 1
    -type f
    -type d
    -writable
    -executable
    -readable
    -user <username>
    -group <groupname>
    -size <size_in_bytes>
    -perm <permission>
    -mtime [-,+]<number_days>

# exclude directory(s) in Find
find / -path /proc -prune -o
find / \( -path dir1 -o -path dir2 -o -path dir3 \) -prune -o

# find text that is human-readable, 1033 bytes in size, not executable
find . -type f -size 1033c ! -executable -exec file {} + | grep ASCII

# Find Setuid binaries
find /bin -perm -4000
find / -xdev -user root \( -perm -4000 -o -perm -2000 \) 2>/dev/null

find / -perm -u=s -type f 2>/dev/null
find / -uid 0 -perm -4000 -type f 2>/dev/null

find / -user root -perm -4000 -print 2>/dev/null
find / -user root -perm -4000 2>/dev/null -exec ls -ldb {} \;

# Find all files containing specific text:
    grep -rnw '/path/to/somewhere/' -e 'pattern'
    -r or -R is recursive,
    -n is line number, and
    -w stands for match the whole word.
    -l (lower-case L) can be added to just give the file name of matching files.
    --exclude
    --include
    --exclude-dir
    --include-dir

    examples:
        grep --include=\*.{c,h} -rnw '/path/to/somewhere/' -e "pattern"         # having .c or .h extensions
        grep --exclude=*.o -rnw '/path/to/somewhere/' -e "pattern"              #exclude files ending with .o extension
        grep --exclude-dir={dir1,dir2,*.dst} -rnw '/path/to/somewhere/' -e "pattern"    #exclude the dirs dir1/, dir2/ and all of them matching *.dst/

Finding files based on permissions

http://www.tutonics.com/2012/12/find-files-based-on-their-permissions.html

Finding files with capabilities set

getcap -r / 2>/dev/null

Basics of the vi editor

Opening a file
  vi filename

Creating text
  Edit modes: These keys enter editing modes and type in the text
  of your document.
  i Insert before current cursor position
  I Insert at beginning of current line
  a Insert (append) after current cursor position
  A Append to end of line
  r Replace 1 character
  R Replace mode
  <ESC> Terminate insertion or overwrite mode

Deletion of text
  x Delete single character
  dd Delete current line and put in buffer
  ndd Delete n lines (n is a number) and put them in buffer
  J Attaches the next line to the end of the current line (deletes carriage return).

Oops
  u Undo last command

cut and paste
  yy Yank current line into buffer
  nyy Yank n lines into buffer
  p Put the contents of the buffer after the current line
  P Put the contents of the buffer before the current line

cursor positioning
  ^d Page down
  ^u Page up
  :n Position cursor at line n
  :$ Position cursor at end of file
  ^g Display current line number
  h,j,k,l Left,Down,Up, and Right respectivly. Your arrow keys should also work if
   if your keyboard mappings are anywhere near sane.

string substitution
  :n1,n2:s/string1/string2/[g] Substitute string2 for string1 on lines
  n1 to n2. If g is included (meaning global),
  all instances of string1 on each line
  are substituted. If g is not included,
  only the first instance per matching line is
  substituted.
  ^ matches start of line
  . matches any single character
  $ matches end of line
  These and other "special characters" (like the forward slash) can be "escaped" with \
  i.e to match the string "/usr/STRIM100/SOFT" say "\/usr\/STRIM100\/SOFT"
  Examples:
  :1,$:s/dog/cat/g Substitute 'cat' for 'dog', every instance
   for the entire file - lines 1 to $ (end of file)
  :23,25:/frog/bird/ Substitute 'bird' for 'frog' on lines
   23 through 25. Only the first instance
   on each line is substituted.

These commands are all prefixed by pressing colon (:) and then entered in the lower
left corner of the window. They are called "ex" commands because they are commands
of the ex text editor - the precursor line editor to the screen editor
vi. You cannot enter an "ex" command when you are in an edit mode (typing text onto the screen)
Press <ESC> to exit from an editing mode.
:w Write the current file.
:w new.file Write the file to the name 'new.file'.
:w! existing.file Overwrite an existing file with the file currently being edited.
:wq Write the file and quit.
:q Quit.
:q! Quit with no changes.
:e filename Open the file 'filename' for editing.
:set number Turns on line numbering
:set nonumber Turns off line numbering

Miscellaneous apt and deb stuff

see installed packages

apt list --installed

for older versions
dpkg --get-selections
# or
dpkg -l

https://versprite.com/blog/apt-mitm-package-injection/ https://blog.packagecloud.io/eng/2017/03/23/create-debian-repository-reprepro/