Integrity

Today’s post is about one leg of the CIA Triad, Integrity, which means protecting data from unauthorized modification. There are a number of things needed to ensure data integrity, but let’s focus on detection. More specifically, let’s focus on detecting changes to files.

Detecting changes

The easiest way to detect if a file has been changed is to hash it and compare that to a previous hash of the file. Hashing algorithms will detect small changes in files to produce very different outputs. For example, this is the SHA256 hash of Allen Ginsberg’s Howl:

902761fbc2b389b0142d70efddb5ed0b31c4fd7cafc0497a199b98fdc9d0966b

and this is the SHA256 hash of Allen Ginsberg’s Howl with an extra space at the end:

05480bfeb192bffd7a702654173e27f1789bdb0a716e93e0f6855de76c27f906

So, having a hash of a file will allow you to quickly determine if it’s been modified. For this reason, hashing is frequently used in File Integrity Monitoring solutions.

Of course, your choice in hashing algorithm matters. A number of security issues have been discovered with the hashing algorithm MD5; a quick search can give you more details. TL;DR don’t use MD5 for integrity checking, use SHA256.

Use Case

You can use it to ensure the integrity of import documents and files. For instance, when submitting an Incident Response Report, you should always add an additional document providing hashes of every file included with the report. This way, it will be easy to detect if the report or supporting documents have been modified.

Here’s a little script I wrote to quickly hash all files in a directory and create a csv

Script

#!/bin/bash

# directory from command line to recursively hash
if [ -z "$1" ]; then
  echo "supply directory with files to hash"
  exit
# input must end in / so the awk command later will function properly.
elif [[ $1 = *"/" ]]; then 
  DIR="$1"
else
  DIR="$1/"
fi

# get basename of directory to name output file
SAVETO=$(basename $DIR)

# write CSV header
echo "File, SHA256" > hashed_$SAVETO.csv

# find all files in directory and:
#   hash it if the file is NOT DS_Store
#   print filename and its hash to file
find $DIR -type f -print0 | xargs -0 openssl sha256 | awk -F \/\/ '$2 !~ /DS_Store/ {print $2}' | awk -F \= '{match($1,"\\)")}{OFS=","}{print substr($1, 0, RSTART-1), $2}' >> hashed_$SAVETO.csv

echo "hashes saved to hashed_$SAVETO.csv"

Just include the directory of files as a command line argument and you’re good to go!

Keeping Track of How Tos

Working at a start-up, I get to wear many hats and gain experience in a wide-range of topics.

But that also means picking up a project, learning skills needed to complete it, then moving on to the next project which may not require any of the knowledge I just acquired. For example, as part of an Incident Response project last year, I needed to determine how easily a password hash could be cracked. So, I learned how to use John the Ripper and about hash cracking in general. After concluding that project, I moved onto the next thing. It wasn’t until the next incident a year later that I needed to use John the Ripper again.

Then there are those random questions you never want to waste time figuring out again. Like, “How do I remove newline characters from within quoted text in this csv so that awk sees the quoted text as one field?”

This quick succession and breadth of topics meant that I’d often have to recall something from a few months ago.

So, how do I prevent having to relearn something I knew last week/month/year?

How Tos! lots and lots of How Tos. So many How Tos that managing them became an issue. But that’s a good problem to have, and one I solved with a simple little bash script.

HOWTO.SH

With this script, you can list, search by topic, open, and create new HowTos.

#!/bin/bash

DIRECTORY="$HOME/Documents/HowTo/"

# default (no parameter)
if [ -z "$1" ]; then
  echo 'Key How_To'
# grep excludes README and .DS_Store, all HowTos start with a lowercase tag for the topic
  find $DIRECTORY -type f -maxdepth 1 | awk -F \/\/ '{print $2}' | grep '^[a-z]' | awk '{OFS="\t"}{print $2, $3}' | sort -n
  echo 'Use "help" for more options'
fi

case $1 in
  "list") echo 'Key HowTo' && find $DIRECTORY -type f -maxdepth 1 | awk -F \/\/ '{print $2}' | grep '[0-9]' | awk '{OFS="\t"}{print $2, $3}';; # grep excludes README and .DS_Store, all HowTos start with a lowercase tag for the topic
  "new")
    read -p "What type of HowTo is it? Enter tag:
TAG TOPIC
b Bash Commands. Note: Many have script written, see ~/BashScripts directory
c Code. Things related to code (pulling, commiting, reviewing, etc)
s Servers. (setup, connect, verify, etc.)
t Text. edit, search, etc
m Miscellaneous
" TYPE
    read -p "What would you like to call it?: " ANSWER
    NUMBER=$(($(find $DIRECTORY -type f -maxdepth 1 | awk -F \/\/ '{print $2}' | grep '^[a-z]' | awk '{OFS="\t"}{print $2}' | sort -rn | head -1)+1))
    if [ ! -d "$DIRECTORY" ]; then mkdir $DIRECTORY; fi
    vim $DIRECTORY$TYPE\ $NUMBER\ $ANSWER
    ;;
  "open") open ~/Documents/HowTo && exit;;
  [[:alpha:]]) echo 'Key HowTo' && find $DIRECTORY -type f -maxdepth 1 | awk -F \/\/ '{print $2}' | awk -v pattern="$1" '$1 ~ pattern {print $2, $3}' | awk '{OFS="\t"}{print $1,$2}';; # second awk had trouble with OFS so pipe it into a third awk to format output correctly.
  [[:digit:]] | [[:digit:]][[:digit:]]*) echo 'must enter a two digit key (01 instead of 1 or 001)';;
  [[[:digit:]][[:digit:]]) find ~/Documents/HowTo -type f -maxdepth 1 -name '[a-z]*'$1' *' -exec vim {} \;;;
  "help") echo "Usage: How_To.sh [option]

OPTIONS:
help Show this page
list List all HowTos (default behavior)
new Create a new HowTo
<topic tag> Search HowTos by topic (see TAG below)
<numeric key> Read HowTo associated with key
open Open the HowTo directory

TAG TOPIC
b Bash Commands. Note: Many have script written, see ~/BashScripts directory
c Code. Things related to code (pulling, commiting, reviewing, etc)
s Servers. (setup, connect, verify, etc.)
t Text. edit, search, etc
m Miscellaneous";;
  *) echo "unknown parameter, use help for options. Search by Topic is limited to one tag. Key to read HowTo must be numeric";;
esac

Using the help option will explain what the Topic Tags mean. Then you can make a new HowTo using the new option. Once you’ve written a couple HowTos you’ll be able to find them easily by searching with the Topic Tag, though sometimes I’ll just list them all out and pipe through grep.

It’s made my work much easier having a quick way to parse through my notes and HowTos. Hope it helps you.