In this lesson you'll learn how to sort data using these sort and unique commands.
Let's start out with some data we already have on the system which is in the ETSI password file.
If you want to sort the contents of a file alphabetically you can use the sort command here you can
see the line that starts with the box and is last because it is last alphabetically.
We look at the top of the output will see what's first alphabetically so let me just pipe this to Les
and we see that 20:00 is the first line because A's at the beginning of the alphabet.
And for example V is near the end of the alphabet.
Let me hit Q to get out of that view.
If you want to reverse the order of the sort.
Use the dash r option
so now the user is last instead of first because we reverse the order.
Let's see what happens when we use sort with numbers.
I'm going to pull out the ID in the password file with the cut command.
So I use a colon as a delimiter and the ID is in the third field.
Now let's use the output of the cut command as standard input to the sort command.
So here this is demonstrating that you don't have to run sort directly against files.
It can accept standard input as well so we'll do this through a pipe.
This might not be what you expected.
The list is sorted but not numerically.
We have 774 81 etc..
However when working with numbers you probably want a numeric sort so that 7 comes first then 8 then
74 then 81.
So on to do that we can use the dash in option.
Now we have the smallest numbers first and the largest numbers last of course you can reverse this order
with a dash r option.
Let's talk about the do you command quickly.
It displays it disk usage.
So let's see how much space is being used in Ford slash bar by the way they're going to be some files
in there that are not readable by a normal user so I'm going to use sudoku to give us root privileges
to look in there.
You'll notice two columns the first column is a number that represents disk usage and by default this
number is in kilobytes.
The second column is the related directory that is using that particular amount of storage.
Now let's perform a numeric sort to find out which directory and forward slash var is using the most
space.
Of course var itself is at the very bottom since var contains all the subdirectories within it.
So it uses the most base but the one before that var lib is the directory in var that is using the most
space and then above that var lib RPM Varla Yarm and so on.
If we look at the forward slash var forward slash lib directory it says it's using ninety two thousand
seven hundred and forty four kilobytes.
If you don't want to see the size in kilobytes you can use the dash h option with you which makes it
print the sizes in a human readable format.
So now we see at the bottom 93 Meg for k 91 mag 83 megabytes and so on.
If we try to sort this human readable data it's not really going to work how we would like it to work
in this system and state that now.
So at the bottom three lines for example you have 93 megabytes than 96 kilobytes and the 962 chiel bytes
so it's not in the proper human readable order.
And we could even try this with the dash and option.
And again we have 700 K 800 Kane on her 72 K which are smaller amounts than those megabyte numbers that
we were just talking about.
Now the good news is that sored has a dash HD option that performs a human numeric sort.
It understands that a number that ends in a is a gigabyte number that ends in a capital M is a megabyte
and so on.
So now we have 93 am at the very bottom then 91 then 83 then 5 megabyte and so on.
And in the case the smaller sizes are at the top of this sorted list so this sort Dasch h works with
human readable numbers.
In a previous lesson we use the net step command to display open ports and let's just walk through that
again.
So we have net step with the dash and to display numbers instead of port names you for a UDP for TCAP
and L for listening port so we enter that and then we can see we have some data here and the data that
we're most interested in is in the fourth column.
And also we need to remove the headers and so one way we talked about doing that was to look for something
common and all the lines and hear a common thing is a colon.
So let me just script or a colon.
And now we have that separated out the headers out of her way again like I said the data that we're
after is in the fourth column.
So he can use Alk. print dollar sign for to get that for us.
Now all we can do is get the port numbers with awk because they're all in the last IFIELD fields being
separated by Holand in this instance so he can do this on Desch f separate like Holen and then print
dollar N.F..
So now we're left with a list of ports.
However they're not sorted so let's fix that and they're all numbers we can use sort ash end to sort
numerically.
Now we have a sorted list but there are duplicate Portes sort can handle this situation too with its
Dasch you option and Dasch you stands for unique and it only displays a line if it has not been previously
displayed before.
So let's use dash in dash you and hit enter.
So above we had 22 22 25 25.
But the output below when we use the dash you option is 20 to 25 and so on.
By the way we don't have to combine the dash you option with the dash in option we can use dash you
on its own.
So here we have a unique list of ports they're just not numerically sorted.
In addition to sort's dash you option.
There's a command called unique that spelled you and I Q Which does something very similar to the dash
you option with unique however the lines coming to it have to be sorted in order for it to work because
it only compares the current line to the previous line.
So let me show you let's do a sort dash in and then pipe that two unique.
So now we have a unique list of ports with no duplicate.
And just to show you that unique doesn't work with an unsorted set of lines.
Let's do that let's remove out this sort here and see what happens then.
So now we have 22 25:22 Well 22 is a repeat.
And it didn't get extracted because it's comparing 22 to 25.
But if we had sorted it it would have been 22 22 and then unique would have noticed.
Ah that's a duplicate.
So I'm only going to print the first one and not the second one.
So at first glance this might seem like an extra step like why would you ever want to use the unique
command if you have to give it sort of data anyway and sort already has it as you option.
Well when you want to know how many occurrences of each line there were use unique Dasch see.
So let's go up here are sort dash and first two unique and let's add the dash see option the first column
is the number of times the line appeared in the output followed by the line or the output itself.
So here we can see there were two instances of 22.
Two instances of 25.
One instance of 68 and so on.
Let's say you want to find out how many ysis log messages a program is generating and you can do that
by doing something like this.
So let's look at the data we're working with at var log messages and if we count the fields here 1 2
3 4 5 the fifth field contains the application name or the program name that is writing to Sisler.
So let's pull that out.
So here you can see System D command that NSU command back to System D at the bottom you see system
s log in and so on.
So let's sort this list and now let's feed this list to unique and get a count.
So here we have some counts one occurrence of LVM eight of network eighty six of network manager and
so on.
But let's say we want to sort this output So let's take the output from unique and run it back through
sort so we can do this sort dash in.
So now we know there were 342 messages generated from the Currall 310 from System D and so on.
You can apply this to all sorts of situations where you want to know how many occurrences of something
there are.
For example if you want to know what IPs are hitting your web server the most you can strip out the
IP addresses sort them feedom two unique Dessie and then you'll end up with a count of hits by unique
IP address.
While we're on the subject of counting I want to spend just a quick minute here on the wc command.
You can think of it as standing for word count but it not only counts words it can count bytes characters
and lines.
Personally I end up using the line count option most often.
So let's provide the ETSI password file as an argument to the wc command.
The first column is the number of lines in the file.
The second column is the number of words and the third column is the number of characters just to be
clear.
ABC really doesn't understand language it considers a word to be any non zero length sequence of characters
delimited by a white space.
We can make WC display a word count with Dash w
just a byte count with S C
and finally just a line count would dash w.
This tells us that there are twenty five accounts on the system because there's one account on each
line in the ETSI password file.
Let's say you wanted to know how many accounts are using the bash shell.
First you could display the lines that match the pattern Bash with the grip command.
Maybe this isn't the greatest example because we can quickly count that there are two lines but let's
say you have hundreds of accounts on a system and there's a lot more output and you can just visually
see and recognize at that moment then what you would want to do is let me count the number of lines
in the output.
So you feed the output of the command into ABC with a dash l option.
Now I know someone is going to bring this up if I don't put it in the video so I just want to be clear
and say that in this particular situation you can also use the dash see option for grip to perform a
count so we can do this grep deceiver account of how many lines contain bashing the ETSI password file.
However if you're not using a command that performs a count then you'll end up having to pipe that output
TWC to perform the count for you.
OK let's get back to sorting.
There's one last option to sort that I want to cover before we wrap things up and that option is Dasch
K which allows you to specify a sort key.
So far we've been sorting on the very first bit of data in a line.
If you have data separated into multiple fields perhaps you want to sort on a field other than the first
one.
So let's go back to our password file.
So that's just canit now would say we want to sort the NC password file based on ID and ID is in the
third column with each column being separated with a colon by default.
Sort uses white space is a field separator.
So to tell sort to use a colon we need to use the Dashti option.
Then we can use the dash key option to provide a sort key.
The simplest sort.
He is a number which represents the field to sort by so AK as the password hell sort to use a colon
as a field separator and tell it to.
On the third field.
This third field happens to be comprised of numbers.
So we're going to use a numeric sort with Nash in of course we can combine this with other options like
Dasch are for a reverse sort as well.
So as we all know the root account has a id of 0 and then you can see the account here on this particular
system with YOU DO YOU HAVE ONE IS been Damon has a new idea to add man has 93 and so on.
Let's do a little demonstration on how to analyze a web server log file using sort and unique.
Let's say you want to know how many times a particular you are always visited first let's look at what
we have to work with.
So I have a access log file here and forward slash vagrant.
My first goal is to extract the L portion from the file.
Now there are multiple ways to do this.
However what I notice is that the u r l is contained within a set of quotation marks.
So let me split on that and see where that takes us to we'll just feed this to the cut.
We'll use double quotation marks as a delimiter and we'll print the second field.
And hit enter.
By the way I didn't have to cast that into cut like I did here.
What I could do is actually supply that file to cut.
Let me just do that here now.
Kurt desde.
To access Lague and we get the same result.
Now it looks like we're left with three columns of data separated by a single space.
The second column has the U R L in it and let's pull that out so we can do this with the cut command
as well.
Again if you saw things differently or think in a different way that's perfectly fine.
Perhaps your mind went to counting the column numbers first something like this so let me just get the
access log here.
And the first column is an IP address so let's call them 1 dashes call to the dash column 3.
OK so one two three four five six seven.
It looks like on the 7th column is where you are Ellas contain and let's test that.
So let's do ahk print dollar signs 7 access log.
OK.
And so we end up with the same data.
I just want to be clear that there is no one exact perfect way to do this.
So I'd just use whatever makes sense to you and however you visualize the data just keep extracting
parts of it and transforming it until it looks like what you need.
OK.
So let's continue on.
Let me go back to my command here using the cut.
Now I want to count the number of times each one of those you or else was visited.
So I know what I can do that with a unique command and I also know that I need to provide unique with
sort of data first.
So when I have to do is pipe this to sort and then I can go back and pipe it through unique with the
dashi option to let it counted for me.
So now we have a count in column 1 and the you are all in column 2.
So now what we can do is actually sort this uniquely counted output with the sort command.
So let me bring in my command and run this through sort dash in.
So it looks like we had 1271 visits to Fort slash W.P. ad man 1265 to Fort slash Explorer and so on.
Now I would say we only want to display the top three most visited or else and we can do this simply
by displaying the last three lines of output with a tell command so will take this big long command
that we've been building up and will pipe it yet again to another command and this command has tail
dash 3 to print the last three lines.
So here you see the three most visited you or else according to that access log file.
Now I'm going to go ahead and put that command into a script so I don't have to solve this same problem
again.
So I'm actually going to copy it then I will edit my script here.
Start with the Shubane We'll tell what the script is doing.
So let the user pass in the log file.
And we want to make sure that that log file exists so we can do a quick check here.
So this says if not exist log file then we'll give them an error message.
Now I can paste my command that I copied earlier and then I'll just change this to be the log of file
variable.
Let me see my changes make my script executable and then give it a try.
So pass no data or can't open anything.
OK I can open a SDF because that doesn't exist.
So let's actually give it a path to our file that does exist.
And here we go it runs the command against what we provide it giving us the three most visited your
else in that file.
So to recap in this lesson you learn how to sort data using the sort command you learn how to use the
dash in option to sort numerically.
You also learned how to use the dash r option to reverse the sort order.
From there you learn how to display only unique lines using sort Dasch you and the unique command.
You also learned how to count items with a wc command.
Finally you use the Dashti and dash options to sort data based on a specific field.