In this lesson you learn how to use the cut and OC commands the cut command is used for cutting out
sections from each line of input it receives and displaying those sections to standard output.
You can use cut to extract pieces of a line by byte position character position or by a delimiter.
This makes cut ideal for extracting columns from a c v file for example.
It is not a shell built in it's a standalone utility.
So we can use the man command to get some information on this command line utility you can perform cuts
by bytes by using the dash B option by characters with a dash option and by fields by using the dash
f option for each one of these options you'll need to supply a range.
Ranges are pretty simple and I'll demonstrate them with a couple of examples.
Also you probably use the dash d option to specify a delimiter when using the dash f option unless you
are working with tab delimited data OK I'm just going to use the text that already exists in the Etsy
password file and then use the cut utility to cut it up or slice it up in a different way.
So let's just look at the contents of that file now.
Let's start out by cutting the password file by character to print the first character of each line
we'll use dash Dasch see one in one.
Here is the range that we're specifying.
So Will do cut.
I see one ETSI password.
And if you notice here at the bottom we have a vagrant user.
And of the box and user at the bottom of our screen.
So when we execute this command the last two lines should begin with the letter V for example.
Sure enough it's v and the last two lines just like we anticipated.
If you supply a single number then only that number is displayed.
So to display the seventh character of each line use dashes see 7.
OK that's not too useful.
This next one is too useful either.
But hang in there with me.
We'll do some actual work in just a minute.
You can specify a starting position and an ending position by connecting them with a hyphen.
So to cut out characters four through seven we could use Dessie for Dash 7.
It's important when you're specifying a range that you do not use spaces.
There is no space between for the dash and the seven in our command.
Just keep that in mind.
OK let's say you want to display every character on a line starting with character for to do that.
Use a range of four dash.
This is useful if you don't know how long each line is or if the lines are of varying lengths.
So cut Desch see for dash.
See password.
You can do the opposite which is to display every character up to and including a position so to display
the first four characters you can use dashi dash for
by the way that range is exactly the same as 1 dash 4.
And we get the same output.
So here is one last range.
You can pick out multiple individual characters by separating them with a comma or example to print
the first third and fifth characters use one comma three come a five.
It's important to point out here that cut won't rearrange the order even if you specify a different
order in the range.
For example this command that I'm about to type generates the exact same output Dessie five comma three
from a 1.
There are a couple of different methods to rearranging the data but we'll get to that later.
And one more thing if you supply a range that doesn't match anything then you'll get a blank line.
So let's try to prevent the nine hundred and ninety ninth character in the password file.
There isn't a 999 character in any of the lines so there is nothing to display for the line and you
end up with a blank line being displayed and said OK.
So that really covers how to use ranges.
You'll notice that I've been using the dash see option to cut on characters but you can use the dash
B option to cut by byte to display the first bite of every line in the ANSI password file.
Well we would use Dasch be one
that's the same in this particular case as dash C-1
however a byte is not always the same as a character because there are some characters that are made
up of multiple bytes and are thusly called multibyte characters.
For example UTF 8 characters are multibyte characters.
Let me display a multibyte character to the screen with Echo.
To display the first character in that string used dash see one that will just pipe the output of echo
as the standard input to the command and do Dessie one.
So what we did there with a pipe is pretty standard and you've been doing it a lot throughout this course.
I just wanted to be explicit and say that you don't have to supply a file for cut to operate Onsen data.
You can also use standard input as well.
OK so Ben 2 the difference between a byte and a character.
So dash C-One prints the NDA and I hope I pronounced that correctly apologies to anyone who speaks Spanish.
But look what happens when we only print the first by.
Would that be one.
The first bite is displayed of the multi-byte Indian character.
In most cases this is not what you want.
Just something to keep in mind.
Let's move on to the dash f option.
It allows you to cut lines by field by default dash.
Splits on a tab anything before a tab is considered to be the first field.
Anything after the first tab and before the second tab is considered to be the second field and so on.
Cut uses the term field but you can think of these as columns if you wish.
Let me generate some tab delimited data with the echo command the dash option to echo allows you to
use some backslash escapes that allow you to do some things like generate a tab character a new line
and so on backslash t represents a tab so we can do this.
Echo dash each one and then Ford slashed he will produce a tab.
Will use the word to forge slash t is another tab and will use the word 3.
So if we want to display just the first field we can do this.
But dash F 1 and to display the second fill cut have to third field is of course cut dash 3.
So what happens if you have data that is not tab separated.
Let's say we're dealing with a C S A V or a comma separated value file.
In this case you need to tell the code command what to use as the delimiter.
Here it's a comma
at the first field the second field and the third field.
Sometimes you'll see people do what I'm about to do here which is to not use quotes around the delimiter
when specifying that delimiter so you might see something like this.
Cut Dashti comma dash F to for example.
You may also see people not put a space after the Dashti and do this.
Either one of those methods work as long as it's a character that's not used or interpreted by the shell.
If you're trying to do this it won't work so we'll do this.
Here you have to quote the forward slash.
Otherwise the shell interprets it as a line continuation character.
It's just one of those little got used that can happen and that's why I suggest you always quote your
delimiter.
The password file is actually made up of a series of columns or fields all separated by a colon.
So let's print the user name and your ID of every user in the password file so will specify the delimiter
as a colon and say give us fields 1 and 3 from the ETSI password file.
Notice that the output is limited by the original delimiter to change that use the dash dash output
dash delimiter option so let's change it to something else here
here's a common situation that you'll face you'll have a C S V file with a header or some other type
of data that contains a header.
Let me create a C S V file on the fly here.
When you do this
you get the header and the output.
So you have two choices the first one is to remove the header before you send the data to cut or remove
it after cut has done its work.
Before we do that let's review the grip command quickly by default.
Grep displays matches to a pattern that you supply.
So if we look for the pattern of first it will display the line or lines that match that pattern.
So we do grab first people.
And here we have three matches and notice that it doesn't display any of the lines that do not match.
Let's narrow down our search such that it only matches the header.
You can do that by supplying more information or a more exact pattern.
If you want to be exact you can use regular expression anchors speaking of regular expressions when
I'm about to show you are my two most commonly used regular expressions ever.
If you never learn any more about regular expressions you'll have at least these two very important
ones at your disposal.
The first regular expression is a carrot symbol.
It matches the beginning of a line.
It matches a position and not a character.
So if we want to match all the lines that start with first use carrot first like so
notice that the results returned are different when you do not use the carrot character.
The second regular expression is the dollar sign.
It matches the end of a line.
Two matches a position and not a character.
So if you want to find all the lines that end in t we can do this grep t dollar sign.
Sort of force an exact match you can start your pattern with a carrot and end it with a dollar sign.
Now we have isolated the header of the file but we want everything except that luckily grep has a handy
option that inverts matching.
That option is Dasch v
the dash via option makes grab display any lines that do not match the pattern supplied.
Now that we've removed the header we can send it to cut.
Another option is to perform the cut first then remove the header.
I don't like this as much because Cut alters the output first making the header change too.
But it does work.
So we can do this.
That's what we get.
That contains the header
and that removes the header by the way cut only handles single character delimiters.
This is fine in most cases but there might be occasions where you would want or need to split on multiple
characters.
Take this example.
At first glance you might think oh I can just split this on the colon.
Let's try that and see what happens.
That leaves the string data which really should be considered part of the delimiter.
It's not part of the actual data itself.
It's a pointer to the real data.
What you would really like to do is this.
But as you can see that doesn't work.
We can do that with all.
However now I'm not saying that this is the only way to handle this situation but it is one way plus
it gives me a chance to briefly cover AWC every good shellscript or should at least be aware of.
Let me just give you the answer first and then I'll explain it to you in just a second.
This is an entire program on a single line the dash capital f option allows you to specify a field separator.
We're telling it to use data Colan as the field separator.
The entire program is contained in the next set of single quotes the braces in awk mean an action.
This makes awk do things or take actions the action we want to take is to print.
As you probably have figured out by now.
Dollar 2 represents the contents of the second field.
So a dollar sign one is the data in the first field assigned to the data in the second field and so
on and so forth.
Let's go back to a previous cut example.
So here we are displaying the first and third fields from the ETSI password file.
To do something very similar and all you can do this.
Here you see that separates dollar sign one and dollar sign three with a space.
That's because the comma in the print statement represents the output field separator.
By default the output field separator is a space in awk if you leave out that comma.
Then the fields just run together like this.
Let's go back to our print statement here I'll just execute this.
Awk has a special built in variable named o f s That's capital O F S and that stands for output fields
separator.
You can change the default from a space to anything you would like.
By changing the value of that variable to change a variable in awk use the dash v option and then perform
the variable assignment.
So to change off as to a comma we can do this.
So I'll go here add a dash.
The option set off as equal to a comma and hit enter.
To be clear it's definitely the office variable that controls the output delimiter being displayed and
not the space used in the print statement.
We get the same result even if we do this.
Let me just add a bunch of spaces like in between here and some here.
Well when I hit enter.
You're going to see that the data is exactly the same instead of setting the off as a variable.
You can just give it a string to print like so
the string are going to print as a comma and then will specify field 3 here.
If you want a space after the comma for example just add that space in your strings.
So let me go back up here and just put a space in my string and hit enter.
All X really lenient with spacing so this is the exact same command.
I can run dollar sign one right up against the streaming and dollar sign 3 there.
Or I could put a lot of space in between these like this and you'll see that that doesn't really affect
the execution of awk.
That may be clearer to you or it may not.
Now let's add some more text to our print statement here.
Let's do this let's say we want a print column one
so hopefully that gives you an idea of how you can use strings with the print statements and how the
output field separator works.
Joe if you remember earlier I said you can't control the order of the data being displayed with cut.
So let's take this example.
It displays the fields in the order that they appear from the input with Orch.
You can change it like so.
We just tell to print the third field first and then the first field.
Then you can combine it with any other additional strings you want so maybe we want to say this is the
UAD
and we'll separate that with a semi-colon log in like.
So in addition to dollar sign one dollar sign to sign 3 and so on or gives us dollar sign and F which
represents the number of fields found.
So to print the last field for every line in a file use dollar sign in f
the password file is very uniform so using dollar sign N.F. isn't exactly groundbreaking here.
But if you are dealing with the regular data that doesn't fit nicely into columns you can often see
something common in each line and then use dollar sign N.F. to shorten the data down or to normalize
that data even if the number of fields is consistent in the data.
It might be easier to say print the last filled or print or NF So you don't have to count the number
of fields first.
If you have a CXXVI file with 47 of her fields or columns and need the last one it's a lot quicker to
use dollar sign NF instead of counting all those columns you can do some math with awk to just surround
it in parentheses So check this out.
What this command does is Prince N.F. minus one which is of course the second to the last field.
So if there are seven fields in F seven minus one is six then it prints six field for example.
Let's generate some irregular data.
You can see that what we really have here is a file with four lines in it and each line is made up of
two columns separated by varying lengths of white space white space being spaces and or tabs.
It would be really hard to make sense of this data was cut because it only allows us to split on a single
character even if we split on a space we wouldn't end up with what we wanted because different lines
have different number of spaces separating the columns.
Also it wouldn't handle lines with tabs.
However awk performs really well in this situation.
By default the field separator for all is whitespace.
Or if you say it another may be even more accurate way.
Awk considers non whitespace characters to be a field by default.
Awk easily handles extraneous spaces at the beginning and end of each line for example.
So really those are the two times that I personally use awk.
One is to use a delimiter that's comprised of more than a single character and the other time is to
handle fields separated by whitespace.