All right! Welcome to this exercise walkthrough. You know the drill: What 2
we're going to do is look at our requirements, and then we're going to go 3
ahead and build a script to satisfy those requirements. So the requirements 4
for this particular scripting exercise are that the script is named run- 5
everywhere.sh, and what it does is, it executes all arguments as a single 6
command on every server listed in the file, and by default that file is /vagrant/servers. This script also executes 7
the provided command as the user executing the script. So, we don't want people to 8
execute this as sudo, we want them to execute it as themselves, and that 9
way they can have their own SSH key set up and that sort of thing and do it 10
that way, and if they want to use root privileges, we'll provide them a way to 11
do that that we'll talk about in just a second here. So, another thing we want to 12
do is use an SSH command that actually times out in a relatively short period 13
of time. We may have not really talked about this in one of our demonstrations 14
or one of our exercises before this, but what can happen is, if you have a server 15
that's down on the network and you try to SSH to it, SSH can take a minute or 16
two before it times out, and of course any of these network commands such as 17
ping or SSH or other similar commands often have a way to control the timeout. 18
So here, the way to do it with SSH is to use the "-o ConnectTimeout=" and 19
then some value. We'll use that option, and so I've just decided to use two seconds. 20
You could leave this out and let it fail on its own, or you could use a longer 21
duration if you want longer than two, but I think this is good for our particular 22
purposes. We also decided that this script is going to allow the user to 23
specify some options. The first option that we're concerned with this -f and 24
then a path to a file, and what this does is allows the user to override the 25
default file of /vagrant/servers. This way they can create 26
their own list, or even sub list, of servers, and execute commands against that 27
list that they provide to to the script so it's not hard coded, if you will. 28
The next option is -n, which stands for "dry run", and this allows for commands to 29
be displayed instead of executed. And another option we're going to give the 30
user the ability to use is -s, which runs the command with sudo or superuser 31
privileges on the remote server. For example, if they need to run a system 32
administration command that requires root privileges, they'll just use -s, 33
they will be connected via SSH via their normal user, and then sudo will be 34
prepended to their command, which allows it to be executed as root on the remote 35
system. And of course, with sudo it has to be configured, number one, and then, 36
number two, there is a log file that tracks who is running what commands with 37
sudo. So that is a very good feature indeed. Okay, so the last option we're going 38
to allow for is -v, which enables a verbose mode, and that just simply 39
displays the name of the server for which the command is being executed on. 40
So, as you know, some commands generate output, and if you want to see what 41
output is coming from what server, then use -v, and then it will show you. So 42
that's the whole idea behind that. Again, we mention this, that we're going to 43
enforce that the script be executed without super user privileges, and if 44
they do execute it with super user privileges, let's just tell them to use 45
the -s option, maybe give them a usage statement, and then exit the script. 46
Speaking of the usage statement, we're going to create one. Much like you would 47
see in a man page for a normal command on a Linux system, we're going to create 48
our own little mini man page. if you will. for this script. If, for any reason, a 49
command was not able to be executed successfully on a remote host, we want to 50
let the user know about that. And also, we want to exit with a zero exit status if 51
everything went okay, or, if everything didn't go okay, we'll just exit with the 52
most recent nonzero exit status from the SSH command. And that could mean that 53
we didn't connect to a host, or that could mean that the command itself 54
failed. The first step for this project is to actually create a small internal 55
network, if you will, of three virtual machines. We're going to 56
create an administration server, and then two other servers that are going to act 57
as the workhorses, if you will, for our little network here. So, obviously the 58
first thing you do is open up a command line session on your local machine. We'll 59
go into the shell class folder, and we're going to create a "multinet" vagrant 60
project. So that's what we're going to name it - nothing special about the name, 61
it's just something I chose - and then what we're going to do is initialize our 62
vagrant project, here, and we're going to use the class image. Now, we need to edit 63
our vagrant file to specify the three machines that we want to have created by 64
vagrant. So I'm just going to come down here below the config,vm,box line, and 65
add my configuration. So we'll create the admin01 server first, here. 66
I'll set the hostname to admin01, 67
give it an IP address of 10.9.8.10. 68
and then I'm just going to copy this configuration to the next one here. 69
And so anywhere I see admin01, I'm just going to change that to server01, and 70
then the IP address is going to be .11 - very similar process here - 71
or server02, so anywhere I see 01 just change it to 02 - well, almost anywhere 72
I see 01. Okay, so let me double check my work here. I have admin01 with an IP 73
address of 10.9.8.10, server01 10.9.8.11, and server02 with 74
10.9.8.12. Okay, this looks good. So now what we need to do is run 75
"vagrant up" and it will create all three of these systems, and then once it's up 76
we'll connect to the admin01 server and start writing our script. 77
Okay, this looks good. Let me run "vagrant status" to confirm. 78
Sure enough, admin01, server01, and server02 are all in the 79
"running" state according to vagrant. And now what we're going to do is connect to 80
the admin01 system and do our work from there. So I'll run "vagrant ssh admin01". 81
Once here, I'm going to move into the shared folder of /vagrant, 82
and now i'm going to start editing our script. So we're going to name it run-everywhere.sh. 83
By default, we're going to have a list of servers located at 84
/vagrant/servers, and I'm just going to put that variable at the 85
very top of the script here. I'm going to call it SERVER_LIST. Another variable I'm 86
going to define at the top of our shell script here is going to be around the 87
SSH options that we're going to use for our timeout settings. Now, this may be 88
something, personally, that I would adjust over time depending on how my network 89
works and things like that. Maybe I'll have other options that I think that 90
I'll need to use in the future, so I'm going to put these types of things at 91
the very top of my script, here. So we'll just, say... 92
Unlike some of our other scripts, we do not want this script run with super user privileges. 93
So, what we're going to do is do a check, like this. 94
So, this says, if the UID is equal to zero, that means we know they're executing 95
this with root privileges, and then we're going to tell them not to do 96
that, and then use this -s option that we're going to allow them to use 97
instead. And -s, in my mind, stands for sudo, or superuser. We're also 98
going to give them a usage statement here, and I'm just going to build that 99
into a function because I know we're going to need this later anyway, when 100
we're doing things like parsing options. So I'm going to put our usage function here. 101
We're going to allow for the n, s, and v options. We're also 102
going to allow for a -f option, but that requires a file if they use that, and 103
then we are going to require a command. 104
By the way, anything they provide on the command line after the options is going 105
to be treated as a single command. We're just going to pass that over to ssh and 106
allow that to be executed. So we're not going to do anything super fancy here. 107
This is going to allow us to do what we need to do, and if the user wants multiple 108
commands to be run on the remote host, then they just need to enclose that in 109
quotation marks so that entire command gets sent over, and not changed by any 110
shell expansion or anything like that. So, we'll get to some of these examples in a 111
minute, but I just wanted to point that out really quick - that we're not 112
promising the user anything other than executing one command on every remote 113
system. 114
Oops, I see a typing mistake here, fix that. 115
Okay, let's continue here. So, we're going to 116
have them execute the script, then we're going to make sure that they're not 117
executing it with root privileges, and if they make it past that check, well, then 118
we can start parsing the arguments or parsing the options. 119
-f requires an argument, so we're going to put a : after that option, and the n, s, and 120
v options do not. 121
So, for the f option we're going to allow them to override the server list. 122
The -n option is just going to be dry run, and we'll 123
just set that variable to true, and then we'll check it later on in our script. 124
S is for sudo and we'll set that to "sudo" and you'll see why I do that here in just a 125
minute. And then we're going to allow for 126
VERBOSE, and we'll set that to true, and then we'll check that later on. And any 127
other option is going to be invalid, and so we're going to teach them how to use 128
the script with a usage statement here. 129
So now that anything that's left on the command line after we parse the options 130
should be a command, so let's shift everything over after the options. 131
So the whole point of this script is for a user to give us a command, and then 132
this script will execute that command on all the systems in the server list, and 133
if they don't supply an argument after options, well, then there are no commands 134
to execute, so we should really tell them how to use the script and exit. So, if 135
there are no commands to execute, give them "usage", and then exit. At this point, 136
anything that remains is the command. 137
So, now that we've checked that they're 138
executing the script with non superuser privileges, and that they actually gave 139
us a command to execute over a list of servers, now what we need to do is make 140
sure that that list of servers actually exists. And, remember, we allowed them to 141
change that with a -f option above, and that means they can give us any kind 142
of data that they want. So, we need to make sure that they gave us a path to a 143
file and that we can read that file. So we'll do a check right here. 144
The if statement reads, if "! -e" (not exist) $SERVER_LIST, then we'll do these two commands here: 145
echo "Cannot open server list", and tell them what file we tried to open. And what we're 146
going to do is send that message to standard error. And we do that, of course, 147
with a >&2 and then we exit our script with a nonzero 148
exit status. Speaking of exit statuses, you know that one of our requirements is 149
to make sure the script does not exit with zero if there was an error somewhere 150
along the way. So, one way we can accomplish this goal is to set an exit 151
status of zero at the beginning, and then check 152
for any nonzero exit statuses from the ssh command as we execute them, and then 153
if there happens to be a nonzero exit status, then overwrite our default exit 154
status for the script and then exit with that exit status. So, what we're going to 155
do here is just say to expect the best, but we're going to prepare for the worst. 156
And we do that with an exit status of zero - we'll assume zero and 157
then otherwise, we'll overwrite it. 158
Okay, so now what we need to do is loop through the server list. 159
So, what this for loop does is it takes the contents of the server list 160
file and assigns it to the variable SERVER. So, the first item in SERVER_LIST 161
will be assigned to SERVER, and we'll go through this loop, and then the 162
second line, or the second server, will be assigned to SERVER and we'll do 163
that until there are no more servers in this server list. Now, we promised that if they 164
were going to be in verbose mode, that we were going to tell them what server 165
we're about to execute a command on before we do it. So, let's just put 166
a quick check in here. 167
Simple enough. And then, what we're going to do is actually 168
build an ssh command with the ssh command itself, the options that we 169
specified at the beginning of the scripts, the server that we're going to 170
be executing the command on, and if we need to execute it with sudo privileges, 171
and then, of course, the command itself. So, the reason why I'm going to put this 172
into a variable is that we're going to need to use this variable twice. If it's 173
a dry run we're just going to print this variable, but if it's not, then we're 174
actually going to execute it. So, in theory, I could have written this out 175
twice, but in kind of the spirit of not repeating myself, I'm just going to 176
create a variable to store this command that we can use in 177
two different ways, as you'll see here in just a second. So, I'm going to store this 178
command in the SSH_COMMAND variable. 179
So, the SSH_OPTIONS variable contains 180
that "-o ConnectTimeout=" setting, and we're going 181
to connect to a server that comes from the server list, which is assigned to 182
$SERVER in the for loop, then we're going to execute the remote command with 183
sudo if they provided the -s option. So, if you remember, above, if they provided 184
-s, we set $SUDO equal to "sudo" - the word "sudo". Now, if they didn't supply the -s 185
option, then the variable SUDO is not set and it will just return an empty 186
string, so it will not run with sudo privileges, and we'll show this in a few 187
minutes here. There are a couple of different ways to handle that, that is 188
one way. You could have also put a check, you could have had SUDO set to "true" 189
and then later on down here check: Oh, is SUDO true? And then if so, then 190
accommodate it there. Or, you can do it like I've done it here. Again, as long as 191
the script works, I don't really care how you get there, as long as it makes sense 192
to you, and it's actually functional. Okay, so, obviously, the last thing left to do 193
here is run the command. If it's a dry run, we're not going to execute anything. 194
We're just going to display what we would have executed. So, let's put a check 195
in right here. 196
So, here, if it's a dry run, we just display the words "DRYRUN" and then the 197
command that we would have executed. Otherwise, we're just going to execute 198
this command. 199
We're going to store the exit status because we're going to check 200
it, here, in a minute. 201
Let's talk through this last little bit of code, here. So, we 202
execute the ssh command, we store the return code or exit value into the 203
SSH_EXIT_STATUS variable, then we check that variable. 204
If it's anything other than zero, then what we're going to do is set the exit 205
status of our program to that - that SSH_EXIT_STATUS variable. So, this 206
allows that default value of zero to be overridden if anything goes wrong along 207
the way. Then we tell the user that the execution failed, and we tell them on 208
what server it failed, and, again, we're sending that to standard error because 209
it is an error message. Here, I chose not to exit right after an error, and 210
the reason for that is, let's say you have a long list of servers - let's say 100 211
servers - and on server number three, you get an error. Well, you may want to 212
continue executing that command on all the servers that are up, or available, or 213
whatever, without immediately exiting because 214
perhaps you just want to know the ones that failed so you can look at them 215
manually later instead of having to figure out, okay, 216
the command got executed on these two servers, but not these other ones, so now 217
I have to fix server number 3, or take it out of the lists, or create a 218
new list that doesn't contain the first two that the command got ran on, and so 219
on. So that was my thinking on not exiting the script there. Of course, if 220
you want to allow the user an option, perhaps you can use a -e option 221
that says, hey, exit on error, and then you could check for that option here and 222
then exit if the user set that, for example. So, again, I love scripting, you 223
can make these scripts do anything you want. So, now that I have my script 224
written, what I'm going to do is actually go to the very top of the script and just 225
kind of scan down it and look for some common issues and common mistakes that 226
everyone makes. For example, not providing a matching quote, missing an 227
opening or closing bracket, spelling of a word, any of those types of errors, and I 228
may not find them all right away, and if not, that's okay. When I execute my script 229
I'll see an error and then I can go back and fix it because, again, I'm working 230
here on a virtual machine on a test system, and it's okay if I break things 231
here, right? I'm not doing it on a production server, and this is not going 232
to wake anyone up in the middle of the night and break anything super important. 233
So, let me just jump to the very top of the script and just start working my way 234
down and see if I find anything that looks obviously wrong to me. 235
So, here, as I'm working my way down, I'm looking to make sure I don't have spaces in between 236
the equal signs when I'm doing variable assignments. I'm making sure that if I 237
have a single quote at the beginning that it ends with a single quote and not 238
a double quote - I have done that in the past, it's a common mistake to make 239
mismatched quotes. If I'm using a bracket I'm going to look to see if have an 240
opening bracket and a closing bracket. If I'm referencing a variable inside 241
a statement, I'm going to make sure that that statement is in quotation marks 242
like this, double quotes so it gets expanded - here I have it in double quotes, 243
so that looks good. I'm just going to keep going down, here. I don't see anything 244
on that screen. 245
Okay, here is an error. I actually need this dollar sign to be 246
before the variable, and I actually need a closing quotation mark, so if I had 247
executed my script, I would have ran into this bug. And I see a very similar thing 248
right here that I did as well - I left off a closing quotation mark. 249
Same thing here. 250
Okay, so today, I was not paying very close attention to my closing quotation marks 251
so I'm really glad I had taken the extra minute here to look over my script. 252
No promises that I'm not going to have an error in a second, but if I do, we'll 253
just troubleshoot it and figure it out at that time. So, I'm just going ahead and 254
save my changes.