1
00:00:00,960 --> 00:00:06,930
Now, let's get started with pipelines, and you're probably wondering what exactly is a pipeline anyway,

2
00:00:07,590 --> 00:00:12,570
and I wanted to make a quick analogy to a car assembly line.

3
00:00:13,320 --> 00:00:16,350
In order to create a car, you need to follow a couple of steps.

4
00:00:16,800 --> 00:00:19,080
So, for example, you first need a chassis.

5
00:00:19,770 --> 00:00:22,500
And on that, you installed the engine.

6
00:00:22,920 --> 00:00:27,750
And once you have the engine, you can add the wheels and so on.

7
00:00:28,830 --> 00:00:34,200
So as you can see, there are a few characteristics to a car assembly line.

8
00:00:34,830 --> 00:00:38,990
There are a series of steps that need to be done in a certain order.

9
00:00:39,240 --> 00:00:44,280
So you cannot start adding the wheels until you have the chassis and engine.

10
00:00:46,320 --> 00:00:48,000
The steps are connected.

11
00:00:48,520 --> 00:00:52,380
You can sort of say like the result from the previous step.

12
00:00:52,380 --> 00:00:56,280
The output from the previous step is the input for the next step.

13
00:00:56,760 --> 00:01:00,120
And some steps, of course, can be done in parallel.

14
00:01:00,120 --> 00:01:01,860
For example, installing the wheels.

15
00:01:02,220 --> 00:01:07,550
You can work on all four wheels at the same time without affecting the output.

16
00:01:09,270 --> 00:01:17,430
Now, of course, many more steps are required in order to build the final product, not only the assembly,

17
00:01:17,430 --> 00:01:24,180
the production of the car is important, but before the car can leave the factory, it needs to be tested

18
00:01:24,180 --> 00:01:29,370
to make sure that a final consumer actually gets a working car so that everything works.

19
00:01:30,330 --> 00:01:33,750
So the end goal is to get the product out of the factory.

20
00:01:35,140 --> 00:01:42,330
Now, you'll notice that building software is, in some regards, quite similar to building a car,

21
00:01:43,510 --> 00:01:50,350
and this is exactly what good lab is supporting us to get the product out of the factory, whatever

22
00:01:50,350 --> 00:01:51,370
that product is.

23
00:01:52,300 --> 00:01:56,230
Now, let's try to build something similar in Gittel Abassi.

24
00:01:57,600 --> 00:02:03,870
From the previous example, you see that we have at least two large stages, we need to build something

25
00:02:04,410 --> 00:02:06,360
and you need to test it, that it's working.

26
00:02:06,750 --> 00:02:11,490
And of course, building itself has a lot of steps but will not get into that.

27
00:02:13,080 --> 00:02:18,900
And testing, of course, can also have a lot of processes and steps in order to make sure everything

28
00:02:18,900 --> 00:02:24,890
works, but as a big picture, they are at least two big steps involved in this process.

29
00:02:26,540 --> 00:02:32,510
Now, the first step in order to get started with Gottleib is to go to get lab dot com and in case you

30
00:02:32,510 --> 00:02:38,790
don't have an account yet, go ahead and create one by clicking on the register once you are registered.

31
00:02:38,840 --> 00:02:39,860
Make sure you sign in.

32
00:02:41,820 --> 00:02:47,370
Initially will probably not have any projects, but exactly, this is what we're going to do, Statoil,

33
00:02:47,370 --> 00:02:49,340
to create our first project.

34
00:02:49,710 --> 00:02:52,710
So here on the right side, click on you project.

35
00:02:54,510 --> 00:02:56,940
When I called this project car assembly line.

36
00:03:03,950 --> 00:03:08,280
And by default, the project will be private, and that is definitely OK.

37
00:03:08,960 --> 00:03:11,810
So here below, I'm going to click on Create Project.

38
00:03:13,560 --> 00:03:20,910
Not the first project has been created, and it actually is an empty its repository where we can start

39
00:03:21,120 --> 00:03:25,400
adding our code or any else that's related to the project.

40
00:03:26,010 --> 00:03:27,750
Now, we only want to build a pipeline.

41
00:03:27,960 --> 00:03:34,200
So for that reason, in order to create a pipeline in catalepsy, we need to create a file that defines

42
00:03:34,200 --> 00:03:34,890
that pipeline.

43
00:03:35,400 --> 00:03:36,720
And that is quite easy.

44
00:03:36,720 --> 00:03:39,440
And we'll do it using the lab interface.

45
00:03:40,830 --> 00:03:43,290
So right here, you can click on your file.

46
00:03:45,320 --> 00:03:53,090
And this is to have a specific name, and that name is Dot Getler, Desh Theeye, Dot Yamal.

47
00:03:54,630 --> 00:04:01,660
And Yamal is a special format for files that define those pipelines in catalepsy.

48
00:04:02,100 --> 00:04:08,370
So you'll probably see this format generally for specifying any configurations and such.

49
00:04:08,790 --> 00:04:17,130
And in Gottleib is the only way in order to specify the pipeline, in order for Gottleib to do anything,

50
00:04:17,130 --> 00:04:19,079
we need to specify a job.

51
00:04:19,440 --> 00:04:25,120
And a job, as the name suggests, is a task that GitHub should do for us.

52
00:04:25,620 --> 00:04:29,610
So let's say the first job is built the car.

53
00:04:29,880 --> 00:04:32,610
So we're going to see here, build the car.

54
00:04:34,130 --> 00:04:34,670
Column.

55
00:04:35,810 --> 00:04:41,300
And then on a new line, I'm going to hit enter and then two spaces.

56
00:04:43,140 --> 00:04:44,400
And then the write script.

57
00:04:49,740 --> 00:04:56,490
Into more spaces Desh Now I can start writing shall commence here.

58
00:04:58,230 --> 00:05:05,390
Now, what we are trying to do is to create a file and to add a few techs to identify.

59
00:05:05,430 --> 00:05:10,620
So we are imagining that that file is our car and then we are going to add a chassis and we're going

60
00:05:10,620 --> 00:05:11,420
to add an engine.

61
00:05:11,440 --> 00:05:16,410
We're going to add wheels and put everything inside that file.

62
00:05:17,310 --> 00:05:18,900
So let's create a folder.

63
00:05:20,440 --> 00:05:22,810
Make directory build.

64
00:05:25,330 --> 00:05:28,210
We said we're going to change the directory.

65
00:05:32,140 --> 00:05:34,000
And now let's create a file.

66
00:05:37,910 --> 00:05:44,690
And create a file called Card, that text, and we're going to use the common touch in order to create

67
00:05:44,690 --> 00:05:51,890
a file, what touch does it changes the timestamp when a file was created or updated.

68
00:05:52,160 --> 00:05:54,950
And in case it doesn't exist, it will simply create a file.

69
00:05:58,330 --> 00:06:05,470
Now, what we want to do is to start adding some information to it, so let's first add to this file

70
00:06:05,470 --> 00:06:06,130
a chassy.

71
00:06:19,440 --> 00:06:25,500
And with this operator, we are going to replace the content of car texte with Chessie.

72
00:06:27,800 --> 00:06:28,850
Let's copy this line.

73
00:06:30,170 --> 00:06:31,280
And an engine.

74
00:06:36,010 --> 00:06:36,760
And Wheelz.

75
00:06:45,160 --> 00:06:46,210
And that's about it.

76
00:06:46,240 --> 00:06:54,340
These are the steps required in order to create a car, so let's save our pipeline configuration by

77
00:06:54,340 --> 00:06:56,560
committing the changes we have made to this file.

78
00:07:00,550 --> 00:07:08,080
As soon as we have saved this file, Getler, BCI has detected our pipeline and you will see here,

79
00:07:08,260 --> 00:07:14,800
this indicator is showing that the pipeline is running and we want to see the current progress on what

80
00:07:14,800 --> 00:07:15,390
it's doing.

81
00:07:15,760 --> 00:07:18,760
All we have to do is click on this image here.

82
00:07:20,860 --> 00:07:26,860
Now, this is our pipeline, and as you can see, it only contains one job because we have only defined

83
00:07:26,860 --> 00:07:27,370
one job.

84
00:07:29,310 --> 00:07:30,720
If I click inside a job.

85
00:07:32,670 --> 00:07:40,700
Good luck will execute the scripts that we have previously written and will display the output, the

86
00:07:40,700 --> 00:07:46,250
tool that is writing the scripts is called the Giddyap Runner, and it very much looks like your own

87
00:07:46,250 --> 00:07:46,820
terminal.

88
00:07:47,300 --> 00:07:49,760
And you will see here exactly the steps that we have done.

89
00:07:49,760 --> 00:07:57,430
We have created a folder build, we have created a file, and we have added this strings to the file.

90
00:07:57,920 --> 00:08:03,620
And now finally it says Jump succeeded because none of our scripts had any issues.

91
00:08:05,660 --> 00:08:09,620
Now, as you have previously said, we would like to add another step.

92
00:08:10,800 --> 00:08:18,630
So we have this step that we just called now build the car, but we also want to test everything.

93
00:08:19,740 --> 00:08:23,190
So let's go back to the project, to the killer YAML file.

94
00:08:24,260 --> 00:08:25,760
And add another step.

95
00:08:36,760 --> 00:08:39,460
So let's call the next job, test the car.

96
00:08:45,400 --> 00:08:52,150
So always remember not to use tabs, but to use spaces when defining the YAML file, and I'm using two

97
00:08:52,150 --> 00:08:54,930
spaces for indentation one too.

98
00:08:55,930 --> 00:08:57,790
And now I'm adding a new script.

99
00:09:02,550 --> 00:09:08,870
So the first step would be to test if the file was created, because we have created a file called Card

100
00:09:09,000 --> 00:09:15,240
that texte and a way to test it would be to use the test command minus F.

101
00:09:21,970 --> 00:09:26,920
And to specify the name of the file that we want to test for existence.

102
00:09:27,960 --> 00:09:35,590
Now, if this command was to fail, if it was not to work, it will return a status code and will inform

103
00:09:35,590 --> 00:09:41,170
Gottleib that something isn't right about our test gripped the entire pipeline will indicate that by

104
00:09:41,170 --> 00:09:41,690
failing.

105
00:09:42,550 --> 00:09:48,640
Let's now add some additional tests in order to check if the contents of the file is as we suspect.

106
00:09:48,910 --> 00:09:50,050
So we now have a car.

107
00:09:50,380 --> 00:09:51,340
Would we want to check?

108
00:09:51,340 --> 00:09:52,390
Do we have a chassis?

109
00:09:52,390 --> 00:09:53,380
Do we have an engine?

110
00:09:53,380 --> 00:09:54,790
And we have wheels inside.

111
00:09:56,380 --> 00:09:58,380
So let's switch inside a built folder.

112
00:10:02,900 --> 00:10:12,380
And what do we do next is to use the GRAP command, which can search for a specific string inside a

113
00:10:12,380 --> 00:10:12,750
file.

114
00:10:13,430 --> 00:10:15,380
So we're going to search for Chessie.

115
00:10:20,430 --> 00:10:23,820
Inside a car, that text file.

116
00:10:26,270 --> 00:10:28,310
The same goes for engine.

117
00:10:30,380 --> 00:10:32,210
And the same goes for Wheelz.

118
00:10:39,540 --> 00:10:47,280
Now, we haven't really specified in which order should this jobs be executed and getler by default,

119
00:10:47,280 --> 00:10:51,760
it will try to round them in parallel and this is definitely something we are not interested in.

120
00:10:52,380 --> 00:10:58,860
So in order to specify to order in which different jobs should be executed, we need to define stages.

121
00:10:59,970 --> 00:11:03,200
So the instruction to defined stages is quite intuitive.

122
00:11:03,210 --> 00:11:04,260
It's called stages.

123
00:11:08,720 --> 00:11:11,870
And we'll have two stages, the stage build.

124
00:11:13,070 --> 00:11:14,150
And the stage test.

125
00:11:16,530 --> 00:11:22,470
Now, what we need to do next is to specify which of these jobs belong to which stage.

126
00:11:25,690 --> 00:11:29,200
So when I say here before the script or even after the script, it doesn't really matter.

127
00:11:30,100 --> 00:11:31,510
This will be the build stage.

128
00:11:35,340 --> 00:11:37,100
And this will be the test stage.

129
00:11:41,030 --> 00:11:43,480
Now let's save to file and see how the pipeline works.

130
00:11:50,690 --> 00:11:57,410
One important thing to note is that Gottleib is offering information regarding this file and it says

131
00:11:57,410 --> 00:12:04,250
here, this gallaccio configuration is valid and it may happen that you make some mistakes inside your

132
00:12:04,250 --> 00:12:04,750
file.

133
00:12:04,820 --> 00:12:10,400
You write something the wrong way and Gottleib will offer you an error and you will need to fix that

134
00:12:10,400 --> 00:12:10,660
file.

135
00:12:10,670 --> 00:12:14,440
So only have valid configurations will be executed by Gottleib.

136
00:12:14,800 --> 00:12:17,490
This is something that happens before the pipeline is executed.

137
00:12:17,490 --> 00:12:20,990
So every time you create, you make changes to your pipeline.

138
00:12:21,860 --> 00:12:23,000
Gottleib will check that.

139
00:12:23,060 --> 00:12:27,250
And if it's not valid, it will not even execute a pipeline because it will not make sense.

140
00:12:28,700 --> 00:12:30,690
Now, let's click on the pipeline and see how it's working.

141
00:12:33,800 --> 00:12:38,990
Now, right from the start, you'll see that our pipeline looks a bit different than it used to look

142
00:12:38,990 --> 00:12:42,590
before, we have now two separate stages.

143
00:12:42,620 --> 00:12:46,310
We have to build stage and now we have to test stage.

144
00:12:48,330 --> 00:12:56,850
Belonging to the build stage is the build card job and belonging to the test stage, is this the card

145
00:12:56,850 --> 00:12:57,220
job?

146
00:12:58,200 --> 00:13:06,090
Of course, stages can have multiple jobs, which can run in parallel and you can add as many stages

147
00:13:06,390 --> 00:13:07,980
and as many jobs as you need.

148
00:13:11,200 --> 00:13:16,570
Now, after waiting a bit, you'll probably notice this, the test tube actually failed.

149
00:13:17,110 --> 00:13:19,560
So let's look into it and see exactly what happened.

150
00:13:22,020 --> 00:13:29,550
It seems that the first test, the test where we are checking if the file was created, fails, and

151
00:13:29,550 --> 00:13:36,870
it seems indeed that this file isn't present here, this is quite normal, because what happens is that

152
00:13:36,870 --> 00:13:40,720
the jobs that we are running, they are sort of independent from one another.

153
00:13:41,520 --> 00:13:46,530
They do not exchange any data, if not told to exchange any data.

154
00:13:46,890 --> 00:13:49,880
So the previous job, the job did something.

155
00:13:50,670 --> 00:13:54,710
But we haven't specified what should happen with the with the file that was created.

156
00:13:55,350 --> 00:14:00,420
So after the job terminated, all the files would basically be removed.

157
00:14:01,050 --> 00:14:06,300
So the environment where the job was executed was simply destroyed and it does not longer exist.

158
00:14:06,720 --> 00:14:12,810
So when the next job is starting and has no idea, it looks inside what it has inside the repository,

159
00:14:12,810 --> 00:14:18,000
more or less, because this is what it's doing, is checking out the repository and then doing the steps,

160
00:14:18,150 --> 00:14:24,810
being told to do and looking inside of the repository and looking and saying, well, I don't have a

161
00:14:24,810 --> 00:14:27,810
car, I don't have a car that takes the file.

162
00:14:29,100 --> 00:14:30,510
And as I said, that is normal.

163
00:14:30,630 --> 00:14:37,530
And we need to instruct the jobs, which is our output, which is our artifact to say like that because

164
00:14:37,530 --> 00:14:38,580
we are building something.

165
00:14:38,580 --> 00:14:40,400
And this is something that we want to keep.

166
00:14:40,410 --> 00:14:42,650
We don't want to throw it away after the process.

167
00:14:43,920 --> 00:14:47,340
So let's change again our configuration and try to get that fixed.

168
00:14:48,770 --> 00:14:52,250
So in order to tell the first job what to save.

169
00:14:53,590 --> 00:14:55,600
We need to define the artifact.

170
00:14:59,530 --> 00:15:01,900
And that is being done by using artifacts.

171
00:15:04,200 --> 00:15:10,350
And of course, the artifacts need a path, so we need to specify where exactly is the artifact located?

172
00:15:15,380 --> 00:15:20,230
And now we can say we can save a file or we can save entire folder.

173
00:15:20,760 --> 00:15:24,010
Now let's simply make this simple and we're going to save the entire folder.

174
00:15:24,830 --> 00:15:28,390
So I'm going to say here, build this is our artifact.

175
00:15:28,400 --> 00:15:32,060
This is what should be saved after this job is executed.

176
00:15:32,540 --> 00:15:34,630
And the rest of it is not interesting.

177
00:15:34,670 --> 00:15:36,050
But what inside build?

178
00:15:36,260 --> 00:15:37,580
This is what we want to keep.

179
00:15:39,770 --> 00:15:42,100
Now, let's give this another try and see if it works.

180
00:15:45,350 --> 00:15:51,710
You notice that every time we make a change in this repository, the pipeline is triggered and the pipeline

181
00:15:51,710 --> 00:15:58,130
will be stronger, not only when we edit the configuration, file the DOT, get lab tested at YAML file,

182
00:15:58,550 --> 00:16:03,680
but every time you make a change inside the repository, you -- or you edit any other file.

183
00:16:03,980 --> 00:16:09,500
The pipeline will start in order to ensure that the change you have made is still properly working.

184
00:16:12,340 --> 00:16:17,410
If you want to follow along with what a specific job is doing, you can simply click on it.

185
00:16:17,920 --> 00:16:22,430
Now, again, you will notice and you're probably thinking, what the heck is going on here?

186
00:16:22,450 --> 00:16:23,250
Why is this?

187
00:16:23,260 --> 00:16:24,340
They're still not working.

188
00:16:25,120 --> 00:16:31,510
And the only way to tell what is happening, why something is failing is to read the logs and see exactly

189
00:16:31,510 --> 00:16:33,040
what is going on.

190
00:16:34,570 --> 00:16:40,760
And you'll notice here again, the first step now appears to be working.

191
00:16:40,780 --> 00:16:48,460
So now somehow the next job has this artifact, has this folder and a card, a text file inside.

192
00:16:49,000 --> 00:16:54,860
It's getting inside that folder, but somehow is not finding the chassis that we have edited.

193
00:16:55,840 --> 00:16:58,810
Now we can go into more debugging on why this is not working.

194
00:16:59,530 --> 00:17:03,890
But the reason for this is that I have added a mistake in the code from the beginning.

195
00:17:04,150 --> 00:17:07,849
Just wanted to show you how important it is to write tests that actually fail.

196
00:17:08,650 --> 00:17:10,390
So I'm going to go back to the project.

197
00:17:12,940 --> 00:17:15,440
And get this built fixed now.

198
00:17:15,460 --> 00:17:23,980
The problem is right here, instead of spending all this, the chassis, the engine and the wheels to

199
00:17:23,980 --> 00:17:26,980
the car that have replaced the entire content.

200
00:17:27,550 --> 00:17:31,030
And the way we do appending is using this operator.

201
00:17:35,590 --> 00:17:36,980
And now it should definitely work.

202
00:17:38,600 --> 00:17:46,420
Any time you need to debugging, feel free to add commands like alerts to list the contents of the folder,

203
00:17:47,590 --> 00:17:52,420
or you can use something like that and say Khadar texte.

204
00:17:53,880 --> 00:17:59,400
And this will show you the contents, so anything that is needed in order to debug the problem, you

205
00:17:59,400 --> 00:18:03,570
can simply edit here just as you would do it when using a normal terminal.

206
00:18:05,640 --> 00:18:12,360
Every time we want to look into what has previously happened, it can go here to secede and you would

207
00:18:12,360 --> 00:18:13,770
be able to see pipelines.

208
00:18:16,710 --> 00:18:24,480
And you can see all the previous ones that we had so we had so far for pipelines, and this is where

209
00:18:24,480 --> 00:18:32,400
we started adding the second step, the testing step, and this is actually the step where we have added

210
00:18:32,400 --> 00:18:38,010
to artifact and jobs that have created artifacts are available for download.

211
00:18:38,020 --> 00:18:41,300
So you can inspect the files that have been generated that way as well.

212
00:18:42,120 --> 00:18:43,950
So you can easily check them here.

213
00:18:44,940 --> 00:18:50,580
We'll see here artefact, but you can view them when you're looking at the job as well.

214
00:18:50,610 --> 00:18:52,290
So if I'm looking at this job again.

215
00:18:58,890 --> 00:19:04,110
Right here on the right side, you will see job artifacts, I can browse them.

216
00:19:05,860 --> 00:19:08,140
Obviously had a built for that I have created.

217
00:19:11,380 --> 00:19:13,600
And I can download this text file.

218
00:19:16,080 --> 00:19:21,830
And if I were inspecting the contents of this file now on my computer after I have downloaded it, you

219
00:19:21,840 --> 00:19:28,470
will see it only contains Swails because Wil's was the last step in my process and it replaced everything

220
00:19:28,470 --> 00:19:29,670
that was previously there.

221
00:19:31,010 --> 00:19:34,600
Going back to the pipeline, let's see if it now properly works.

222
00:19:38,800 --> 00:19:46,930
And now we will see both stages are successful, so the build stage passed and a test stage passed as

223
00:19:46,930 --> 00:19:52,570
well, and we can look into detail at the console output and you will see here.

224
00:19:54,760 --> 00:20:02,320
The first test was successful, we can go inside to build for the, um, we can view the contents of

225
00:20:02,320 --> 00:20:10,330
the car that takes the file using a cat key at command and then with grep, we are searching the contents

226
00:20:10,330 --> 00:20:13,600
of this file for Chessie, for engine and four wheels.

227
00:20:14,230 --> 00:20:17,950
And all these commands succeed for that reason.

228
00:20:18,160 --> 00:20:20,290
The entire job is then successful.

229
00:20:20,450 --> 00:20:26,200
Then this is our test on this very simple to process car assembly line.

230
00:20:27,600 --> 00:20:35,370
Now, let's take a final look at the repository you will still see inside a project that only the Gottleib

231
00:20:35,760 --> 00:20:38,840
pipeline definition file is available.

232
00:20:39,390 --> 00:20:44,490
So the built for life that we have created, the Khadar takes the file.

233
00:20:44,670 --> 00:20:46,830
They are not added to the repository.

234
00:20:47,340 --> 00:20:49,650
They are created in the pipeline.

235
00:20:49,920 --> 00:20:52,860
And we have decided we want to save them as artifacts.

236
00:20:53,280 --> 00:20:57,390
And these artifacts are independent of the GETT project.

237
00:20:58,050 --> 00:21:03,150
So for that reason, they are not committed inside of project, but they are available separately for

238
00:21:03,150 --> 00:21:06,090
download or for additional steps, whatever is needed.

239
00:21:07,960 --> 00:21:14,910
So this is how you can use Getler theeye in order to create a very, very simple pipeline with two stages,

240
00:21:15,250 --> 00:21:21,880
we have used stages in order to define which our two stages build and test in the first stage.

241
00:21:21,910 --> 00:21:28,890
We have built the car using very simple commands and in the second part we tested that.

242
00:21:29,620 --> 00:21:31,390
What if we added to the car?

243
00:21:31,390 --> 00:21:38,560
Text file is right and using artifacts, we have made a communication between these two steps.