1
00:00:00,150 --> 00:00:04,050
OK, let's do something a little more advanced with the prompter you will read and next logs and create

2
00:00:04,050 --> 00:00:10,830
a simple dashboard Engine X reverse proxy that was installed at the beginning of the course, he had

3
00:00:10,830 --> 00:00:14,700
the reverse proxy Bafana with Engine X. So every request is going by.

4
00:00:14,700 --> 00:00:20,170
That reverse proxy is being locked into a log file and make it read that using prompts in Loki.

5
00:00:20,530 --> 00:00:24,900
Also in Loki will use what's called the pattern parser, but will go into that.

6
00:00:24,900 --> 00:00:31,560
First, we open up our scraped config file conflict, prompt Obama on our Quanta server and add this

7
00:00:31,560 --> 00:00:32,619
extra section here.

8
00:00:32,640 --> 00:00:37,850
This is a second script config called Engine X. This target is local host.

9
00:00:37,860 --> 00:00:42,830
The job name is Engine X and the path is VAR Log Engine X Star Log.

10
00:00:42,950 --> 00:00:46,580
So going on to Mega-fauna Server, OK, someone might have found a server.

11
00:00:46,590 --> 00:00:49,010
We'll have a look at that folder where the logs are.

12
00:00:49,020 --> 00:00:54,090
So c d the logs and generics ls lh.

13
00:00:54,270 --> 00:01:00,120
There are the log files that Engine X is saving, and you can see that they're accessible via the IDM

14
00:01:00,120 --> 00:01:00,450
group.

15
00:01:00,570 --> 00:01:06,120
So our prompt, our user is already in the IDM group, but if you are using a specific user for prompt,

16
00:01:06,120 --> 00:01:09,450
they'll then make sure that user is in the IBM group so that it can read the logs.

17
00:01:09,580 --> 00:01:12,600
OK, so let's now edit the prompt her config file.

18
00:01:12,600 --> 00:01:19,450
So see the usr local bin Telis L'hygiene that was config prompt.

19
00:01:19,450 --> 00:01:24,510
Tell this so that the nano config from tail y eml.

20
00:01:24,520 --> 00:01:26,730
OK, this is my existing pronto.

21
00:01:26,760 --> 00:01:31,410
Remember, I've explicitly said that two nine nine seven you can let that to zero if you like.

22
00:01:31,420 --> 00:01:38,340
That's the URL that my local prompt tail is sending a sending to a local Loki as existing script config

23
00:01:38,340 --> 00:01:40,400
whose job name is bollocks.

24
00:01:40,410 --> 00:01:43,050
I added the host label there for gharana.

25
00:01:43,080 --> 00:01:49,170
Now I've positioned my cursor where I want to start pasting now going to just copy that part, including

26
00:01:49,170 --> 00:01:53,150
the whitespace control, see right click and a paste case.

27
00:01:53,160 --> 00:01:57,570
So job name and your next set of conflicts targets local host labels and generics.

28
00:01:57,570 --> 00:02:04,500
And that is the path to the log files that will read so VAR Log Engine X style log book save that control

29
00:02:04,500 --> 00:02:11,700
x yes and to its restart restart from tail and check its status.

30
00:02:12,060 --> 00:02:15,930
It looks good active running and I'm not seeing any problems.

31
00:02:16,350 --> 00:02:24,660
OK, so we can now go into Carafano and open up, explore and find a new entry here on the job code

32
00:02:24,660 --> 00:02:31,680
Engine X, so click Engine X and that is the log stream selector job equals Engine X and show logs,

33
00:02:32,120 --> 00:02:36,000
and we can begin to see logs that prompt tail is now pushing into Loki.

34
00:02:36,360 --> 00:02:43,190
OK, so we can see here the file name is VAR Log Engine X Access Log Host Crafar Nut Job Engine X.

35
00:02:43,200 --> 00:02:47,820
There's another one access local Engine X. And if we look at the details of the launch, we can see

36
00:02:47,820 --> 00:02:50,180
that was opposed to the Lokey service.

37
00:02:50,190 --> 00:02:53,970
That's the IP address of my Esquibel server using a low key push method.

38
00:02:54,030 --> 00:02:58,560
If remember, I set my prompt in the mosque will server to go via the Engine X reverse proxy where it

39
00:02:58,560 --> 00:02:59,940
was using the domain name.

40
00:02:59,940 --> 00:03:08,070
And so that IP address there is my actual server that I'm using to make this video and I'm making requests

41
00:03:08,070 --> 00:03:11,970
to the graphical user interface every time I press a button on the Griffon user interface.

42
00:03:12,000 --> 00:03:18,420
Anyway, there's a lot of information in these log lines here that we can query, but this is a good

43
00:03:18,420 --> 00:03:23,790
opportunity to learn a new feature in Loki, and that is the pattern parser.

44
00:03:24,330 --> 00:03:29,430
The pattern parser will allow us to take parts of those log lines and create labels from them.

45
00:03:29,460 --> 00:03:33,770
So, for example, job equals Engine X will pass a pattern over that.

46
00:03:33,780 --> 00:03:38,820
So pipe pattern we're matching a string and putting matches into labels.

47
00:03:38,820 --> 00:03:44,400
So in this pattern here, according to new labels called method and Status, if I look at the log line

48
00:03:44,400 --> 00:03:47,580
here, there's those two hyphens there, there's two hyphens there.

49
00:03:47,610 --> 00:03:53,550
So we could be taking the first property that IP address and putting that into a value or that one.

50
00:03:53,550 --> 00:03:56,120
But we're taking method and status.

51
00:03:56,130 --> 00:03:58,260
So those here, that's the method.

52
00:03:58,260 --> 00:04:01,590
The word post and status is the number two I four.

53
00:04:01,590 --> 00:04:02,810
So copy that line.

54
00:04:02,820 --> 00:04:05,340
Put that into your query to pop pattern.

55
00:04:05,340 --> 00:04:09,990
We're matching that pattern, that whole string and creating two new labels called method and Status.

56
00:04:09,990 --> 00:04:14,640
And if it can match that string and find values to put into method and status, it will create them

57
00:04:14,640 --> 00:04:15,720
as new labels for us.

58
00:04:15,800 --> 00:04:16,890
We'll say that's a shift.

59
00:04:16,890 --> 00:04:18,089
Enter now.

60
00:04:18,089 --> 00:04:23,610
If I look at one of these lines, I've now got two new labels here method and status, so I can now

61
00:04:23,610 --> 00:04:27,270
start using those labels further in my query here.

62
00:04:27,300 --> 00:04:30,810
So, for example, let's count all by time, so can credit graph.

63
00:04:31,260 --> 00:04:38,760
So going to the beginning count over time bracket, what to the end and we'll say for range of one minute

64
00:04:38,910 --> 00:04:41,460
and then we'll close that off with the bracket shift.

65
00:04:41,460 --> 00:04:44,910
And so we now start getting a graph of the zoom to that day.

66
00:04:44,940 --> 00:04:45,410
There we go.

67
00:04:45,420 --> 00:04:51,270
We can start to see the different kinds of methods and status codes that our engineers service proxy

68
00:04:51,270 --> 00:04:51,630
is things.

69
00:04:51,630 --> 00:04:55,770
So status 200 Forget method status two hundred post method.

70
00:04:55,800 --> 00:04:59,820
There was a status four hundred down there instead of two 04 if I zoom in.

71
00:04:59,900 --> 00:05:04,250
To say that section there that looks at the more interesting, these are the kinds of status codes that

72
00:05:04,250 --> 00:05:08,960
we're seeing and that quite common to means okay, but you might get lots of four or four errors and

73
00:05:08,960 --> 00:05:10,250
that means file not found.

74
00:05:10,280 --> 00:05:13,340
You might get sort of a five hundred errors, which means it's a problem with the application running

75
00:05:13,340 --> 00:05:15,370
behind your web server or reverse proxy.

76
00:05:15,380 --> 00:05:17,620
So I'll create a four for error now.

77
00:05:17,630 --> 00:05:24,350
So if I go to the recipe code dot net and it just type in some junk that will return for a four page

78
00:05:24,350 --> 00:05:25,070
not found for a.

79
00:05:25,550 --> 00:05:27,860
Now we'll see that you now end your next job now.

80
00:05:27,900 --> 00:05:33,800
OK, so if I change that query to the last five minutes, I just zoom into that section there.

81
00:05:34,310 --> 00:05:37,740
There is a for for just the red line there.

82
00:05:37,760 --> 00:05:40,760
So that's the four a that I just generated about 10 seconds ago.

83
00:05:40,790 --> 00:05:41,240
Excellent.

84
00:05:41,270 --> 00:05:45,980
So on a busy web server, it's good to see what all the status codes are because if you're suddenly

85
00:05:45,980 --> 00:05:48,760
getting status five hundreds, it will stand out like a sore thumb.

86
00:05:48,770 --> 00:05:52,210
If you see a sudden rise in four iPhones, you're not as a problem as well.

87
00:05:52,220 --> 00:05:55,550
And there are many status codes and you can look those up on the internet what they mean anyway.

88
00:05:55,580 --> 00:05:56,280
So that's good.

89
00:05:56,300 --> 00:05:59,530
So they're looking at the typical log lines that you get from an engineer.

90
00:05:59,630 --> 00:06:05,810
So it's a small sample here familiar on there are many values, so we can see this IP address requesting

91
00:06:05,810 --> 00:06:10,360
that's called remote address is a time that's called time local here.

92
00:06:10,370 --> 00:06:15,440
We don't have a remote user, but you might see that sometimes there is a method that's post.

93
00:06:15,440 --> 00:06:16,260
There is the quest.

94
00:06:16,260 --> 00:06:19,330
That's the path that was being requested from your web server.

95
00:06:19,340 --> 00:06:22,300
There was a protocol payslip one point one.

96
00:06:22,310 --> 00:06:26,790
You'll see different versions of how you stop being requested by move further along.

97
00:06:26,810 --> 00:06:31,820
That's the status code to have seen that Bytes sent zero HTP refer.

98
00:06:31,820 --> 00:06:32,910
We're not saying that day.

99
00:06:32,960 --> 00:06:34,130
You might find that value.

100
00:06:34,130 --> 00:06:38,720
Sometimes I use to be user agent pronto, but when I'm using my browser that I used to be user agent

101
00:06:38,720 --> 00:06:41,270
is usually something more complicated.

102
00:06:41,270 --> 00:06:47,360
Like that Mozilla five app, a web kit, etc. So all those values can be extracted by modifying our

103
00:06:47,360 --> 00:06:47,810
pattern.

104
00:06:47,840 --> 00:06:54,830
OK, so here's an example where I write labels for remote address and time like also copy that string

105
00:06:55,010 --> 00:06:56,780
and will replace the whole load.

106
00:06:57,110 --> 00:07:03,200
And so if I look at the labels, it now says remote address and time local soccer, my queries to refine

107
00:07:03,200 --> 00:07:04,820
on those two values if I needed to.

108
00:07:05,120 --> 00:07:11,240
What I'm going to do is modify this one and add method status back so that you can see that we can use

109
00:07:11,240 --> 00:07:12,500
all those values if we want.

110
00:07:12,600 --> 00:07:15,500
My address time or method was at that position.

111
00:07:15,500 --> 00:07:18,860
Their method and status was at that position.

112
00:07:18,860 --> 00:07:20,900
Their status shift.

113
00:07:20,900 --> 00:07:21,470
Enter.

114
00:07:21,520 --> 00:07:26,330
OK, if I look at the labels, I'm also sing status and method again as well.

115
00:07:26,340 --> 00:07:27,290
So method post.

116
00:07:27,320 --> 00:07:32,690
Now, it's not advisable to create variables for all of these things if you don't actually using them

117
00:07:32,690 --> 00:07:34,370
because it's just not good for performance.

118
00:07:34,380 --> 00:07:36,020
Pretty well, I'm just showing you that it's possible.

119
00:07:36,050 --> 00:07:41,270
Also, you can change the names of the labels anything you like if you prefer my address like that.

120
00:07:41,420 --> 00:07:44,630
For example, it now says from my address like that.

121
00:07:44,660 --> 00:07:45,890
So you got the freedom.

122
00:07:45,890 --> 00:07:49,840
So the Pattern Pass is actually really good and it's actually very fast as well.

123
00:07:49,850 --> 00:07:55,090
In the past, you would use something like rejects in that position, but they say the pattern parser

124
00:07:55,100 --> 00:07:56,660
is now the fastest way of doing this.

125
00:07:56,660 --> 00:07:58,250
So and it looks pretty easy as well.

126
00:07:58,250 --> 00:07:59,460
So we use a pattern passel.

127
00:07:59,510 --> 00:08:00,710
OK, so excellent.

128
00:08:00,740 --> 00:08:03,650
Now I'm just going to get rid of my address there, so I'm not going to use it.

129
00:08:03,660 --> 00:08:09,200
I'm not going to use time local either, but I'm going to create a graph from that, but also group

130
00:08:09,320 --> 00:08:09,850
as well.

131
00:08:09,860 --> 00:08:12,230
So because then I'll use that in a dashboard.

132
00:08:12,290 --> 00:08:20,090
So going back to count over time bracket for one minute and close off that bracket, I'm going to sum

133
00:08:20,090 --> 00:08:21,710
that some until.

134
00:08:21,710 --> 00:08:27,190
So we're creating one line and then I'm going to group by status there.

135
00:08:27,660 --> 00:08:31,120
So I've got a simple graph now that is just showing status codes.

136
00:08:31,130 --> 00:08:35,990
I don't really care about the method, but if I did care, I could just put in method like that and

137
00:08:35,990 --> 00:08:39,200
I've got the label standing using the method and the status.

138
00:08:39,289 --> 00:08:40,070
I'm not going to use it.

139
00:08:40,130 --> 00:08:43,039
Also, another thing that I haven't shown you as well.

140
00:08:43,039 --> 00:08:51,590
You can change the order of this grouping clause by saying some high status and then that's the remainder

141
00:08:51,590 --> 00:08:52,300
of the query.

142
00:08:52,480 --> 00:08:54,450
And so that's the same result.

143
00:08:54,500 --> 00:08:55,400
So that's an option.

144
00:08:55,400 --> 00:09:00,380
If you prefer written like that some by status, then your query that also works for method, some by

145
00:09:00,380 --> 00:09:04,790
status method, some by status method, job, if you wanted to.

146
00:09:04,790 --> 00:09:05,720
But I'll do that.

147
00:09:05,870 --> 00:09:07,370
Now I'm going to use that in a dashboard.

148
00:09:07,370 --> 00:09:14,690
So copy that and let's create a new dashboard that an empty panel select low-key pace that query and

149
00:09:14,690 --> 00:09:23,720
to apply and just save this very quickly, calling it Engine X save and we can just to start down to

150
00:09:23,720 --> 00:09:25,240
15 minutes, for example.

151
00:09:25,250 --> 00:09:25,770
And then we go.

152
00:09:25,820 --> 00:09:30,980
I can add a log panel as well, so I can see the related log files.

153
00:09:31,040 --> 00:09:36,740
So that's at a panel time series or chosen logs.

154
00:09:38,400 --> 00:09:39,060
Low key.

155
00:09:40,330 --> 00:09:41,350
Curly brackets.

156
00:09:42,060 --> 00:09:43,950
Job engineers.

157
00:09:45,120 --> 00:09:46,320
Very good apply.

158
00:09:47,450 --> 00:09:53,510
Position at that time is that for the last five minutes and I want to know something more about that,

159
00:09:53,510 --> 00:09:56,060
for example, I can zoom in or consuming.

160
00:09:56,180 --> 00:09:58,190
Excellent and see the related little clients.

161
00:09:58,340 --> 00:10:01,570
OK, so that's very quickly a basic Engine X dashboard.

162
00:10:01,610 --> 00:10:04,790
I'm going to pause the video, create something a little more complicated.

163
00:10:04,790 --> 00:10:10,820
So anyway, I have go on and buy an extra panel here, which uses the bar gauge there just to create

164
00:10:10,820 --> 00:10:17,370
a summary of the remote addresses and how many times they're making a call to my web server.

165
00:10:17,390 --> 00:10:18,260
That's the query there.

166
00:10:18,350 --> 00:10:19,670
I'm using the bar gauge.

167
00:10:19,670 --> 00:10:24,530
Some can have a time job Engine X pattern remote address for the time range.

168
00:10:24,540 --> 00:10:27,620
I'm using this dollar range instead of one minute I'm using.

169
00:10:28,770 --> 00:10:35,490
The range property there are remote address that means that when I change the time here, the numbers

170
00:10:35,610 --> 00:10:42,210
will be more reflective of how many times in that period the last five minutes we can see that one of

171
00:10:42,210 --> 00:10:47,640
these IP addresses is making a lot of requests to my server, so I could deny that IP address.

172
00:10:47,640 --> 00:10:50,760
If I wanted to put any way, I'll save that save.

173
00:10:51,000 --> 00:10:53,940
Let's go back to the dashboard and I'll just reposition it.

174
00:10:55,130 --> 00:11:02,630
Like so anyway, this dashboard, Jason, here I'll put on my documentation so they can copy and paste

175
00:11:02,810 --> 00:11:08,020
that is down here on the Sample and snakes dashboard, so you could copy that whole lot to the clipboard,

176
00:11:08,030 --> 00:11:13,310
go to dashboards, manage or save that for I go.

177
00:11:14,260 --> 00:11:21,610
Import by panel, Jason Paste that let me go to the copy load, that name already exists, so I'm going

178
00:11:21,610 --> 00:11:24,040
to change it to something else import.

179
00:11:24,250 --> 00:11:30,330
OK, so I've got that loaded so we can see here straight away what's going on with my next reverse proxy

180
00:11:30,340 --> 00:11:30,880
anyway.

181
00:11:31,360 --> 00:11:37,750
Just so that, you know, my Gravano server is under a current dose, so I'm getting a lot of junk actually

182
00:11:37,750 --> 00:11:38,920
being sent to the server.

183
00:11:38,920 --> 00:11:39,940
We can see that down here.

184
00:11:39,970 --> 00:11:43,870
So if your Gryphon, a server, is on the internet, there are possibilities you might start getting

185
00:11:43,880 --> 00:11:46,240
dosed if someone wants to detox you.

186
00:11:46,240 --> 00:11:48,080
So I'm using digital lotion.

187
00:11:48,080 --> 00:11:50,170
Digital Lotion has an ebook firewall.

188
00:11:50,500 --> 00:11:56,710
So, for example, on a networking under firewalls, you can create a firewall called anything you like.

189
00:11:56,830 --> 00:12:00,910
Set your inbound and outbound roles and you can apply it to a droplet.

190
00:12:00,940 --> 00:12:05,730
So, for example, I can apply it to my grandfather droplet, but I've already done that so far.

191
00:12:05,740 --> 00:12:09,020
Just go backwards and modify my existing one.

192
00:12:09,040 --> 00:12:15,850
Right now, I have all IP version four enabled for hACE2 TPS, so I'm just going to edit that rule.

193
00:12:15,880 --> 00:12:21,070
I'm going to delete that rule and just have those two explicit I-Pace that are allowed to query you

194
00:12:21,100 --> 00:12:24,040
to be a smug, often a server that's port four four three.

195
00:12:24,040 --> 00:12:25,810
So I'll save that now.

196
00:12:25,810 --> 00:12:32,040
If I go back into Jovana, we'll start to see that these numbers will start dropping down.

197
00:12:32,050 --> 00:12:34,090
So fast forward this video for a moment.

198
00:12:34,780 --> 00:12:40,690
So all these extra IP addresses on the right here are all being blocked, except for the two drive explicitly

199
00:12:40,690 --> 00:12:42,970
allowed in my firewall.

200
00:12:43,450 --> 00:12:48,340
One of those is my server that I'm creating this video from, and the other one is my MySchool.

201
00:12:48,340 --> 00:12:49,070
So OK.

202
00:12:49,090 --> 00:12:51,160
So you can see now that the graph is going down.

203
00:12:51,190 --> 00:12:55,010
So I go, that's one of the things that serves only to get divorced occasionally.

204
00:12:55,030 --> 00:12:55,510
Excellent.

205
00:12:56,680 --> 00:13:04,270
So if I looked at the last one minute see with this case, we can see that there are less remote addresses

206
00:13:04,270 --> 00:13:09,250
and eventually just be just the two, which are my two IP that I've explicitly allowed.

207
00:13:10,310 --> 00:13:10,760
Excellent.