1 00:00:00,840 --> 00:00:05,160 And these next few videos we're going to look at ElasticSearch, I'm going to use Lustick Search seven 2 00:00:05,160 --> 00:00:05,880 point ten. 3 00:00:06,840 --> 00:00:12,540 Now I'm going to set up ElasticSearch DataSource, which connects to another server where I'm running 4 00:00:12,540 --> 00:00:19,260 the ElasticSearch process, you might already have ElasticSearch on your network, but I'm going to 5 00:00:19,260 --> 00:00:21,840 just install a minimal set. 6 00:00:22,020 --> 00:00:29,410 Anyway, the demonstration elasticsearch is another popular tool use for monitoring infrastructure. 7 00:00:29,460 --> 00:00:33,240 You may prefer every other data there is in this video. 8 00:00:33,240 --> 00:00:36,450 I'm going to set it up and connect to do some sample queries. 9 00:00:36,450 --> 00:00:42,840 And then in the next few videos, I'll set up Volbeat to read log files, which is similar to Lokey 10 00:00:42,870 --> 00:00:48,580 and also demonstrate Metric Beat, where I'll monitor metrics from a Windows server. 11 00:00:48,600 --> 00:00:54,420 And at the end you will be able to set up an ElasticSearch server in a simple configuration and set 12 00:00:54,420 --> 00:00:56,640 up Volbeat Metrically Collectors'. 13 00:00:56,640 --> 00:01:01,790 So ElasticSearch is very sophisticated and warrants its own course. 14 00:01:01,800 --> 00:01:03,720 So I'm going to get to show you some of the basics. 15 00:01:03,750 --> 00:01:10,020 It has collectors and those collectors are pushing data to the main service. 16 00:01:10,020 --> 00:01:13,580 And then we have an ElasticSearch data source which will read from the main service. 17 00:01:13,800 --> 00:01:16,590 Now ElasticSearch runs in the job of VM. 18 00:01:16,800 --> 00:01:20,790 I'm going to recommend a minimum of two gigabytes of RAM. 19 00:01:20,790 --> 00:01:25,710 You can run this on one gig of rent, but I haven't had much success doing that. 20 00:01:25,720 --> 00:01:29,190 So what I've done is gone and created a new server for myself on Digital Ocean. 21 00:01:29,190 --> 00:01:33,130 Two gigs of RAM and everything else doesn't really matter. 22 00:01:33,150 --> 00:01:34,740 That's the most important thing for me. 23 00:01:34,740 --> 00:01:39,360 And it is Ubuntu twenty point zero four and I have connected. 24 00:01:39,600 --> 00:01:40,770 This is S.H.. 25 00:01:40,920 --> 00:01:47,700 Okay, so the instructions for installing ElasticSearch can be found here, so I'll be following the 26 00:01:47,700 --> 00:01:50,520 Debian package manager instructions. 27 00:01:50,700 --> 00:01:56,150 So this is where I got my information from, but I have put it on this page so I can copy and paste 28 00:01:56,160 --> 00:01:56,560 it easier. 29 00:01:56,740 --> 00:01:59,190 First thing I want the public and key. 30 00:02:00,510 --> 00:02:07,170 Right, click presenter done instore dependencies, most systems will have this already, but I'm just 31 00:02:07,170 --> 00:02:07,960 going to do it anyway. 32 00:02:08,100 --> 00:02:09,210 I didn't need it. 33 00:02:09,440 --> 00:02:12,210 OK, save the repository definition. 34 00:02:13,290 --> 00:02:14,070 Press enter. 35 00:02:14,520 --> 00:02:20,880 OK, and now I can update and install ElasticSearch package using IAPT. 36 00:02:23,260 --> 00:02:30,520 OK, that is the elasticsearch server installed, but I haven't started or configured it yet, so we 37 00:02:30,520 --> 00:02:35,710 can verify just by calling status that it is there and it can be found, but it is not running. 38 00:02:35,710 --> 00:02:36,570 So that's OK. 39 00:02:36,730 --> 00:02:37,670 That was expected. 40 00:02:37,690 --> 00:02:40,030 Let's start the service now. 41 00:02:41,130 --> 00:02:47,610 Now, if you used one gig of RAM, it might not start at this point, but I'm using to gigs of RAM. 42 00:02:51,760 --> 00:02:53,180 OK, that looks good. 43 00:02:53,200 --> 00:02:54,790 Let's check the status again. 44 00:02:55,870 --> 00:02:57,280 Very good, active running. 45 00:02:59,060 --> 00:02:59,630 So. 46 00:03:00,870 --> 00:03:04,650 Actually, I can just verify that by using locally. 47 00:03:05,960 --> 00:03:09,000 And we go, there is a response on the server. 48 00:03:09,200 --> 00:03:14,480 Now, if you had problems, you can read the log files from General. 49 00:03:15,950 --> 00:03:20,760 You elasticsearch looks pretty good to me if you had any areas that would show up there, OK, when 50 00:03:20,760 --> 00:03:26,400 it was installed, it created a new user called ElasticSearch Pegasi ElasticSearch and they are the 51 00:03:26,400 --> 00:03:27,760 processes that it's running. 52 00:03:27,850 --> 00:03:33,960 OK, so now it's time to configure ElasticSearch, but we can KDAY into the ATC folder. 53 00:03:34,290 --> 00:03:40,140 So c.D, ATC ElasticSearch, Alice El-Hage. 54 00:03:40,410 --> 00:03:45,700 And these are the files, the configuration that I'll need to edit here is Lastic search y amount, 55 00:03:45,730 --> 00:03:47,100 Goudeau and I. 56 00:03:48,320 --> 00:03:54,350 ElasticSearch, why email they this scroll down and. 57 00:03:55,710 --> 00:04:02,100 By default, the only way you can query the ElasticSearch server is through localhost. 58 00:04:02,310 --> 00:04:07,500 As I just demonstrated, using KOLD before one twenty seven zero zero one, if I want to make this server 59 00:04:07,500 --> 00:04:12,080 accessible from other servers, I need to change the bar and address. 60 00:04:12,090 --> 00:04:15,970 So I'm going to use zero zero zero zero, which will bind to all network interfaces. 61 00:04:15,990 --> 00:04:17,880 I also need to add. 62 00:04:19,959 --> 00:04:28,600 Transport host, local host, and I'll leave the port as default, so control X, yes, and restart. 63 00:04:31,310 --> 00:04:36,770 Now, the reason for doing that is I will need my data source to be able to connect to this external 64 00:04:36,770 --> 00:04:38,590 server where I'm running ElasticSearch. 65 00:04:38,600 --> 00:04:45,110 So if I didn't do that, then the only way I could connect to the server was if I had given or running 66 00:04:45,110 --> 00:04:47,990 on the same server as well on localhost. 67 00:04:48,170 --> 00:04:50,450 OK, so that is still restarting. 68 00:04:51,780 --> 00:04:58,860 Starting and restarting and take 10 seconds, roughly my experience status or a good active of running. 69 00:04:58,870 --> 00:05:02,220 Now, this also means that I can access this from the Internet. 70 00:05:02,400 --> 00:05:09,600 I can access that external IP address of my ElasticSearch server con nine to hundred enter and it will 71 00:05:09,600 --> 00:05:11,550 show a response in the browser. 72 00:05:11,710 --> 00:05:18,300 Now, we should lock this down because there are services on the Internet that will spam this elasticsearch 73 00:05:18,300 --> 00:05:18,710 server. 74 00:05:18,720 --> 00:05:20,330 It doesn't take long to be discovered. 75 00:05:20,340 --> 00:05:25,500 So straight away when I'm going to do is I'm going to restrict deport nine two hundred using IP tables. 76 00:05:27,160 --> 00:05:36,610 So down here, I'm going to clear allow local host Access Port 9100, I'm also going to allow my Ghafar 77 00:05:36,610 --> 00:05:39,160 server, which is at i.p. 78 00:05:40,120 --> 00:05:42,940 That currently and I'm going to drop everything else. 79 00:05:44,760 --> 00:05:45,450 Very good. 80 00:05:46,310 --> 00:05:49,590 This time there are the rules there except drop. 81 00:05:49,850 --> 00:05:53,210 Now, I shouldn't be able to access that in the browser that will time out. 82 00:05:58,510 --> 00:05:59,470 OK, very good. 83 00:05:59,680 --> 00:06:07,090 OK, so before we can connect to it in California, we'll need to create an index on it and have some 84 00:06:07,090 --> 00:06:08,290 data in it, for example. 85 00:06:08,320 --> 00:06:11,020 So let's have a look at the data source configuration. 86 00:06:11,290 --> 00:06:18,010 So data sources do at a data source elasticsearch there is select that now. 87 00:06:18,040 --> 00:06:21,980 Mihiro is going to be that Colen nine 200. 88 00:06:22,000 --> 00:06:27,340 Now, the Ippei of Microfauna Server has been added to the IP tables, so they should bind it. 89 00:06:28,740 --> 00:06:34,920 Down here, we have to set the index nine, which we haven't created yet, and also a time filled name 90 00:06:34,950 --> 00:06:38,640 which has credit and I'm going to connect to seven plus. 91 00:06:39,040 --> 00:06:41,320 So if I try to save that now, it's not going to work. 92 00:06:41,340 --> 00:06:42,020 That's OK. 93 00:06:42,210 --> 00:06:47,650 Let's do some configurations to ElasticSearch to create at least an index that we can query. 94 00:06:47,740 --> 00:06:53,790 OK, so going back up here, I'm going to create an example index in more or less the search server 95 00:06:53,790 --> 00:06:56,010 called index one down there. 96 00:06:56,020 --> 00:07:02,640 So copy that and we'll put index address nine two hundred, index one. 97 00:07:02,640 --> 00:07:04,800 And it's putting a new index presenter. 98 00:07:04,950 --> 00:07:05,460 Very good. 99 00:07:05,460 --> 00:07:07,440 Acknowledged true index one. 100 00:07:08,100 --> 00:07:10,380 We can view the index metadata. 101 00:07:12,130 --> 00:07:14,710 It's some information about the index, the. 102 00:07:16,140 --> 00:07:22,230 I've used the pretty query string that just prints it out indented like that, you can leave that off 103 00:07:22,230 --> 00:07:27,210 if you like, like so and it's not quite so easy to read. 104 00:07:27,510 --> 00:07:28,250 It's up to you. 105 00:07:29,590 --> 00:07:34,660 Let's add some data to the index so that we've got something to read inside Ravana. 106 00:07:36,530 --> 00:07:44,930 OK, so clear, and if we go so pull content type application, Jason posting, that's post there to 107 00:07:44,930 --> 00:07:48,380 localhost nine to the index, it's a doc. 108 00:07:48,560 --> 00:07:53,080 And the data here inside these single quotes is the Jason. 109 00:07:53,240 --> 00:08:00,260 Now, I'm creating two new metrics in there named ABC with a value one to three, another one called 110 00:08:00,260 --> 00:08:06,080 Najm with the value X, Y, Z and a timestamp key in the timestamp. 111 00:08:06,080 --> 00:08:10,580 I'm just getting the current dict written in oiseau format. 112 00:08:10,730 --> 00:08:15,680 It's like every other data source of Carfagna will rate it once a timestamp against every row. 113 00:08:15,720 --> 00:08:17,170 So this is what I'm doing here. 114 00:08:17,180 --> 00:08:17,450 So. 115 00:08:17,450 --> 00:08:22,130 Presenter OK, so that was created a successful one, but it's true. 116 00:08:22,160 --> 00:08:28,490 Now just to demonstrate what this line does, again, if I was to just write data, I second in the 117 00:08:28,490 --> 00:08:32,539 console, it just writes the current data in UTC. 118 00:08:32,570 --> 00:08:38,510 So that's what I'm inserting into the timestamp column just up there in my index one. 119 00:08:38,730 --> 00:08:42,200 So anyway, we can now view the contents of the index. 120 00:08:42,440 --> 00:08:43,760 OK, there we go. 121 00:08:43,770 --> 00:08:50,930 There's the contents of the index and the value that I just inserted is the ABC one, two, three name 122 00:08:50,930 --> 00:08:52,700 and a timestamp thing. 123 00:08:52,730 --> 00:08:55,520 Whatever was at the time we executed that line. 124 00:08:55,550 --> 00:08:58,550 OK, so let's look at that value there and that value. 125 00:08:58,550 --> 00:09:03,860 They're now going back into the data source configuration ElasticSearch. 126 00:09:03,860 --> 00:09:11,720 The index name is Index one, and that's my timestamp field I'm using at Timestamp, the scientist that 127 00:09:11,720 --> 00:09:12,360 we go index. 128 00:09:12,390 --> 00:09:13,450 OK, excellent. 129 00:09:13,490 --> 00:09:16,130 Right now explore ElasticSearch. 130 00:09:16,280 --> 00:09:22,010 OK, so straight away it's shown me that there's something there, count one behind my head so we can 131 00:09:22,310 --> 00:09:25,910 zoom into that and look at that as raw data. 132 00:09:25,940 --> 00:09:26,540 And there we go. 133 00:09:26,540 --> 00:09:33,620 There's my line that I just inserted before indexed one TalkTalk ABC one, two, three nine x, y, 134 00:09:33,620 --> 00:09:33,950 z. 135 00:09:34,250 --> 00:09:36,080 At that time stamp, it's to another one. 136 00:09:40,160 --> 00:09:49,130 OK, this time I can do something else like five, six am, and, oh, anything really does matter. 137 00:09:50,170 --> 00:09:57,130 A successful one by refresh that at five seconds and look at that for the last five minutes, there 138 00:09:57,130 --> 00:09:58,170 are two rows. 139 00:09:58,240 --> 00:10:00,360 Now there's the second row there straight away. 140 00:10:00,370 --> 00:10:06,370 You could right now a Python script or anything else that can send data or anything. 141 00:10:06,370 --> 00:10:10,560 You like as many metrics as you like to that elasticsearch server. 142 00:10:10,570 --> 00:10:15,640 And you can read them in Safonov now and you can create a dashboard from that. 143 00:10:15,670 --> 00:10:22,210 So just very quickly, dashboards and each write a new dashboard at a new panel. 144 00:10:23,490 --> 00:10:27,240 These elasticsearch would go and save that. 145 00:10:28,880 --> 00:10:33,190 ABC and we go or read another one. 146 00:10:34,330 --> 00:10:35,670 So as I. 147 00:10:37,580 --> 00:10:39,890 ElasticSearch as a table. 148 00:10:40,790 --> 00:10:41,600 And we go. 149 00:10:42,780 --> 00:10:43,970 Look at the raw data. 150 00:10:44,700 --> 00:10:47,760 Go save that five. 151 00:10:48,620 --> 00:10:52,120 I go and there we go. 152 00:10:52,140 --> 00:10:53,310 So excellent. 153 00:10:55,390 --> 00:10:58,270 Last five minutes or so already. 154 00:10:58,300 --> 00:11:04,160 That is an example of putting data into LASTIC search and reading it in a dashboard microphone. 155 00:11:04,180 --> 00:11:13,130 OK, now you might have a lot of indexes in your elasticsearch so you can check those pool cat indices. 156 00:11:13,180 --> 00:11:15,410 I have one called index one. 157 00:11:16,240 --> 00:11:18,280 Now, I can also delete an index. 158 00:11:18,280 --> 00:11:20,160 So experiment as much as you like. 159 00:11:20,170 --> 00:11:26,190 And then when you're finished you can always delate index, one, two, three, etc.. 160 00:11:26,230 --> 00:11:27,310 I'm going to leave it there for now. 161 00:11:28,280 --> 00:11:33,980 OK, remember also for your service on the Internet like mine, and you use IP tables and you restart 162 00:11:33,980 --> 00:11:39,870 your server IP tables, rules won't be sites, so you'll need to save them time. 163 00:11:41,090 --> 00:11:43,830 Yes, I'm just installing I type was persistent. 164 00:11:43,860 --> 00:11:45,080 Yes, yes. 165 00:11:49,450 --> 00:11:50,270 OK, very good. 166 00:11:50,290 --> 00:11:55,490 Now, when I reboot my server ever, the IP table's rules will be reapplied. 167 00:11:55,680 --> 00:11:56,350 Okay, so excellent. 168 00:11:56,380 --> 00:11:58,690 So whether you use ElasticSearch or not is up to you. 169 00:11:58,840 --> 00:12:04,570 ElasticSearch has many different options for licensing, so you can either try the free version which 170 00:12:04,570 --> 00:12:09,380 I used or you can pay for it or register, etc., etc. far. 171 00:12:09,580 --> 00:12:09,960 OK. 172 00:12:10,480 --> 00:12:16,290 In the next video I'll set up File Beat and I'll read system logs from my server. 173 00:12:16,300 --> 00:12:23,980 So it will be similar to what I did with when setting up the Lokey datasource impromptu but being done 174 00:12:23,980 --> 00:12:25,570 using Lastic search. 175 00:12:25,570 --> 00:12:27,080 Volbeat excellent.