1 00:00:00,330 --> 00:00:06,090 In this video, I'll set up a Volbeat service to read system dialogues, so that's the diagram that's 2 00:00:06,090 --> 00:00:09,660 megafauna serve other smaller search server that we set up in last video. 3 00:00:09,690 --> 00:00:10,440 And it works. 4 00:00:10,470 --> 00:00:17,730 So now to install a dedicated service whose job is to read log files and send it off to the elasticsearch 5 00:00:17,730 --> 00:00:21,290 server and you can install FALBE on any server you want. 6 00:00:21,570 --> 00:00:24,600 So there are the instructions to download from this page. 7 00:00:24,790 --> 00:00:26,860 I'm going to use the Debian manager. 8 00:00:27,330 --> 00:00:28,560 These are my commands here. 9 00:00:28,800 --> 00:00:29,880 Log on to my server. 10 00:00:29,910 --> 00:00:33,990 I'm going to use my speaker server this then presenta. 11 00:00:34,870 --> 00:00:38,740 All right, C.D., Etsi Whitebait. 12 00:00:40,300 --> 00:00:43,560 These are the files that came with Will edit the Volbeat, why? 13 00:00:43,990 --> 00:00:46,330 OK, so let's check if it's running. 14 00:00:48,380 --> 00:00:55,100 Right, it's not that's good now with Volbeat, it can read log files from many, many things that store 15 00:00:55,100 --> 00:01:00,590 log files many common processes such as Web servers or radios or anything else. 16 00:01:00,610 --> 00:01:05,420 So to get a list of modules of falbe can use, you can type Volbeat modules list. 17 00:01:06,210 --> 00:01:11,480 OK, so here it's showing me a whole bunch of processes that you might recognize. 18 00:01:11,480 --> 00:01:18,620 Tomcat Systems, Semantic Readers', Rabbit, MQ Postgres, Smyers, Cucamonga, DB Fana, lots and 19 00:01:18,620 --> 00:01:19,550 lots and lots of things. 20 00:01:19,700 --> 00:01:23,930 Now all those modules are disabled either on the disabled. 21 00:01:24,120 --> 00:01:27,680 What I'm going to do is enable the system module, the. 22 00:01:29,110 --> 00:01:36,340 By falbe modules and able system, Folbigg modules enable system now by check that list again and just 23 00:01:36,340 --> 00:01:39,020 scroll up you'll see enabled system. 24 00:01:39,040 --> 00:01:39,450 Excellent. 25 00:01:39,470 --> 00:01:39,850 So. 26 00:01:40,880 --> 00:01:42,660 It still hasn't started yet. 27 00:01:42,680 --> 00:01:46,040 That's good, that's edit the Y amount. 28 00:01:48,330 --> 00:01:52,850 OK, so Volbeat Input's blog OD's. 29 00:01:56,840 --> 00:02:00,440 I'm not using Kobana, so I can safely comment that out. 30 00:02:01,440 --> 00:02:07,440 OK, so here's where we set the address of the elasticsearch server, you could be running this locally, 31 00:02:07,440 --> 00:02:11,910 but I'm not running this on a different server, on a different part of the world and my elasticsearch 32 00:02:11,910 --> 00:02:12,290 server. 33 00:02:12,300 --> 00:02:14,040 So we put the IP address in. 34 00:02:15,210 --> 00:02:17,700 Which was that, and it was Port nine 200. 35 00:02:17,730 --> 00:02:23,280 Now this server won't be able to send messages to that IP port because it's IP restricted. 36 00:02:23,290 --> 00:02:25,680 So I'll out of IP will in a moment. 37 00:02:25,740 --> 00:02:30,330 Now, while this search server is not set up with username and password or even HTP, is there anything 38 00:02:31,260 --> 00:02:32,970 that could change that if I wanted to? 39 00:02:34,120 --> 00:02:41,050 Now, these processes here, I'm not using those things, so I'm just going to comment those out, but 40 00:02:41,230 --> 00:02:44,620 you can leave that if you want or you can comment about the whole lot. 41 00:02:45,100 --> 00:02:48,850 You can experiment with that you like, but I'm going to turn them all off just to see what happens. 42 00:02:50,030 --> 00:02:57,650 I mean, that is all good now if I look at this in more detail, what is actually happening in this? 43 00:02:59,180 --> 00:03:00,170 File here. 44 00:03:01,670 --> 00:03:09,650 When it opens up, it's looking for the modules in path config modules D Y amount, so control X to 45 00:03:09,650 --> 00:03:11,150 save what I have. 46 00:03:11,340 --> 00:03:11,930 Yes. 47 00:03:12,080 --> 00:03:23,000 And Ellis oh there's the module D older there that's go into that modules D, lists H, and we can say 48 00:03:23,210 --> 00:03:28,550 these are the actual Y emails for each of the modules and they're all disabled. 49 00:03:29,510 --> 00:03:33,710 But if I look at system down there, it doesn't have disabled after it. 50 00:03:33,740 --> 00:03:40,250 We can actually inspect what's in that voicemail or any other just by typing cat system or email. 51 00:03:40,250 --> 00:03:42,260 And that's the Y Amelle. 52 00:03:42,260 --> 00:03:42,980 That's it. 53 00:03:42,990 --> 00:03:45,350 That's just saying module system sist log enable. 54 00:03:45,350 --> 00:03:45,740 True. 55 00:03:45,860 --> 00:03:49,400 And you can look at each of those module files if you want to learn more about those. 56 00:03:49,400 --> 00:03:52,370 But this is not a course on ElasticSearch or Volbeat really. 57 00:03:52,370 --> 00:03:54,550 I'm just showing you how to quickly set it up. 58 00:03:55,490 --> 00:03:57,560 Now let's start Volbeat. 59 00:03:59,680 --> 00:04:03,550 And I can check his status very good. 60 00:04:03,580 --> 00:04:08,290 That's likely not sending data to ElasticSearch servers because IP restricted, so. 61 00:04:09,700 --> 00:04:17,079 Let's just see whether I can reach that despite by using a telnet, so telnet nine 200 trying in, that 62 00:04:17,079 --> 00:04:18,760 will just time out eventually. 63 00:04:19,820 --> 00:04:23,960 Because this server is unable to connect at Port nine 200. 64 00:04:25,420 --> 00:04:28,970 I'm just using Tonette just to verify if I can connect to the poor or not. 65 00:04:29,530 --> 00:04:33,460 OK, so that has timed out and able to connect to remote house. 66 00:04:33,610 --> 00:04:38,950 So let's log on to the basic search server and I'm going to list all IP tables rules and insert a new 67 00:04:38,950 --> 00:04:40,960 rule for my other server. 68 00:04:41,020 --> 00:04:43,240 This one just is a code. 69 00:04:43,780 --> 00:04:44,770 So copy that. 70 00:04:44,770 --> 00:04:46,300 IP tables, line numbers. 71 00:04:46,900 --> 00:04:52,900 I go and I'm going to insert a new rule here, a position to for my ISP code post. 72 00:04:53,740 --> 00:04:54,430 Copy that. 73 00:04:55,480 --> 00:05:00,840 They are happy tables, source IP will be that way. 74 00:05:00,910 --> 00:05:03,850 Let's double check that again and there we go. 75 00:05:03,880 --> 00:05:08,950 I'm accepting messages to Port nine 200 from Espie code on. 76 00:05:09,400 --> 00:05:15,280 Let's just telnet again from the other server just to see just to verify that telnet can connect. 77 00:05:15,310 --> 00:05:16,620 There we go next. 78 00:05:16,630 --> 00:05:19,300 So I can just get out of that by pressing to see. 79 00:05:19,330 --> 00:05:19,780 There we go. 80 00:05:19,900 --> 00:05:20,400 Excellent. 81 00:05:20,680 --> 00:05:27,220 That should now be sending information off to my best search service. 82 00:05:27,400 --> 00:05:28,150 So there we go. 83 00:05:28,150 --> 00:05:34,870 The Volbeat process is now running on my code server, sending off data about system logs to the elasticsearch 84 00:05:34,870 --> 00:05:35,400 server. 85 00:05:35,440 --> 00:05:38,680 Let's configure ElasticSearch data source down there. 86 00:05:38,820 --> 00:05:42,090 So let's go into data sources ElasticSearch. 87 00:05:42,400 --> 00:05:49,870 Now, I have to tell ElasticSearch to look at a new index for Volbeat, or I could create a new data 88 00:05:49,870 --> 00:05:53,880 source completely in the existing index, one that I set up in the last video. 89 00:05:53,890 --> 00:06:01,540 But to know what name the index is going to be, go on to ElasticSearch Server, which is this one here. 90 00:06:01,690 --> 00:06:04,860 And we need to look at the catalog again of indexes. 91 00:06:05,800 --> 00:06:07,180 So that was up here. 92 00:06:09,780 --> 00:06:11,760 OK, cat indices. 93 00:06:13,440 --> 00:06:19,220 And it showed me this index, one which I credit the last video, and then there's the new fall beat, 94 00:06:19,380 --> 00:06:27,800 seven, 10 with today's date now for the index, I can just use Volbeat seven dot star or 10 dot star. 95 00:06:27,810 --> 00:06:29,610 So I'll just demonstrate that. 96 00:06:29,880 --> 00:06:35,500 So I'm replacing this index here with Volbeat 10 dot star. 97 00:06:35,640 --> 00:06:39,630 The field nine time stamp is already correct in the Folbigg index. 98 00:06:39,630 --> 00:06:40,270 So that's good. 99 00:06:40,290 --> 00:06:42,120 So scientist go index. 100 00:06:42,150 --> 00:06:50,940 Okay, let's now go to the explorative and we're looking at the count of system dialogues now from my 101 00:06:51,000 --> 00:06:52,060 S.P. card server. 102 00:06:52,080 --> 00:06:54,240 I look at raw data. 103 00:06:55,350 --> 00:06:56,730 We can see fall. 104 00:06:57,030 --> 00:07:02,640 That's the index agent, Eisenheim Espie code, the nationwide agent type Volbeat. 105 00:07:04,400 --> 00:07:08,420 System event data set systems to slog and event module system. 106 00:07:09,430 --> 00:07:11,020 There's a lot of information. 107 00:07:11,050 --> 00:07:16,540 OK, so let's just try a query on that, so let's filter by process, name by process name. 108 00:07:17,990 --> 00:07:20,330 Kolan system. 109 00:07:21,390 --> 00:07:23,820 The NASA shamy the system day process. 110 00:07:24,150 --> 00:07:29,250 So anyway, there is just an enormous amount of data in there that you can look at. 111 00:07:30,820 --> 00:07:35,510 About age 11 and how many happened and the counts. 112 00:07:36,280 --> 00:07:41,860 OK, now I've only set up Falbe on one server there I speak, but I can set it up on multiple servers 113 00:07:41,860 --> 00:07:48,010 and they would all come through as showing a different host down here, hostname, for example, so 114 00:07:48,010 --> 00:07:49,300 we could do a search on that. 115 00:07:49,430 --> 00:07:57,430 The host name ISP code, and they're all ISP code already, so there was no difference. 116 00:07:57,490 --> 00:08:00,010 Also, you have the option to set tags. 117 00:08:00,010 --> 00:08:08,410 If you wanted to use tags, you can edit the file based email, scroll down to the tags section. 118 00:08:10,160 --> 00:08:11,280 OK, here it is. 119 00:08:11,300 --> 00:08:21,200 So tax tax service name, Web, Teare, whatever you want, I'm going to say my that describes one of 120 00:08:21,200 --> 00:08:22,220 my service. 121 00:08:22,220 --> 00:08:23,600 You could be saying anything. 122 00:08:23,600 --> 00:08:28,850 You like that string, however you think you want to logically order your service, Web service, database 123 00:08:28,850 --> 00:08:30,320 service, anything you like. 124 00:08:30,320 --> 00:08:33,799 My example, tag exit is restart Volbeat. 125 00:08:35,360 --> 00:08:39,799 Remember, if you make any changes to module's or the webmail, restart Volbeat. 126 00:08:41,080 --> 00:08:42,220 OK, very good. 127 00:08:42,250 --> 00:08:49,290 Now, back in Ghafar, after 10 seconds or so, there'll be a new field for tag that we can search for, 128 00:08:49,420 --> 00:08:50,830 see what it's called. 129 00:08:54,090 --> 00:08:54,540 There we go. 130 00:08:54,570 --> 00:09:00,530 Takes my example tag so you can have multiple Volbeat or with different tags and you can just filter 131 00:09:00,540 --> 00:09:02,570 or Istar, for example. 132 00:09:02,610 --> 00:09:03,570 That will show me everything. 133 00:09:03,570 --> 00:09:07,400 For my example tag, I ask you, Will, it's another example. 134 00:09:07,410 --> 00:09:14,270 I don't have anything with Tag Mosque or even with a name like you only have one falbe service set up. 135 00:09:14,280 --> 00:09:15,880 But anyway, hopefully you get the idea. 136 00:09:15,930 --> 00:09:21,900 Now also the datasource there, I just reconfigured an existing data source and changed the index name 137 00:09:21,900 --> 00:09:24,240 and it just so happened at the same time. 138 00:09:24,240 --> 00:09:24,900 Field name. 139 00:09:24,900 --> 00:09:32,080 But you could also write a novel, a data source, ElasticSearch ElasticSearch. 140 00:09:32,310 --> 00:09:34,680 This is my original one scientist. 141 00:09:37,610 --> 00:09:38,160 There we go. 142 00:09:38,570 --> 00:09:40,610 Version seven, excellent. 143 00:09:41,790 --> 00:09:45,780 And now when I explore, I have to elasticsearch. 144 00:09:46,700 --> 00:09:48,140 Data sources to choose from. 145 00:09:48,720 --> 00:09:49,190 So excellent. 146 00:09:49,220 --> 00:09:54,040 And remember, if you made changes, you IP tables, rules on your server should persist. 147 00:09:54,390 --> 00:09:59,810 So on my ElasticSearch server, I'm just going to persist those IP tables rules just in case I do a 148 00:09:59,810 --> 00:10:00,200 reboot. 149 00:10:00,650 --> 00:10:04,970 OK, so that's how to use Falbe as a collector and then rate it through the ElasticSearch datasource 150 00:10:04,970 --> 00:10:06,070 in Gravagna. 151 00:10:06,110 --> 00:10:06,890 Excellent. 152 00:10:07,130 --> 00:10:10,790 In the next video we'll look at Metrick and all install that on a window sized.