1

00:00:06,050 --> 00:00:07,820

So just a quick recap.



2

00:00:07,820 --> 00:00:16,190

Until now we have learned how to install Redis and how to configure it and how to connect to Redis server



3

00:00:16,190 --> 00:00:17,240

using Redis client.



4

00:00:19,980 --> 00:00:25,500

So in this server or in this session, I am again going to connect to Redis client, using Port which



5

00:00:25,500 --> 00:00:32,590

is 6 3 7 0 and if we see the total number of keys available in Redis right now is only 1.



6

00:00:32,590 --> 00:00:33,080

So



7

00:00:35,930 --> 00:00:44,640

I will clear this by doing Flushall  and if you see again we should not have anything in the Redis



8

00:00:44,970 --> 00:00:45,200

.Okay.



9

00:00:45,200 --> 00:00:47,310

So right now we do not have anything saved in Redis



10

00:00:50,650 --> 00:00:56,120

And if you want to write something again to Redis,  the command which we need to use it Set Command, followed with



11

00:00:56,190 --> 00:00:56,720

the key.



12

00:00:56,870 --> 00:01:02,900

So which is going to be, let's say, in this case I want to keep all the Country short code along with the



13

00:01:02,900 --> 00:01:06,850

Country's Description. So I can put something like this.



14

00:01:08,090 --> 00:01:16,640

This is a key value pair for country short code and the country's Full Name. so once I do that and



15

00:01:16,850 --> 00:01:21,950

if I do key * I'm going to get this back and if I want to see the value of any key.



16

00:01:21,980 --> 00:01:27,110

So I need to use get Command followed  with the key name.



17

00:01:27,110 --> 00:01:33,800

So if I do like this. It is going to give me the value, which is against this, which is stored against



18

00:01:33,800 --> 00:01:36,130

this ID



19

00:01:36,420 --> 00:01:40,250

Remember you can also put a key in the ""



20

00:01:40,260 --> 00:01:41,750

It is going to work the same way.



21

00:01:47,600 --> 00:01:52,060

And I can insert more data by following the same approach.



22

00:01:57,560 --> 00:02:04,290

So that's how you do something, you know, Sequential. It is  like Sequential Way of doing things.



23

00:02:04,340 --> 00:02:10,350

But let's say, if you want to insert or  you want to do a Mass Insertion, where you want to upload



24

00:02:10,350 --> 00:02:14,870

like 1 million records and more than millions of records in Redis.



25

00:02:14,870 --> 00:02:16,370

So what is the best way to do it.



26

00:02:17,950 --> 00:02:19,840

So in this session we are going to learn that



27

00:02:23,030 --> 00:02:32,160

for Demo purpose I'm going to exit from here and then we are, let me clear the screen and right now i am in



28

00:02:32,900 --> 00:02:35,040

Redis Admin User's Home Directory.



29

00:02:37,930 --> 00:02:41,660

Here I have created a folder, which is called Raw_data.



30

00:02:41,770 --> 00:02:49,480

So once we go inside this folder. I have downloaded a sample file which is called countries. csv



31

00:02:49,540 --> 00:02:56,840

So if I show you some data which is  available in countries. csv is, something like this.



32

00:02:56,840 --> 00:03:03,510

So it has countries name and also it has countries code, which is separated by  comma



33

00:03:06,960 --> 00:03:07,330

OK.



34

00:03:07,370 --> 00:03:09,980

So this is like a csv, which I have.



35

00:03:10,400 --> 00:03:12,800

But using Redis, will not be able to upload this



36

00:03:12,800 --> 00:03:13,190

csv.



37

00:03:13,220 --> 00:03:19,640

So we need to convert it to a format, which is actually recognizable by Redis and which can be used



38

00:03:19,640 --> 00:03:21,170

by Redis mass insertion.



39

00:03:25,260 --> 00:03:34,520

So if you follow Redis Documentation, the mass insertion mechanism which they have, which they advise



40

00:03:34,530 --> 00:03:36,500

is called something like



41

00:03:36,550 --> 00:03:40,020

Redis-cli  and --pipe.



42

00:03:40,280 --> 00:03:40,770

Okay.



43

00:03:41,240 --> 00:03:50,010

So the way it works is, basically you create a file which contains list of all the key value pairs OK.



44

00:03:50,010 --> 00:03:52,250

And that actually should be in this format.



45

00:03:52,260 --> 00:03:59,390

So it is going to be like you know, a list of all the set commands, which is actually available in a file



46

00:04:00,090 --> 00:04:01,120

and then you use Redis cli  pipe.



47

00:04:01,120 --> 00:04:01,460

to pass



48

00:04:01,480 --> 00:04:05,070

all this file.



49

00:04:05,070 --> 00:04:09,320

I mean the content of this file, to Redis server.



50

00:04:09,530 --> 00:04:16,540

And then, Redis server is  going to process all of this data in a sequentially, in a Sequential



51

00:04:16,540 --> 00:04:17,220

Manner.



52

00:04:17,420 --> 00:04:21,710

And then it is going to reply you back ,once all of the work is done.



53

00:04:22,340 --> 00:04:22,650

Okay.



54

00:04:22,680 --> 00:04:24,500

So why this is faster.



55

00:04:24,500 --> 00:04:32,560

The reason is,if you run all of these commands one by one. If you use set command and provided the value



56

00:04:32,560 --> 00:04:35,430

of, you know, Key Value Pair basically one by one.



57

00:04:35,440 --> 00:04:40,870

So Redis Server has to respond to each of the queries back.



58

00:04:40,870 --> 00:04:46,930

So you are making a request sequentially and then actually Redis has to reply back you with a response.



59

00:04:46,930 --> 00:04:54,960

So it increases the number of response and hence it also increases the  total amount of Throughput.



60

00:04:54,970 --> 00:05:02,530

Basically the total amount which is taken in the,  in order to process the request.



61

00:05:02,550 --> 00:05:06,220

Now when you follow this  Redis cli pipe approach.



62

00:05:06,220 --> 00:05:12,300

What it does is, let's say you have one million records or one million Key Value Pairs which you want



63

00:05:12,300 --> 00:05:13,130

to insert to Redis.



64

00:05:13,170 --> 00:05:17,400

So you supply the complete instruction at one go.



65

00:05:17,490 --> 00:05:18,920

And then Redis processes it.



66

00:05:18,960 --> 00:05:21,720

And then Redis response you back.



67

00:05:21,720 --> 00:05:21,940

OK.



68

00:05:21,960 --> 00:05:25,920

So basically Redis has to respond you back only one time



69

00:05:30,320 --> 00:05:31,670

So let's see this in action.



70

00:05:35,390 --> 00:05:36,950

So the file which I have is,



71

00:05:40,560 --> 00:05:43,380

you  know countries. csv, which is having Name and Code.



72

00:05:43,830 --> 00:05:50,100

And if you know bit of awk command, if I want to print only the first column. I can do something like



73

00:05:50,100 --> 00:06:01,150

this.



74

00:06:02,190 --> 00:06:04,140

So if you know awk command and



75

00:06:04,230 --> 00:06:09,950

If I run something like this, by default the daily meter which is used is space.



76

00:06:10,230 --> 00:06:10,470

OK.



77

00:06:10,500 --> 00:06:17,370

So first column is going to be treated anything, which comes before a space and then anything which



78

00:06:17,370 --> 00:06:21,770

comes after a space is going to be treated as a Second Column.



79

00:06:21,780 --> 00:06:26,660

But in our case, we do not have this value  separated  by space. Rather what I have is



80

00:06:26,900 --> 00:06:30,040

These values are separated by a , separator.



81

00:06:30,080 --> 00:06:35,210

So that's why I am using -F to specify the field daily meter.



82

00:06:35,270 --> 00:06:42,330

And now if I print this. So I have got content of First Column.



83

00:06:42,890 --> 00:06:44,900

Okay, so First Column contains Country Names



84

00:06:48,880 --> 00:06:55,530

and if I do a Second Column, then I'm going to get all the Short Codes.



85

00:06:56,350 --> 00:06:56,700

Okay.



86

00:06:56,710 --> 00:06:58,230

Don't worry much about the data.



87

00:06:58,240 --> 00:07:01,430

The data is just the sample data, which I have downloaded.



88

00:07:01,430 --> 00:07:04,800

So sometimes you see like, its a big list of string.



89

00:07:04,810 --> 00:07:07,210

So this is something which can be ignored



90

00:07:11,100 --> 00:07:11,450

okay.



91

00:07:11,460 --> 00:07:16,510

So now the format, in which we want the data is like a Redis



92

00:07:16,530 --> 00:07:16,950

Command.



93

00:07:17,120 --> 00:07:17,390

Okay.



94

00:07:17,400 --> 00:07:21,680

So we should have Set as the keyword which is Redis



95

00:07:21,680 --> 00:07:22,140

Command.



96

00:07:22,140 --> 00:07:29,480

To insert the data and then we want space followed with the Key and then space followed with the Value.



97

00:07:29,500 --> 00:07:30,700

So what I have done is



98

00:07:39,090 --> 00:07:42,750

I use some awk command,  to prepare the data and I have



99

00:07:46,500 --> 00:07:47,610

ran something like this



100

00:07:53,930 --> 00:07:54,350

okay.



101

00:07:54,620 --> 00:08:00,830

If you see, this has given me Output or something which we wanted to Export.



102

00:08:01,220 --> 00:08:07,070

And after that what I did is, i  removed everything which comes after space because the value which Redis



103

00:08:07,110 --> 00:08:13,580

is going to recognize is, Set keyboard which is Redis command to insert the Data and then space then



104

00:08:13,580 --> 00:08:16,940

followed with a Key Name and then space followed with the Value.



105

00:08:17,330 --> 00:08:24,700

So in this case, if  your value contains something, which is including Spaces. You



106

00:08:24,700 --> 00:08:31,540

can actually quote the value within " " or otherwise if it is just for the practice purpose,



107

00:08:31,540 --> 00:08:37,240

you may get rid of the space.  So I have already prepared the  file and the file which I have prepared is called



108

00:08:37,240 --> 00:08:39,910

countrycode_ names. csv



109

00:08:46,830 --> 00:08:53,690

So if you see, the content of this file is very clean and I basically what I have in this file is, something



110

00:08:53,690 --> 00:09:00,260

like this. So I have set keyword, followed with Keys.  Which is Country Short Code and then followed with



111

00:09:01,550 --> 00:09:08,490

a Value, which is the Description of the Country.



112

00:09:08,550 --> 00:09:09,150

So now,



113

00:09:11,870 --> 00:09:17,810

let's once check, how many total number of Records  are available and the way to do is, you can use



114

00:09:17,840 --> 00:09:22,150

wc-L  command, followed  with a File Name.



115

00:09:22,340 --> 00:09:28,250

So what this going to do is, it is going to count the total number of lines which are available in that



116

00:09:28,250 --> 00:09:36,480

file.  So we see, we have around 249  Key Value Pairs, which are available in this file.



117

00:09:36,480 --> 00:09:40,280

Which we want to insert to Redis. Okay.



118

00:09:40,300 --> 00:09:43,150

So now  the way to insert  to Redis is,   you use cat



119

00:09:43,150 --> 00:09:46,030

command.  Let me clear the screen once.



120

00:09:49,380 --> 00:09:51,090

So the way to do is, you use cat



121

00:09:51,090 --> 00:09:58,380

command, followed with the the File, which you have prepared in order to pass all this insertion to Redis.



122

00:09:58,380 --> 00:10:00,760

Okay.



123

00:10:00,770 --> 00:10:05,470

Just make sure the file, you have already prepared in the format, which is accepted by Redis.



124

00:10:05,510 --> 00:10:09,740

Otherwise, you are going to get errors. So use



125

00:10:09,740 --> 00:10:10,240

cat command



126

00:10:10,250 --> 00:10:18,380

Space followed with the File Name and then you use a Pipe symbol and then you use Redis cli



127

00:10:18,380 --> 00:10:26,750

and since we are running, we are not running it on a default Port of 6 3 7 9.  We are running Redis server



128

00:10:26,770 --> 00:10:36,770

on Port 6 3 7 0.  So we also have to specify Port explicitly,  by providing the Port and also in this



129

00:10:36,770 --> 00:10:38,270

case we are running host.



130

00:10:38,270 --> 00:10:43,040

I mean the Redis server itself is running on the same local machine. So we do not have to provide the



131

00:10:43,040 --> 00:10:50,300

Host name, but in case you want to connect to Redis server, which is running remotely. So you can also provide



132

00:10:50,300 --> 00:10:58,110

the Host name by typing -h and then the name of the Host name.



133

00:10:59,370 --> 00:11:03,270

OK so it can be any IP address on which Redis is running.



134

00:11:03,570 --> 00:11:10,910

So in this case we do not need this.



135

00:11:11,060 --> 00:11:11,380

All right.



136

00:11:11,390 --> 00:11:12,470

So that's basically it.



137

00:11:12,500 --> 00:11:17,360

So what it will do is, it is going to read everything, which is there in Country Code Names and then going



138

00:11:17,360 --> 00:11:22,490

to pass it to Redis cli, to perform the operation and let's see what is the response we get.



139

00:11:23,390 --> 00:11:24,590

So I'm going to hit enter



140

00:11:27,330 --> 00:11:30,170

and why we got response in this way is



141

00:11:33,680 --> 00:11:35,900

because we use this command.



142

00:11:35,900 --> 00:11:44,330

So basically what it did is, it passed all of those lines one by one to Redis cli and we got actually



143

00:11:44,510 --> 00:11:47,660

the same number of response back.



144

00:11:47,830 --> 00:11:51,070

Now what ever number of records which we had in this file.



145

00:11:51,530 --> 00:11:51,870

OK.



146

00:11:51,920 --> 00:11:53,060

This is not we want to do.



147

00:11:53,060 --> 00:11:56,070

So we should have also put --pipe.



148

00:11:56,750 --> 00:12:02,910

So what this will do is, it is going to give all of these instruction at once to Redis sever.



149

00:12:03,020 --> 00:12:08,810

And then we are going to get only  one Response back.  OK we are not going to get these many number



150

00:12:08,810 --> 00:12:16,320

of Responses back from Redis. So let's run this command. Hit enter and now you see this time we have got



151

00:12:16,320 --> 00:12:27,100

only one response.  OK,  so it says, replies to 49 which means we have been able to update to 49



152

00:12:27,270 --> 00:12:30,320

Key Value Pair into Redis.



153

00:12:30,360 --> 00:12:32,340

And also there were 0 errors.



154

00:12:32,530 --> 00:12:38,340

So you saw the difference between  previous approach and this approach is,  that in previous approach



155

00:12:38,370 --> 00:12:42,560

we have got almost 249 Responses from Redis Server



156

00:12:42,720 --> 00:12:50,960

And in this we have received only one Response and using this approach you can insert millions of record



157

00:12:50,960 --> 00:12:56,940

in a very fast manner to the Redis Server.  Now to verify the result



158

00:12:57,310 --> 00:12:58,700

let's log into Redis Client



159

00:13:04,090 --> 00:13:09,850

and we can run dbsize command to see the total number of Records which are inserted to Redis right



160

00:13:09,880 --> 00:13:10,190

now.



161

00:13:10,510 --> 00:13:14,620

So we see total 247 Key Value Pairs.



162

00:13:15,130 --> 00:13:22,760

If you remember, in the file we had 249 total number of lines.  Which means you might be thinking



163

00:13:22,760 --> 00:13:29,660

that there should be 249 records inserted  to Redis but that's not the case.



164

00:13:29,660 --> 00:13:32,840

There may be you know,  key value  which is Duplicate.



165

00:13:33,170 --> 00:13:40,830

So in the case of Duplicate, what Redis does is, it just updates the previous value with a new value or



166

00:13:40,840 --> 00:13:49,730

let's say, if the Key Name was same,  but it had a different Value.  So the previous value whatever was inserted



167

00:13:49,770 --> 00:13:52,980

is going to replace with the New Value.



168

00:13:53,000 --> 00:13:53,200

Okay.



169

00:13:53,230 --> 00:13:58,780

Now to illustrate this point,  we can do something like, let's say we have a key which is called Color.



170

00:13:59,290 --> 00:14:02,060

In which we have a value of Red.



171

00:14:02,620 --> 00:14:08,390

And if I do Get Color,  we are going to get back Red color.



172

00:14:08,540 --> 00:14:15,670

Let's say I try to insert one more color,  which is Yellow.



173

00:14:16,400 --> 00:14:21,110

And once I do Get color,  what do we think I'm going to get back in return.



174

00:14:23,430 --> 00:14:29,700

If you think I'm going to get 2 values which is Red and Yellow, so that's not how Redis works.



175

00:14:29,700 --> 00:14:37,190

So what it has done is,  it has replaced, you know,  whatever value which  was against this key,  which is called Color



176

00:14:37,430 --> 00:14:41,110

to Yellow.  Okay so I should be getting Yellow in return.



177

00:14:42,050 --> 00:14:44,660

As you see when I run this Get command I get Yellow.



178

00:14:45,050 --> 00:14:52,550

So basically if you have Duplicate Keys, so whatever the latest value which you have supplied with that



179

00:14:52,550 --> 00:14:54,380

key is going to be your Last Value.



180

00:14:54,710 --> 00:14:59,400

So that's the reason,  total number of records which we see is only 247



181

00:15:02,560 --> 00:15:02,870

okay.



182

00:15:02,910 --> 00:15:04,860

I have cleared the screen by passing clear



183

00:15:04,860 --> 00:15:10,060

command and then we are going to do keys *



184

00:15:10,080 --> 00:15:19,300

So it will show us  list of all the Keys,  which are available in Redis and here we can see we have,  these



185

00:15:19,300 --> 00:15:31,560

many number of, we have now 248 total records and if I want to see any Country, you know Full Description



186

00:15:31,560 --> 00:15:35,160

so I can use Get command and I can supply the Country Code



187

00:15:38,050 --> 00:15:38,350

okay.



188

00:15:38,390 --> 00:15:39,130

Something like this.



189

00:15:46,890 --> 00:15:48,810

Now before we move on to next session,



190

00:15:51,830 --> 00:15:52,970

let me show you something.



191

00:15:52,970 --> 00:15:53,360

So



192

00:16:00,220 --> 00:16:01,330

let me clear the screen.



193

00:16:01,360 --> 00:16:07,200

And now to see whether the Redis process  is running or not we can use ps -ef command.



194

00:16:07,330 --> 00:16:13,260

If you are on Unix or Linux  and then you can grep by Redis keyword.



195

00:16:13,750 --> 00:16:18,050

So you are going to get something like this response.



196

00:16:18,070 --> 00:16:18,340

Okay.



197

00:16:18,340 --> 00:16:20,200

So this is the Redis process, which is running.



198

00:16:20,230 --> 00:16:28,110

So here you can see Redis server,  which is running and it is running on Local host which is being denoted



199

00:16:28,110 --> 00:16:33,410

by this IP address and also it is running on,  listening on Port  6 3 7 0.



200

00:16:33,420 --> 00:16:35,760

Now let's say, if someone



201

00:16:39,360 --> 00:16:46,600

actually kills this process or let's say the server reboots or there is any incident happens using which



202

00:16:46,600 --> 00:16:47,730

Redis, Redis



203

00:16:47,880 --> 00:16:56,540

application crashes and once this happens,  we are going to see whether  the data which was



204

00:16:56,540 --> 00:16:58,030

in Redis still exist or not.



205

00:16:58,250 --> 00:16:58,570

OK.



206

00:16:58,590 --> 00:17:06,350

So to simulate that I'm going to type kill -9 and going to kill the process,  which is running right



207

00:17:06,380 --> 00:17:07,190

now.



208

00:17:07,250 --> 00:17:07,520

Okay.



209

00:17:07,520 --> 00:17:13,310

So once we do that,  Redis process which was running on,  running with this process idea is going to get



210

00:17:13,310 --> 00:17:13,630

killed.



211

00:17:16,000 --> 00:17:21,850

Now to make sure whether the process is killed,  we can try to connect to Redis again by using



212

00:17:21,900 --> 00:17:22,180

Redis cli



213

00:17:27,850 --> 00:17:32,090

and this time you see when I,  actually i supplied the wrong



214

00:17:34,930 --> 00:17:35,440

Port.



215

00:17:35,460 --> 00:17:35,670

Yeah.



216

00:17:35,700 --> 00:17:41,770

So the process was previously running on this ports.  So once we do this,  now we can see we are no longer



217

00:17:41,770 --> 00:17:43,240

able to connect to Redis.



218

00:17:46,870 --> 00:17:47,060

Now



219

00:17:47,060 --> 00:17:52,780

let me start Redis Server again by using Redis Server command



220

00:17:52,950 --> 00:17:58,590

And if you remember we have also created a Configuration File,  which is available inside



221

00:17:58,590 --> 00:18:02,920

Redis -stable folder. So whenever I am starting the server.



222

00:18:02,930 --> 00:18:06,350

I am also supplying the Redis Configuration File,  which we have created.



223

00:18:07,070 --> 00:18:11,630

So make sure whatever configuration file which you have created,  you need to supply it whenever you are



224

00:18:11,630 --> 00:18:12,720

starting Redis  server.



225

00:18:14,720 --> 00:18:21,530

And since I want to run it, in a , as a background process.  I have also provided & in the end.



226

00:18:22,460 --> 00:18:26,890

So hit enter and now Redis  has started again.



227

00:18:28,860 --> 00:18:35,000

You can verify the same by using ps-ef  command again and grepping based on Redis  keyword.



228

00:18:35,010 --> 00:18:41,410

Now you can see Redis  is running,  on the process ID is this 2 6 6 4 9.



229

00:18:41,450 --> 00:18:44,660

And the Port on which it is listening on is 6 3 7 0.



230

00:18:46,040 --> 00:18:50,420

Let me clear the screen and let's connect to Redis ,  using Redis Client.



231

00:18:55,550 --> 00:18:55,830

okay.



232

00:18:55,850 --> 00:19:04,370

Now let's run dbsize command and see how many records are there after Redis  has restarted.



233

00:19:04,390 --> 00:19:12,400

So you see now,  since we simulated Redis  crash and we actually shut down Redis  process and then we



234

00:19:12,400 --> 00:19:13,950

started it again.



235

00:19:13,960 --> 00:19:18,560

Now if I run dbsize, I do not see anything available in Redis. It is showing me 0



236

00:19:18,610 --> 00:19:26,050

And if I do Keys * to see all the keys which are  available in  Redis.   I do not get anything.



237

00:19:26,050 --> 00:19:29,140

So basically whatever was stored in Redis  has gone.



238

00:19:29,820 --> 00:19:30,120

OK.



239

00:19:30,130 --> 00:19:36,850

So if you remember from our initial tutorials,  we have learnt that Redis  by default keeps everything



240

00:19:36,910 --> 00:19:42,630

in memory and by default there is no persistence which is enabled.  OK.



241

00:19:42,640 --> 00:19:46,870

So now we need to learn the way how to enable Persistence.



242

00:19:46,900 --> 00:19:53,380

So let's say whenever Redis  is failing or this server on which Redis is running is rebooting,  we do not



243

00:19:53,380 --> 00:20:01,440

want our data to lose. Okay,  we don't want any data loss.  So we are going to learn what are the different approaches



244

00:20:01,440 --> 00:20:02,610

which we can follow to



245

00:20:05,750 --> 00:20:06,560

keep our data.



246

00:20:06,560 --> 00:20:07,250

Persistent.