Integrating LVM with Hadoop


 Integrating LVM with Hadoop

These are the steps that are to be followed on the data node.

1. Firstly, add physical hard disks to the data node. Here I have added two hard disks.

2. Convert the hard disks to physical volume as volume groups can only be created with physical volumes.


3. We can see the info of physical volumes by using the command "pvdisplay"


4. Create a volume group with the above physical volumes. 
#vgcreate vg_name /disk-name1 /disk_name2... /disk_namen

5. We can use vgdisplay and vg-group_name to get the info about the volume group.


6. Create a partition in the logical volume of size that you want to contribute to the namenode. The command is "lvcreate --size <value> --name <value> vg_name"

7. Now format the partition of the logical volume.

8. Create directory you want to contribute to the namenode and mount the above logical volume to the directory.

9. Use "df -h" command to check if the logical volume is mounted to the desired directory.



10. Now, start the datanode and connect to the namenode and check the volume contributed to the namenode.

11. We can extend the size of datanode by extending the logical volume.
Use the command "lvextend --size <value> /dev/vg_name/lv_name"
And then format only the resized/added part by using the command 
"resize2fs /dev/vg_name/lv_name"

As you can see, I have increased my logical volume storage without unmounting or stopping the services.

ARTH- The School of Technologies
ARTH2020.18.13

Sai Kishen Kothapalli
Tamanna Verma
Vaibhav Maan
Rahul Kumar

Comments

Post a Comment

Popular posts from this blog

AWS SQS

Image
AWS SQS What is SQS in AWS? Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. SQS eliminates the complexity and overhead associated with managing and operating message oriented middleware, and empowers developers to focus on differentiating work. Using SQS, you can send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available. Get started with SQS in minutes using the AWS console, Command Line Interface or SDK of your choice, and three simple commands. SQS offers two types of message queues. Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery. SQS FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.   Amazon   SQS   is a message queue service   used by ...
Read more

Automation of Technology using Python-Script

Image
 Automation of Technology using Python-Script To Integrate Docker With Python3 Description — — — — — — — — — — — — — — — — — — — — — — — — To Configure Docker on Top of REDHAT LINUX 8 To Start Docker To Stop Docker To launch a container on Docker To Start and attach Container (if already exists) To stop container To check for Running Container To check for all container (Either running or stopped) To delete/remove the container on docker — — — — — — — — — — — — — — — — — — — — — — — — — SEE MENU OF DOCKER ON PYTHON: STEP 1. To Configure Docker on Top of REDHAT LINUX 8 There has to some step to follow to configure docker on Redhat linux 8. In my case docker repo is already exists i have configured so i pressed ‘y’ and My Docker configured. STEP 2. To Start Docker STEP 3. To   Stop Docker STEP 4. To Launch a container/OS on Docker There has some step required to Launch container Download container Install and run container In my Case I downloaded Ubuntu and Install Ubuntu 14.04....
Read more

Increasing or decreasing static partition size in Linux

Image
 Increasing or decreasing static partition size in Linux 1. Create a static partition. Here I have created a partition of 5Gb. 2. Now, create a partition. Here I am creating a sub-partition of 1Gb.   3. Now, format and mount the partition. I am creating a folder with the name static and mounting the partition sdd1 to it. mkfs.ext4 /dev/sdd1--> For formatting mkdir /static mount /dev/sdd1 /static --> For mounting 4. Now, Create a file in /static. Go to static using cd /static. 5. In order to increase the size of a static partition, we need to unmount it and also delete the existing partition. unmount /dev/sdd1 --> For unmounting fdisk /dev/sdd1 and then use 'd' for deleting the partition and 'w' to save the changes. 6. Create a new partition and enter the desired size you want the partition to have. I am not entering any size here, So it will take all the 5Gb from the partition I created of 5Gb i.e., /dev/sdd 7. Here, we can clearly see that the size is now 5Gb....
Read more