Common disks include CDs, U disks, hard disks, floppy disks, tape drives and other disks. The disks are formatted at a low level when they leave the factory to divide the logical structure of the disk, divide tracks, sectors, and disk attributes. Etc.
Disk partition
Partition: partition Create multiple independent file systems
MBR: master boot record Master boot record:
Track 0 sector 0 512bytes: 446bytes: bootloader
: magic number identifies whether MBR is valid
Disk partition formatting:
/
/bin, /sbin: binary program, /bin command used by ordinary users, /sbin management Commands used by the operator; programs needed to run the operating system itself
/usr/bin, /usr/sbin: binary programs, basic system tools
/usr/local/bin, /usr/local/sbin : Third-party program;
/lib
/usr/lib
/usr/local/lib
/etc
/tmp
/var
log
mail< br>cache
/proc: kernel image
/sys: hardware information
/media
/mnt
/opt :
/boot: boot directory, kernel
/home /USERNAME, /root:
/dev :
Device type:
b: random device
c: linear device
hard disk device: random access device
I/O, I /O Controller The integrated chip is called the controller
I/O Adapter The non-integrated chip is called the adapter.
IDE, ATA: Parallel, 133MBPS = 133/8 Mb/s
SCSI: small computer system interface, UItraSCSI 320MBPS = 320/8MB/s
1000 0,15000RPM Parallel
SATA: Serail, 300MBPS, 600MBPS, 6GBPS Bus speed Serial
SAS :
USB: (1.1, 2.0, 3.0) Serial
USB3.0 480MBPS
SSD: Solid-State Disk is also commonly used to connect to SATA port or USB port
Linux, /dev/
IDE, ATA: /dev/hd[az]
IDE: Master, Slave
The first IDE controller connects to the hard disk,
The second IDE controller connects to the optical drive, hdc
SCSI, SAS, SATA, USB: /dev/sd[az]
/dev/sda, /dev/sdb
File system: usually provided by the kernel
windows :
NTFS
FAT32
Linux:
ext2
ext3
ext4
xfs
reiserfs
nfs
iso9600
jfs
brtfs
To format a partition is to create a file system
RAW : Raw device
Low-level format: Create disk, divide track
Advanced format: Create file system
MBR: Master Boot Record
512bytes :
446bytes: bootloader
64bytes: Partition table, where each 16bytes identifies a partition
2bytes: a magic number that identifies whether the current hard disk is bootable
Extended partition:
divide into logical partitions again
fdisk, sfdisk , part
fdisk <= 15 partition
fdisk -l [DEVICE]
fdisk [DEVICE]
Interactive command:
d: delete a partition
n: create a new partition< br>p: Display the current partition status
w: Save and exit
q: Exit without saving
l: System ID corresponding to each partition type
83: linux
82: linux swap
8e: LVM logical volume
fd: soft raid
t: modify Specify the System ID of the partition
fdisk -l /dev/sda
Check the partition information recognized by the kernel: cat /proc/partitions
Let the kernel re-detect and read the hardware disk partition Table: partprobe
A warning appears, no problem on Red Hat 5, restart on Red Hat 6
RHEL6 let the kernel re-read the hard disk partition table:
partx -a [partitions] DEVICE clear Add one or all disks
Next, format the partition to create a file system
linux, mkfs
mkfs -t fstype /dev/part
RHEL5: default ext3
RHEL6: default ext4
mkfs -t ext3 /dev/sda3
blkid dev/sda3: display device uuid and file system type
mkfs -t ext4 = mkfs.ext4 = mke2fs -t ext4
mkfs -t ext3 = mkfs.ext3 = mke2fs -j = mke2fs -t ext3
mkfs -t ext2 = mkfs.ext2 = mke2fs = mke2fs -t ext2
Configuration file: /etc/mke2fs.conf: used to set default features and file system-specific features;
mke2fs -t {ext2|ext3| ext4}
-b {1024|2048|4096}: Block size
The block size depends on the CPU’s support for the memory page frame size. The default page frame size of the x86 system is 4K;
-L label: set Volume label
-m #: The proportion of the total space reserved for management use
-r #: Adjust the number of blocks reserved for management
-E: Adjust the file system extended attributes
vim /etc/mke2fs.conf can define the file type
To display the block size:
tune2fs -l /dev/sda3 | grep’Block size’
-l: display File system super block information
-L label: set volume label
-m: set the proportion of the total space occupied by reserved management blocks
-r #: adjust the number of reserved management blocks
-o: set default Mount options
-O: Set the default characteristics of the file system
-E: Adjust the file system extended attributes
e2label: Display or set the volume label
d2label DEVICE [Volume_label]< br>
mke2fs -t ext3 -b 1024 -L MYDATA /dev/sda3
tune2fs -l /dev/sda3 | grep’Reserved’: reserved block size
Mount: mount
mount [-t fstype] DEVICE MOUNT_POINT
mount [-t fstype] LABEL=”volume label” MOUNT_POINT
mount: all devices mounted on the current system
umount DEVICE
umount MOUNT_POINT
After mounting, the original data will be hidden, so it cannot be mounted on the common directory of the system.
When uninstalling, make sure that no process is accessing The mounted device, otherwise, it cannot be unmounted
Track: Track
Sector: Sector
Cylinder: Cylinder
/etc/fstab: The system will automatically mount when it is turned on Loaded file system:
FileSystem: a software, implemented in the kernel
windows:
NTFS,FAT32
Linux: vfs(virtual file system)
usi9660
nfs
cifs
ext2, ext3, ext4, xfs, jfs,reiserfs, vfat, GFS, GFS2, OCFS2
mount -t fstype [options] DEVICE MOUNT_POINT
DEVICE :
device File: /dev/sda1, /dev/hdb2
Volume label: LABEL=”label”
UUID: UUID=”uuid”
MOUNT_POINT:
blkid PARTITION< /p>
mkfs -t [fstype] PARTITION
Key points: 1. The file system must be supported before it can be used, that is, the corresponding kernel module of the kernel, or it has been integrated into the kernel
2. There must be corresponding File system creation tool, this is usually m kfs.fstype
mkfs -t ext2 = mkfs.ext2 = mke2fs
mkfs -t ext3 = mkfs.ext3 = mke2fs -j = mke2fs -t ext3
mkfs -t ext4 = mkfs. ext4 = mke2fs -t ext4
mkfs -t xfs = mkfs.xfs
mke2fs
-t
-L
-m
-b< br>-E
tune2fs
-l
e2label
mount
-o: used to specify mount options, commonly used mounts Options
ro: mount read-only
rw: read-write, the default is read-write
moatime: turn off update access time
auto: whether it can be mounted by “mount -a”
defaults: equivalent to rw, suid, dev, exec, auto, nouser, async, and relatime
sync: synchronous write
async: asynchronous write
dev: allow device files to be enabled
remount: Remount
loop: local loopback device
mount -o remount,other_options DEVICE
cdrom –>sr0
cdrom –> hdc
mount -o loop,other_options DEVICE MOUNT_POINT: Used to mount the loopback device
-n: When mounting the file system, do not update the /etc/mtab file
-r: equivalent to “”- o ro”, read-only mount
fuser /media/
fuser -km /media/
umount /media/
fuser [MOUNT_POINT]: View the Mount point process
fuser -km [MOUNT_POINT]: Forcefully kill the process that is accessing the secondary mount point
/etc/fstab
Mounted device:
device file, Volume label, uuid
Mount point:
File system type: ext2,ext3,…
Mount options: defaults means to use default options, multiple options Separate by comma
Dump frequency: dump
0: Never backup
1: Daily backup
2: Backup every other day
Self-check sequence:
0: No Detection
1: The first detection: Generally only the root file system is the first detection
2:
9:
mount -a: Mount /etc/fstab All support auto mount options File system
Linux swap partition: swap
It must be an independent partition, independent file system
How big is the swap partition?
Scientific computing server: swap memory is OK As large as possible, such as 2 times the physical memory
application server, the swap memory is as small as possible, such as only 1G
Physical memory is less than 2G, swap=memory*2
2G
How to check the memory size
free
-m: Convert to MB
-g: Converted to GB
shared: shared memory
buffer: buffer
cache: cache
Create swap partition:
mkswap DEVICE
Enable swap partition:
swapon DEVICE : Enable the specified swap partition
swapon -a: Enable all swap partitions
Disable the swap partition:
swapoff DEVICE
swapoff -a
Automatically enable the swap partition at boot, which can be defined in /etc/fstab
DEVICE swap swap defaults 0 0
Mount point: swap
File system: swap
SysV style network service script
# service network restart
# /etc/rc.d/init.d/network start
# /etc/init.d/network
vim service.sh
#!/bin/bash
#
srvname=`basename $0`
lockFile=””/var/subsys/$srvname”
if [$# -lt 1 ]; then
echo ” Usage:$srvname {start|stop|r estart|status}”
exit 3
fi
if [$1 ==’start’ ]; then
if [-e $lockFile ]; then
echo “$ srvname is running…”
else
touch $LockFile &> /dev/null
echo “Start $srvname successfully.”
fi
elif [$1 ==’stop’] ; then
if [-e $lockFile ]; then
rm -f $lockFile &> /dev/null
echo “Stopping $srvname finished.”
else
echo “$srvname is stopped.”
fi
elif [$1 =’restart’ ]; then
rm -f $lockFile $> /dev/null
touch $LockFile &> /dev/null
echo “Restarting $srvname finished.”
elif [$1 =’status’ ]; then
if [-e $lockFile ]; then
echo “$srvname is running.”
else
echo “$srvname is stopped.”
fi
else
echo “Usage:$srvname {start|stop|restart|status}”
exit 4
fi
< p>File system:
rootfs: Root file system
/var/log/messages
Leaf node: File
Mount:
Partition: mkfs, mke2fs
mkfs -t xfs
Advanced formatting: Create a file system
file system, manage files, and realize file access by name;
File system: Located in the kernel
System calls
open(),close,read,write,create,delete
The 0th sector of the disk, the front of the disk is
MBR : Master Boot Record to boot the operating system of the entire disk
446: bootloader
64: 16*4 file partition table
2: 5A bootable
Boot block: 1024bytes , The first two sectors, the operating system on which partition is the boot block to boot
index: Store in the index: size, permission, owner, group, timestamp, corresponding disk block, block pointer,
Small files use block pointers, large files use second-level pointers, and larger files use third-level pointers
Each entry is called inode (index node), and the stored data is called metadata: MetaData inode number< br>The entries have a number called inode. Each entry is 128 bytes.
/ The root is self-referenced.
In order to avoid the full table scan to find free disk blocks, use bitmap
block bitmap
inode bitmap
block
block group: block group
100w: 5w blocks, 20 groups;
block group 0 bytes after the 1024th byte are called super Block
Super Block; GDT; Block Bitmap; inodeBitmap; InodeTable; DataBlocks
Super Block is not included in every block group, but there are backups in some groups, generally 0, 1, 3,5,7,9
GDT: group description table
tune2fs -l
Display super block information
dumpe2fs: Output ext file system information
-h : You can also display superblock information
du: disk usage
-s: summary, display the sum of the sizes of all files in a specified path
-h: human-readable
df: disk free disk The amount of free space
-l: display only local file system
-h: human-readable
-P: display in posix compatible style
-i: display inode usage information instead of 1k block< /p>
Delete, move, copy
Link file:
Hard link: For example, two different paths to the same inode
ls -l: The second paragraph shows Number refers to the number of times the file has been hard-linkedFeatures:
Cannot be used across partitions;
Cannot create hard links to directories;
Delete the original file path, which does not affect the use of linked files.
All files point to the same inode, so they are the same A file
soft link, symbolic link, symblc link
ls -l: display the file of file type 1
Test whether the file is a symbolic link file in bash, use
-L or -h
Features:
Not limited by partition;
Can create directories;
Delete the original file, the link file will not be used
The link file and the source file point to different inodes, so, Not the same file
ln [-s] SRC_FILE LINK_FILE: with -s to create a symbolic link file, without -s to create a hard link file
-v: verbose: show the detailed process
p>
ext file system: extended attributes
lsattr
chattr +|-|= [options]
i: read only
a: only appends
file compression , Decompress, archive
Compression format: gz, bz2, xz, zip, Z
Compression algorithm: different algorithms, the compression ratio will be different
compress: FILENAME.Z
uncompress
gzip: .gz
gzip /PATH/TO/SOMEFILE: The original file will be deleted after the compression is complete
-d: Uncompress
-#: 1-9 , Specify the compression ratio, the default is 6:
gunzip: gunzip /PATH/TO/SOME_FILE.gz: The original file will be deleted after the decompression is complete
zcat /PATH/TO/SOMEFILE.gz: No decompression In the case of, check the content of the text file
bzip2: .bz2
Compression tool with a larger compression ratio than gzip
bzip2 /PATH/TO/SOMEFILE
-d
-#: 1-9, the default is 6
-k: Keep the original file when compressing
bunzip2 /path/to/somefile.bz2
xz: .xz< br>xz /PATH/TO/SOMEFILE
-d
-#: 1-9, the default is 6
-k: keep the original file when compressing
unxz
xzdec can also be solved Compress
xzcat
zip: A tool for both archiving and compression
zip FILENAME.zip FILE1 FILE2 …: Do not delete the original file after compression
unzip FILENAME.zip
tar: archive tool.tar
-c: create archive file
-f FILE.tar: manipulate archive file
-x: expand archive
–xattrs: keep it when archiving The extended attribute information of the file
-t: Do not expand the archive, check which files are archived
-C: Expand to the specified directory
tar -xf file.tar -C / tmp/files
xzcat: View content without decompression
-zcf: Archive and call gzip compression
-zxf: Call gzip to decompress and expand archive
-jcf: bzip2
-jxf :
-Jcf: xz
-Jxf :
cpio: archiving tool
IDE or SATA hard disk attribute management tool:
# hdparm
Supplementary command:
dd: convert and copy a file
copy: copy files
dd: copy 0 1 of the underlying code< br>dd if=/path/to/src_file source file
of=/path/to/dst_file target file
bs=1024 | 2k copy 1024 byte block size
count=2 copy two Blocks
dd if=/var/file of=/tmp/file
It is not necessary to copy, how many bytes can be copied
MBR backup
dd if=/ dev/sda of=/root/sda_mbr.backup bs=512 count=1 MBR backup
dd if=/sda_mbr.backup of=/dev/sda bs=512 count=1
< p>dd if=/dev/zero of=/root/testfile bs=4k count=1024
/dev/zero: 0 generator
/dev/null: bit bucket
Exercise: Write a script
From the keyboard Let the user enter several files, the script can archive these files and compress them into one file
read :
-p: PROMPT gives a prompt
#!/bin/bash
2 #
3 read -p “input three files” FILE1 FILE2 FILE3
4 read -p “Destination:” DEST
5 read -p “Compress[gzip|bzip2|xz]” COMP
6
7 case $COMP in
8 gzip)
9 tar -zcvf ${DEST}.tar.gz $FILE1 $FILE2 $FILE3 ;;
10 bzip2)
11 tar- jcvf ${DEST}.tar.bz2 $FILE1 $FILE2 $FILE3 ;;
12 xz)
13 tar -cf ${DEST}.tar $FILE1 $FILE2 $FILE3
14 xz ${DEST }.tar
;;
15 *)
16 echo “Unknown”
17 exit 9
;;
18 esac
while loop: suitable for Scenario with unknown number of loops
Syntax:
while CONDITION; do
statement
…
done
Judging whether the hadoop user is logged in every 5 seconds< br>#!/bin/bash
#
who | grep’hadoop’ &> /dev/null
RETVAL=$#
while [$RETVAL -ne 0 ]; do
sleep 5
who | grep’hadoop’ &> /dev/null
RETVAL=$#
done
echo “hadoop is logged in”
Write a script:
1) Show a menu to the user:
d|D) show disk usages.
m|M) show memory usages.
s|S) show swap usages .
*) quit.
2) When the user gives the option, the corresponding content will be displayed;
Extension:
When the user chooses to complete, after displaying the corresponding information, do not exit; instead, let the user select again and display the corresponding content again; except that the user uses quit;
#!/bin/bash
#
raid and mdadm
IDE: 133Mbps parallel
SATA: 300Mbps, 600Mbps, 6Gbps serial
USB 3.0: yes 480Mbps serial
SCSI: Small Computer System Interface
10K, 15K Parallel
SAS: 2.5 inches
RAID: Redund ant Arrays of Independent Disks
Level: It only means that the disk organization is different, there is no distinction between upper and lower;
0: Stripe performance improvement: read, write
no redundancy (redundancy capability)
Space utilization: nS
at least 2 disks
1: Mirror performance: write performance decline, Improved read performance
Redundancy capability
Space utilization: 1/2
At least 2 disks
4: Check code
5: Take turns to verify performance: Read, write improved
Redundancy capability is available
Space utilization rate: (n-1)/n
At least 3 disks
10 Performance: Both read and write are improved
Redundancy capability is available
Space Utilization rate 1/2
At least 4 disks
01 Performance: Read, write improved
Redundancy capability is available
Space utilization rate 1/2
At least 4 disks
50 Performance: Read, write improved
Redundancy capability is available
Space utilization (n-2)/n
At least 6 disks
jbod: Multiple disks connected into one disk
Performance Performance: No improvement
Redundancy: No
100% space utilization
At least 2 blocks
Hardware RAID
Software RAID
Logical RIAD:
/dev/md#
md: kernel module
mdadm: make any block device into RAID
mode command:
create mode
-C< br>Special options
-l: level
-n: number of devices
-a: automatically create device files for it
-c: CHUNK size 2^n, default is 64K
-x #: Specify the number of free disks
Management mode
–add, –remove, –fail
Monitor mode
-F
Growth mode
-G
Assembly mode
-A
View the detailed information of the RAID array
mdadm -D /dev/md#
— detail
RAID0
2G :
4: 512MB
2: 1G
mdadm -C /dev/,d0
RAID1< br>2G :
2: 2G
mdadm -C /dev/md0 -a yes -l 0 -n 2 /dev/sda{5,6}
cat /proc/mdstat
Disable the array
mdadm -S /dev/mad0
–stop
mdadm -A /dev/md1 /dev/sda7 /dev /sda9
Remove the wrong disk
mdadm /dev/md1 -r /dev/sda8
Assemble into a new disk
mdadm /dev/md1 -a /dev /sda9
watch: periodically execute the specified command, and display the result in full screen
-n #: specify the period length, the default unit is seconds, the default is 2
format: watch- n #’COMMAND’
Save the current RAID information to the configuration file for later assembly:
mdamd -D –scan> /etc/mdadm.conf
mdadm -S /etc/md0 disable
mdadm -A /etc/md0 enable
RAID5 :
2G: 3, 1G
Usually used RAID10
Speed
Availability
Mirror, mirror, checksum
MD: Multi
mdadm: user space tool, management tool
MD,DM
/dev/md#
meta device
DM: Device Mapper
Logical device
RAID,LVM2
DM: LVM2
Snapshot
Multipath
fdisk /dev/sda
Create two new disks, 8e: LVM format
partprobe /dev/sda
pv
pvcreate /dev/ sda{10,11} Create PE
pvs or pvdisplay display created PE
pvscan
vg
vgcreate, vgremove,
create vg : vgcreate myvg /dev/sda{10,11}
vgcreate VG_NAME /PATH/TO/PV
-s#: PE size, default is 4MB
vgcreate myvgs -s 8M /dev /sda{10,11}
Shrink vg, remove a pv
pvmove /dev/sda11 remove data on pv
vgreduce myvg /dev/sda11
Erase pv
pvremove /dev/sda11
Add pv
pvcreate /dev/sda11
vgextend myvg /dev/sda11
lv< br>lvcreate, lvremove, lvextend
lvcreate -n LV_NAME -L #G VG_NAME
lvcreate -L 50M -n testlv myvg
lvs, lvdisplay,
Format after creation
mke2fs -j /dev//myvg/testlv
Mount
mount /dev/myvg/testlv /mnt
ls /mnt displays a new Partition
Remove a logical volume
lvremove /dev/myvg/testlv
1. Extending the logical volume does not affect data:
lvextend
-L [+ ]# /PATH/TO/LV Physical Boundary
resize2fs -p /PATH/TO/LV Logical Boundary
2. Reducing the logical volume does not affect the data
Note: 1. Cannot be online Reduce, uninstall first
2.Ensure that the reduced space size can still store all the original data;
3.Before shrinking, you should force check the file to ensure that the file system is in a consistent state;
1, df -lh
2, e2fsck -f /dev/myvg/testlv
3, resize2fs /PATH/TO/LV 2G
lvreduce -L [-]# /PATH/TO/LV
3. Snapshot volume 1. The life cycle is the entire data duration. During this duration, the amount of data growth cannot exceed the snapshot volume Size; 2. The snapshot volume should be read-only; 3. It is in the same volume group as the original volume; lvcreate -sp r|wlvcreate -L # -n SLV_NAME -s -pr /PATH/TO/LV
< /p>