1. ?NameNode overview
a. NameNode is the core of HDFS.
b, NameNode is also called Master.
c. NameNode only stores HDFS metadata: the directory tree of all files in the file system, and tracks files in the entire cluster.
d, NameNode does not store actual data or data sets. The data itself is actually stored in DataNodes.
e. NameNode knows the block list and location of any given file in HDFS. Using this information, the NameNode knows how to build the file from the block.
f. NameNode does not persistently store the location information of the DataNode where each block in each file is located. This information will be reconstructed from the data node when the system is started.
g. The NameNode is very important to HDFS. When the NameNode is shut down, the HDFS/Hadoop cluster cannot be accessed.
h, NameNode is a single point of failure in the Hadoop cluster.
i. The machine where the NameNode is located is usually configured with a large amount of memory (RAM). 
2. ?DataNode overview
a. DataNode is responsible for storing actual data in HDFS.
b, DataNode is also called Slave.
c, NameNode and DataNode will maintain constant communication.
d. When the DataNode starts, it publishes itself to the NameNode and reports the list of blocks it is responsible for holding.
e. When a DataNode is shut down, it will not affect the availability of data or the cluster. The NameNode will arrange for the blocks managed by other DataNodes to perform copy replication.
f. The machine where the DataNode is located is usually configured with a large amount of hard disk space. Because the actual data is stored in the DataNode.
g, DataNode will send heartbeats to NameNode periodically (configured in the dfs.heartbeat.interval configuration item, the default is 3 seconds). If NameNode does not receive the heartbeat sent by DataNode for a long time, NameNode will consider this DataNode Invalidate.
The reporting interval of h, block takes the parameter dfs.blockreport.intervalMsec, if the parameter is not configured, the default is 6 hours.
?
3. ?HDFS working mechanism
NameNode is responsible for managing metadata of the entire file system; DataNode is responsible for managing the storage of specific file data blocks; Secondary NameNode assists NameNode in metadata backup.
The internal working mechanism of HDFS remains transparent to the client. Clients requesting access to HDFS are all made by applying to the NameNode.
————————————————
3.1. ?HDFS data writing process
Detailed step analysis:
1. The client initiates a file upload request, establishes communication with the NameNode through RPC, and the NameNode checks whether the target file already exists and whether the parent directory exists , Return whether it can be uploaded;
2, which DataNode server the client requests the first block to transmit to;
3, NameNode according to the number of backups and racks specified in the configuration file Perception principle performs file allocation and returns the addresses of available DataNodes such as: A, B, C;
Note: Hadoop is designed with data security and efficiency in mind, and data files are stored in three copies on HDFS by default , The storage strategy is one copy locally, one copy on another node in the same rack, and one copy on a node in a different rack.
4. The client requests one of the three DataNodes, A, to upload data (essentially an RPC call to establish a pipeline), A will continue to call B upon receiving the request, and then B will call C to connect the entire pipeline After the establishment is completed, return to the client level by level;
5. The client starts to upload the first block to A (first read the data from the disk and put it into a local memory cache), in the unit of packet (default 64K) , A receives a packet and passes it to B, and B passes it to C; each time A passes a packet, it puts a packet into a response queue and waits for a response.
6. The data is divided into packet data packets and transmitted sequentially on the pipeline. In the opposite direction of the pipeline, ack (command correct response) is sent one by one, and finally the first DataNode node A in the pipeline will pipeline ack is sent to the client;
7. After the transfer of a block is completed, the client again requests the NameNode to upload the second block to the server.
3.2. ?HDFS read data process
Detailed step analysis:
1. The Client initiates an RPC request to the NameNode to determine the location of the requested file block;
2 The NameNode will return part or all of the block list of the file as appropriate. For each block, the NameNode will return the DataNode address containing the copy of the block;
3. These returned DN addresses will be obtained according to the cluster topology. Calculate the distance between DataNode and client, and then sort by two rules: the closest to Client in the network topology is ranked first; the DN status reported by the timeout mechanism in the heartbeat mechanism is STALE, and such ranking is lower;
< p>4. The Client selects the DataNode with the highest order to read the block. If the client itself is a DataNode, then the data will be obtained directly from the local;
5. The essence of the bottom layer is to establish a Socket Stream (FSDataInputStream) , Repeatedly call the read method of the parent class DataInputStream until the data on this block is read;
6. After reading the block of the list, if the file reading has not ended, the client will continue Get the next block list from the NameNode;
7. After reading a block, checksum verification will be performed. If an error occurs when reading the DataNode, the client will notify the NameNode, and then take the next one. The DataNode of the block copy continues to read.
8. The read method is to read block information in parallel, not one block at a time; NameNode only returns the DataNode address of the block containing the client request, not the data of the requested block;
< p>9. In the final read, all blocks will be merged into a complete final file.