An introduction to zookeeper
Because you want to use kafka, But I don’t want to use the built-in Kafka, and considering that it needs to be used in other places, such as distributed jobs, I think it is better to be independent.
There are a lot of information on zookeeper at present, but one is that I need to exercise my ability to write blogs, and the other is that in case there are some things that others have not mentioned, it will surprise everyone, it is not beautiful.
Zookeeper official introduction: a service that provides distributed coordination for distributed applications.
Design Goals
Simple: zookeeper allows mutual coordination through a shared layer namespace. The organizational structure of this namespace is very similar to a standard File system. The namespace is composed of data items called znodes. In zookeeper, there are similarities with files and file directories. Unlike a typical file system, it is designed for storage. Zookeeper data is stored in memory, another meaning is that it can achieve high productivity. Zookeeper can be used in a large-scale distributed system, and it can keep running when a unit fails, and it realizes complex synchronization capabilities on the client.
ZooKeeper is replicated
Zookeeper replicates through a set of hosts, called ensemble. The servers supporting the zookeeper service must be mutually accessible. In a persistent storage, they keep an image state in memory, together with a transaction log and mirror.
The client connects to a single zookeeper service. The client maintains a tcp connection, sends requests through it, gets requests and watch events, and sends heartbeat packets. If a tcp connection is broken, the client will connect to another service.
ZooKeeper is ordered
Every imprint of zookeeper updates a number, reflecting the order of all zookeeper affairs. Subsequent operations can use this order to achieve high-level abstraction, just like synchronization.
ZooKeeper is fast (ZooKeeper is fast)
It is especially fast in the read-dominated load. The zookeeper application runs on thousands of machines, and when it works best, it usually reads more than writes, and the ratio is about 10:1
Data model and naming layer (Data model and the hierarchical namespace)
The namespace is much like an independent file system. A name is a sequence of path elements divided by slash (/). Each node is marked as a path in the zookeeper’s namespace to use
Nodes and ephemeral nodes (Nodes and ephemeral nodes)
Unlike a standard file system, each node can be associated with data in a zookeeper namespace, and this namespace also has child nodes. It is like a file system allowing a file to be a directory. (Zookeeper is designed to store coordination data: status information, configuration, location information, etc., so the stored data in each node is usually small, in the range of 1 byte to kb) We use term znode to let the zookeeper we are discussing The data node is clearer.
znodes maintains a state structure that contains the version number for each data change, ACL change, and timestamp to allow cache verification and coordinated updates. Every time the data of a znode changes, the version number increases automatically. For example, in an example, whenever a client retrieves data, it always accepts a version of the data.
The data stored in each node in a namespace is read and written atomically. Read all data associated with a znode, and replace all data with one write. Each node has an access control list (ACL) to restrict who can do that.
zookeeper also has the concept of a temporary node. These znodes exist as long as the session that created these znodes. If the session ends, the znode is deleted.
Coordinating updates and watches (Conditional updates and watches)
zookeeper supports the idea of watches. The client can set a watch on a znodes. When the znode is changed, a watch will be triggered and removed. A watch is fired, and the client receives a packet, saying that the znode has been changed. Then if the connection from a service to the client is broken, the client will receive a local notification.
Guarantees
Zookeeper is a very fast and very simple, since its goal, though, is to plan as a basic to a more complex service structure, as Synchronize. It implements a set of guarantees.
1 The timing is the same, and the update from a client is applied in a first-come-first-served order
2 atoms, update either succeeds or fails
3 single system image, a client will see the appearance of the same service, otherwise the client is connected to that service.
4 Reliable, once an update has been requested, he will continue to know that a client covers the update from that time forward.
5 Timely, the face of the client system is guaranteed to be up to date within a certain time limit.
Simple API
Create guarantee
delete Delete a node
exists Test Does a node exist in a location
get read data from a node
set write a data to a node
get children Retrieve the children of a node List of
sync waiting for data to be propagated
Implementation
The zookeeper component shows the higher level of the zookeeper service Element. In the request process, each server that supports the service copies each copy of its own components.
The replication database is an in-memory database. Contains the actual data tree. Updates are recorded on the hard disk in order to restore. Then writes are serialized to the hard disk before they are applied to the memory.
Each zookeeper server serves the client. The client connects to a server exactly to submit the request. Read requests are serviced locally from each server database. For requests to change the service status, write requests are processed by a contract agreement.
As part of the contract agreement, all write requests from the client are encouraged to write to a separate server named leader. The rest, called followers, accept message proposals and deliver messages from the leader. The message layer processing failure replaces the leader and synchronizes followers and leaders.
two zookeeper use
Download: http://mirrors.shu.edu.cn/apache/zookeeper/zookeeper-3.4.13/
wget http://mirrors.shu.edu.cn/apache/zookeeper/zookeeper-3.4.13/ zookeeper-3.4.13.tar.gz
Look at the directory:
First look at the configuration file conf/zoo.cfg:
tickTime=2000
dataDir=/var/lib/zookeeper
clientPort=2181
- tickTime < /dt>
- Subtle is the basic unit, it is used as heartbeats and the minimum session timeout, the supermarket will be twice ticktime.
- dataDir
- Store in-memory database mirroring, unless otherwise specified, update the database Transaction log
- clientPort
-
Listening port
We create a cfg file, and then add the above The configuration is written, and then you can start it now:
Then go to connect:
bin/zkCli.sh -server 127.0.0.1:2181
< /div>
Such a result proves that the connection is successful
Use help to query commands:
Command to play by yourself. This must be practiced. Next, let’s take a look at zookeeper replication.
Running Replicated ZooKeeper
Run zookeeper in a separate Mode is very convenient for development and testing, but in the production environment, you are running in a replication mode, a server replication group in the same application, called quorum, and then replication mode , All servers have the same configuration file in quorum.
Note: For replication mode, at least 3 servers are required, and it is strongly recommended that the number of servers be an odd number.
The following is the configuration of the copy mode:
tickTime=2000
dataDir=/var/lib/zookeeper
clientPort=2181 initLimit=5 syncLimit=2 server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888initLimit: the new entry is used for obsolescence, zookeeper is used to limit the quorum to connect to one The length of time of the leader.
syncLimit: Limit a service from a leader who is obsolete.
If you specify a tickTime, in the above example, the timeout is 10 seconds.
On server.x, when the service starts, it knows that the service is looking for the file myid in the data directory. That file contains the number of this service.
wget http://mirrors.shu.edu.cn/apache/zookeeper/zookeeper-3.4.13/zookeeper-3.4.13.tar.gz
tickTime=2000
dataDir=/var/lib/zookeeper
clientPort=2181bin/zkCli.sh -server 127.0.0.1:2181
tickTime=2000
dataDir=/var/lib/zookeeper
clientPort=2181 initLimit=5 syncLimit=2 server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888