Until now we've seen how to create zNodes. But creation is not the single thing that Apache ZooKeeper does.
A virtual conference at the intersection of Data and AI. This is not a conference for the hype. Its real users talking about real experiences.
- 40+ speakers with the likes of Hannes from Duck DB, Sol Rashidi, Joe Reis, Sadie St. Lawrence, Ryan Wolf from nvidia, Rebecca from lidl
- 12th September 2024
- Three simultaneous tracks
- Panels, Lighting Talks, Keynotes, Booth crawls, Roundtables and Entertainment.
- Topics include (ingestion, finops for data, data for inference (feature platforms), data for ML observability
- 100% virtual and 100% free
👉 Register here
In this article we cover 3 other available operations: edit, removal and bulk. Each of them is presented in separated part inside which we can find quick theoretical presentation and JUnit test cases.
Edit in Apache ZooKeeper
Edit of created zNodes groups permissions and date changes. These operations can be made with setACL(String, List<ACL>, int) and setData(String, byte[], int ). As you can see, the last parameter of these methods is an integer. It represents the version of zNode which we want to change. This parameter is a little bit confusing. We could think that we are allowed to modify old versions of zNodes. However, it's not the case, according to the ZooKeeper Javadoc and one of further presented test cases. To avoid memorizing current version of given node, we can pass -1 which matches any node's versions.
The version value is incremented on every modification. It can be observed through org.apache.zookeeper.data.Stat class, storing information about given zNode. It defines 2 types of versions: one representing data changes and another one representing ACL changes. In the case of zNode having some children, the version of parent is not impacted by changes made on children nodes.
We can see now corresponding test cases:
@Test
public void should_correctly_update_znode_content() throws KeeperException, InterruptedException {
zooKeeper.create(ZNODE_CONTENT_UPDATE, "Home directory".getBytes(),
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
Stat initialStat = zooKeeper.exists(ZNODE_CONTENT_UPDATE, false);
zooKeeper.setData(ZNODE_CONTENT_UPDATE, "New home directory".getBytes(), ALL_VERSIONS);
Stat newStat = zooKeeper.exists(ZNODE_CONTENT_UPDATE, false);
assertChanges(initialStat, newStat);
// Children and ACL weren't changed in this test case, so they should be equal
assertThat(initialStat.getCversion()).isEqualTo(newStat.getCversion());
assertThat(initialStat.getAversion()).isEqualTo(newStat.getAversion());
}
@Test(expected = KeeperException.BadVersionException.class)
public void should_fail_on_updating_only_old_version() throws KeeperException, InterruptedException {
zooKeeper.create(ZNOODE_OLD_VERSION, "1".getBytes(),
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);
Stat initialStat = zooKeeper.exists(ZNOODE_OLD_VERSION, false);
// This update increases zNode version
zooKeeper.setData(ZNOODE_OLD_VERSION, "3".getBytes(), ALL_VERSIONS);
// And this one should fail since we try to update a version which is not current
zooKeeper.setData(ZNOODE_OLD_VERSION, "2".getBytes(), initialStat.getVersion());
}
@Test
public void should_correctly_update_children_node() throws KeeperException, InterruptedException {
// create parent
zooKeeper.create(ZNODE_CHILDREN_UPDATE, "Parent".getBytes(),
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
Stat initialParentStat = zooKeeper.exists(ZNODE_CHILDREN_UPDATE, false);
// create and update child
zooKeeper.create(ZNODE_CHILDREN_UPDATE_CHILD, "Child".getBytes(),
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
Stat initialChildStat = zooKeeper.exists(ZNODE_CHILDREN_UPDATE_CHILD, false);
zooKeeper.setData(ZNODE_CHILDREN_UPDATE_CHILD, "New child".getBytes(), ALL_VERSIONS);
// get new states after changes
Stat newParentStat = zooKeeper.exists(ZNODE_CHILDREN_UPDATE, false);
Stat newChildStat = zooKeeper.exists(ZNODE_CHILDREN_UPDATE_CHILD, false);
assertChanges(initialChildStat, newChildStat);
// Parent was not changed, so the version should be still the same
assertThat(initialParentStat.getVersion()).isEqualTo(newParentStat.getVersion());
assertThat(initialParentStat.getDataLength()).isEqualTo(newParentStat.getDataLength());
assertThat(initialParentStat.getMtime()).isEqualTo(newParentStat.getMtime());
assertThat(initialParentStat.getMzxid()).isEqualTo(newParentStat.getMzxid());
assertThat(initialParentStat.getCzxid()).isEqualTo(newParentStat.getCzxid());
assertThat(initialParentStat.getCtime()).isEqualTo(newParentStat.getCtime());
// Unlike in previous test, children were changed, so the children version should be incremented
assertThat(initialParentStat.getCversion()).isEqualTo(newParentStat.getCversion());
// Children and ACL weren't changed in this test case, so they should be equal
assertThat(initialParentStat.getAversion()).isEqualTo(newParentStat.getAversion());
}
@Test
public void should_correctly_detect_acl_changes() throws KeeperException, InterruptedException {
zooKeeper.create(ZNODE_ACL_UPDATE, "ACL".getBytes(),
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
Stat initialStat = zooKeeper.exists(ZNODE_ACL_UPDATE, false);
zooKeeper.setACL(ZNODE_ACL_UPDATE, ZooDefs.Ids.READ_ACL_UNSAFE, ALL_VERSIONS);
Stat newStat = zooKeeper.exists(ZNODE_ACL_UPDATE, false);
assertThat(initialStat.getVersion()).isEqualTo(newStat.getVersion());
assertThat(initialStat.getDataLength()).isEqualTo(newStat.getDataLength());
assertThat(initialStat.getMtime()).isEqualTo(newStat.getMtime());
assertThat(initialStat.getMzxid()).isEqualTo(newStat.getMzxid());
assertThat(initialStat.getCzxid()).isEqualTo(newStat.getCzxid());
assertThat(initialStat.getCtime()).isEqualTo(newStat.getCtime());
assertThat(initialStat.getCversion()).isEqualTo(newStat.getCversion());
// This time ACL was modified, so it should be reflected on this test
assertThat(initialStat.getAversion()).isNotEqualTo(newStat.getAversion());
}
private static void assertChanges(Stat oldStat, Stat newStat) {
// According to doc, version is bumped
assertThat(oldStat.getVersion()).isNotEqualTo(newStat.getVersion());
// New data should be defined too
assertThat(oldStat.getDataLength()).isNotEqualTo(newStat.getDataLength());
// Modification stats (last time and last id) should be different
assertThat(oldStat.getMtime()).isNotEqualTo(newStat.getMtime());
assertThat(oldStat.getMzxid()).isNotEqualTo(newStat.getMzxid());
// But creation date should remain the same
assertThat(oldStat.getCzxid()).isEqualTo(newStat.getCzxid());
assertThat(oldStat.getCtime()).isEqualTo(newStat.getCtime());
}
Removal in Apache ZooKeeper
zNodes removal is quite easy operation. It needs a single call to delete(String, int) ZooKeeper's method. Here also, the last parameter represents version to delete. As in the case of edition, we can't remove other version than the current one.
The removal is a little bit complicated when zNode has children nodes. The delete method is not recursive, so we should first delete all children and only at the end their parent. However, there is simpler solution to do it. Instead of using delete method, we can pass by ZKUtil.deleteRecursive(ZooKeeper, String) utilitary method where the second parameter represents parent directory to delete.
We can see these assumptions in following test cases:
@Test
public void should_correctly_remove_znode_by_specific_version() throws KeeperException, InterruptedException {
zooKeeper.create(HOME_TO_DELETE, "Home directory".getBytes(),
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
int version = zooKeeper.exists(HOME_TO_DELETE, false).getVersion();
zooKeeper.delete(HOME_TO_DELETE, version);
assertThat(zooKeeper.exists(HOME_TO_DELETE, false)).isNull();
}
@Test(expected = KeeperException.BadVersionException.class)
public void should_correctly_remove_znode_by_specific_version_which_is_not_current() throws KeeperException, InterruptedException {
String path = "/home_multi_versions";
zooKeeper.create(path, "Home directory".getBytes(),
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);
int version = zooKeeper.exists(path, false).getVersion();
zooKeeper.setData(path, new byte[0], ALL_VERSIONS);
zooKeeper.delete(path, version);
}
@Test(expected = KeeperException.NotEmptyException.class)
public void should_fail_on_removing_znode_with_children_when_children_are_not_deleted_first() throws KeeperException, InterruptedException {
createTree();
zooKeeper.delete(HOME_WITH_CHILDREN, ALL_VERSIONS);
}
@Test
public void should_correctly_remove_znode_when_children_are_removed_before() throws KeeperException, InterruptedException {
createTree();
zooKeeper.delete(HOME_WITH_CHILDREN_1, ALL_VERSIONS);
zooKeeper.delete(HOME_WITH_CHILDREN_2, ALL_VERSIONS);
zooKeeper.delete(HOME_WITH_CHILDREN, ALL_VERSIONS);
assertThat(zooKeeper.exists(HOME_WITH_CHILDREN, false)).isNull();
assertThat(zooKeeper.exists(HOME_WITH_CHILDREN_1, false)).isNull();
assertThat(zooKeeper.exists(HOME_WITH_CHILDREN_2, false)).isNull();
}
@Test
public void should_delete_recursively_with_zkutils() throws KeeperException, InterruptedException {
createTree();
ZKUtil.deleteRecursive(zooKeeper, HOME_WITH_CHILDREN);
assertThat(zooKeeper.exists(HOME_WITH_CHILDREN, false)).isNull();
assertThat(zooKeeper.exists(HOME_WITH_CHILDREN_1, false)).isNull();
assertThat(zooKeeper.exists(HOME_WITH_CHILDREN_2, false)).isNull();
}
private void createTree() throws KeeperException, InterruptedException {
zooKeeper.create(HOME_WITH_CHILDREN, "Root".getBytes(),
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
zooKeeper.create(HOME_WITH_CHILDREN_1, "1".getBytes(),
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
zooKeeper.create(HOME_WITH_CHILDREN_2, "2".getBytes(),
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
}
Bulk operations in Apache ZooKeeper
ZooKeeper class has also some methods to make multiple operations in a single call. This specific method is multi(Iterable<Op>). It uses an abstraction of operation represented by org.apache.zookeeper.Op class. This class contains some of factory methods which can be used to create Op instances corresponding to executed operation: create for zNode creation, delete for removal, setData for data definition and check for zNode check operation.
Bulk operations is all-or-nothing ones, ie. either all operations succeed or all fail. Operations results are represented by classes corresponding to operations categories: CreateResult for create, SetDataResult for data definiot, DeleteResult for delete and CheckResult for check. A special kind of object is ErrorResult which represents operation failure. The results are returned by multi() method.
Below you can find two cases showing bulk operations in Apache ZooKeeper:
@Test
public void should_correctly_execute_bulk_znodes_creation() throws KeeperException, InterruptedException {
zooKeeper.create(NODE_2, "Content".getBytes(),
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);
Op createNode1 = Op.create(NODE_1, "content".getBytes(),
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);
Op removeNode2 = Op.delete(NODE_2, ALL_VERSIONS);
/*
* {@code OpResult} is only an abstract class. It contains
* specific implementations for creation, delete,
* data setting or version checking operations.
*/
List<OpResult> results =
zooKeeper.multi(Lists.newArrayList(createNode1, removeNode2));
assertThat(results).hasSize(2);
assertThat(results).extracting("type").containsOnly(ZooDefs.OpCode.create, ZooDefs.OpCode.delete);
assertThat(results.stream().map(result -> result.getClass().getCanonicalName()).collect(Collectors.toList())).containsOnly(
"org.apache.zookeeper.OpResult.CreateResult", "org.apache.zookeeper.OpResult.DeleteResult"
);
assertThat(zooKeeper.exists(NODE_1, false)).isNotNull();
assertThat(zooKeeper.exists(NODE_2, false)).isNull();
}
@Test
public void should_execute_only_some_valid_operations() throws KeeperException, InterruptedException {
Op createNode1 = Op.create(NODE_1, "content".getBytes(),
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);
Op createNode2 = Op.create(NODE_1, "content".getBytes(),
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);
Op removeNode3 = Op.delete(NODE_2, ALL_VERSIONS);
try {
zooKeeper.multi(Lists.newArrayList(createNode1, createNode2, removeNode3));
fail("Should fail on executing bulk operation with 1 failure");
} catch (KeeperException.NodeExistsException exception) {
List<OpResult> results = exception.getResults();
assertThat(results).hasSize(3);
assertThat(results).extracting("type").containsOnly(ZooDefs.OpCode.error);
assertThat(results.stream().map(result -> result.getClass().getCanonicalName())
.collect(Collectors.toList())).containsOnly(
"org.apache.zookeeper.OpResult.ErrorResult"
);
}
assertThat(zooKeeper.exists(NODE_1, false)).isNull();
}
Edit, removal and bulk operations are 3 topics presented in this article oriented more in practice. Its first part shows two methods useful in zNode modification. The modification can apply on stored data or associated permission. In the part about delete we saw that delete is easy for file-looking zNode. In the case of parent-directory role, it's better to use utilitary deleteRecursive method. The last part illustrated the use of bulk operations through multi() method.
