Technical: Hadoop – ZooKeeper – Client – Cloudera

Technical: Hadoop – ZooKeeper – Client (Cloudera)

Introduction

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_topic_21.html

ZooKeeper is a high-performance coordination service for distributed applications. It exposes common services — such as naming, configuration management, synchronization, and group services – in a simple interface so you don’t have to write them from scratch. You can use it off-the-shelf to implement consensus, group management, leader election, and presence protocols. And you can build on it for your own, specific needs.

What are we trying to do

Review the ZooKeeper client bundled with Cloudera Hadoop distribution.  The tool is known as ZooKeeper-Client.

Configuration Validation

Folder – /etc/init.d/*cloudera*


ls -la /etc/init.d/*cloudera*

Screen Shot:

Listing -- etc--init-d--cloudera

Service – Status all



sudo service --status-all 2> /dev/null | grep -i "cloudera"

Screen Shot:

Service -- status-all

ZooKeeper Client

ZooKeeper Client – Launch Shell

To launch Zookeeper, please issue:


zookeeper-client

Output:

zookeeper-client Launch

(press enter) to get a command entry point

ZooKeeper Client – Help

To get a listing of commands:


help

Output:

zookeeper-help

ZooKeeper Client – Quit

To close the shell, issue the Quit command.


quit

Output:

ZooKeeper -- Quit

ZooKeeper Client – Connect

To connect to another ZooKeeper, issue Open :



Syntax:
   connect <hostname>:<portNumber>

Sample:
   connect hadoopHR:2181

Output:

ZooKeeper - Client - ConnectTo

ZooKeeper Client – ls

ZooKeeper primary object are folders and files.  To get a list of folders and files, issue:

Syntax:

   ls <folder-name>
Sample:

   ls /
   ls /hbase
   ls /zookeeper

On a base Zookeeper install, the base folders are /hbase and /zookeeper.

Output:

ls /hbase

Zookeeper - Client -- ls hbase

 ls /zookeeper

Zookeeper - Client -- ls zookeeper

ZooKeeper Client – Create Folder

To create your own folders:

Syntax:

   create <folder-name> <Associated-ID>
Sample:

   create  /corporate corp 
   create  /corporate/HR  corpHR

Zookeeper - Client - Create Folders

ZooKeeper Client – Remove Folder

To remove your own folders:

Syntax:

   rmr <folder-name>
Sample:

   rmr  /corpSec8

ZooKeeper - Client - Remove Folder

ZooKeeper Client – getAcl

To get permissions issue the getAcl command.

To get permission set for folder /advert:

Syntax:

   getAcl <folder-name>
Sample:

   getAcl  /corporate/HR

Output:

Zookeeper - Client - getAcl

Explanation:

  • Scheme -> world
  • User -> anyone (default and only allowable user)
  • Permission –> crdwa

To get permission set for folder /advert:


Syntax:

   getAcl <folder-name>
Sample:

   getAcl  /advert

Output:

ZooKeeper - Client - getAcl - Digest Authentication (folder advert)

Explanation:

  • Scheme -> digest
  • User -> dadeniji
  • Password –> safetec
  • Permission –> crdw

ZooKeeper Client – setAcl

To set permissions issue the setAcl command.

We have included setAcl commands as simply an engineering exercise.  I will discourage employing them for the following reasons.

  • The folder can become totally inaccessible
  • They are indomitable

Indomitable:

  • They can not be removed  – There is no resetAcl API
  • They are not cumulative

ZooKeeper Client – setAcl – Scheme (Host)

To set permission for specific Hosts or hosts that are in same domain, use:

Everyone whose hosts name has the corp.com moniker:

Syntax:

   setAcl <folder-name> <host>:<domain-name>:<permission-set>

Sample:

   setAcl /advert host:corp.com:crwda

The host whose FQDN name is appServer1.corp.com:

Syntax:

   setAcl <folder-name> <host>:hostname:<permission-set>

Sample:

   setAcl /advert host:appServer1.corp.com:cdrwa

ZooKeeper Client – setAcl – Scheme (IP Address)

To assign all permissions to a specific IP Address {10.0.4.70}:

Syntax:

   setAcl   <folder-name> ip:<ipAddress>:<permisson-set>
Sample:

   setAcl  /corpSec7 ip:10.0.4.70:cdrwa

ZooKeeper - Client - setAcl - IPAddress (folder corpSec7)

To validate that things are good, issue getAcl:

To review the Permission set, use getAcl:

Syntax:

   getAcl   <folder-name>
Sample:

   getAcl  /corpSec7

Output:

ZooKeeper - Client - getAcl - IPAddress (folder corpSec7)

ZooKeeper Client – setAcl – Scheme (World)

Anyone

For the following use case scenario:

  • Folder -> /corporate
  • Authentication Provider -> world
  • User –> anyone (The only valid user is the “anyone” user)
  • Permission -> crwda
Syntax:

   setAcl <folder-name> <scheme>:<permisson-set>

Sample:

   create /advert /advert
   setAcl /advert world:anyone:crdwa

Output:

Zookeeper - Client - setAcl - world

 

ZooKeeper Client – setAcl – Digest Authentication

To allow anyone within our local network the ability to use the /corporate/HR folder, do the following:

For the following use case scenario:

  • Folder -> /corporate
  • Authentication Provider -> digest
  • User –> dadeniji
  • User password –> waTER
  • Permission -> crwda
Syntax:

   setAcl <folder-name>:<scheme>:<permisson-set>

Sample:

   setAcl /corporate digest:dadeniji:waTER:crdwa

Output:

Zookeeper - Client - setAcl - Digest Authentication

For the following use case scenario:

  • Folder -> /advert
  • Authentication Provider -> digest
  • User –> dadeniji
  • User password –> safetec
  • Permission -> crwd
Syntax:

   setAcl <folder-name>:<scheme>:<permisson-set>

Sample:

   setAcl /advert digest:dadeniji:safetec:crdw


Output
:

ZooKeeper - Client - setAcl - Digest Authentication (folder advert)

ZooKeeper Client – Stat

Get folder Stats:

Syntax:

   stat <folder-name>

Sample:

   stat  /hbase
   stat  /zookeeper

Output:

ZooKeeper - Client - Stat

Error Messages:

Error Message – NoAuthException


Exception in thread "main" org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /corpSec7
at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.setACL(ZooKeeper.java:1375)
at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:733)
at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:593)
at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:365)
	at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:323)
	at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)

Explanation:

  • One always has to be careful when setting permissions
  • And, it seems once they are set, it is difficult to change them

Logging:

Logging – Log File – Location

ZooKeeper log files are kept in /var/log/zookeeper

Logging – Log File – Name

The naming convention for log file is:

zookeeper-cmf-zookeeper1-SERVER-<FQDN>.log

References:

References – Installation

References – Getting Started

References – Programmer

References – Zookeeper & SASL

References – Mailing List

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s