Technical: Hadoop/Cloudera [CDH]/Hive v2 – Installation

Technical: Hadoop/Cloudera [CDH]/Hive v2 – Installation

Pre-requisites

There are quite a few pre-requisites that should be met prior to installing and running Hive.

They include:

Packages

The list of packages that comprise Hive Installation are:

Package Description
Hive base package that provides the complete language and runtime (required)
hive-metastore provides scripts for running the metastore as a standalone service (optional)
hive-server provides scripts for running the original HiveServer as a standalone service (optional)
hive-server2 provides scripts for running the new HiveServer2 as a standalone service (optional)

Installation – Package – Hive

 

Introduction

Let us review Hive’s base package.

Get Package Info

Use “yum info” to get a bit of information about the RPM.


Syntax:
      yum info <package-name>

Sample:
      yum info hive

Output:

Hadoop - Hive - package - hive -- info (v2)

Install Package – hive

Install Hive.


Syntax:
      sudo yum install <package-name>

Sample:
      sudo yum install hive

Output:

Hadoop - Hive - package - hive -- dependency

Dependencies

The Dependencies are listed

Package Description
hadoop-client Hadoop – Client
hadoop-mapreduce Hadoop – MapReduce (Version 1)
hadoop-yarn Hadoop – MapReduce (Version 2)

Install Log

The Installation Log.

Hadoop - Hive - package - hive -- Install - Log

Installation – Package – Hive-Metastore

Introduction

Install Hive Metastore.

Get Package Info

Use “yum info” to get a bit of information about the RPM.


Syntax:
      yum info <package-name>

Sample:
      yum info hive-metastore

Output:

Hadoop - Hive - package - hive-metastore -- info

Install Package – Hive-Metastore

Install Hive-MetaStore.


Syntax:
      sudo yum install <package-name>

Sample:
      sudo yum install hive-metastore

Output:

Hadoop - Hive - package - hive-metastore -- Install - Log

Installation – Package – Hive-Server2

Introduction

There are two versions of Hive Servers; Hive-Server and Hive-Server2.

Hive-Server2 is the latest and as there are no major reasons to shy from it, we will chose it over the earlier version (Hive-Server).

Get Package Info

Use “yum info” to get a bit of information about the RPM.


Syntax:
      yum info <package-name>

Sample:
      yum info hive-server2

Output:

Hadoop - Hive - package - hive-server2 -- info

Install Package – Hive-Server2

Install Hive Server 2.


Syntax:
      sudo yum install <package-name>

Sample:
      sudo yum install hive-server2

Output:

Hadoop - Hive - package - hive-server2 -- Install - Log

Configure – Hive MetaSource – Create Database & Schema

Introduction

The Database and Schema will be created in this section.

File Listing

There are a few SQL files that are bundled with Hive.  Here is a current file section:


Syntax:
      ls <folder> 

Sample:
      ls /usr/lib/hive/scripts/metastore/upgrade/mysql

Output:

Hadoop - Hive - SQL - Folder List

The files can broken down into sections:

  • ###-HIVE-####.mysql.sql ==> Each changes and bug fixes are packaged into a particular MYSQL module
  • hive-schema-<major>-<minor>-<delta>.mysql.sql ==> The accompanying SQL for each Hive version is packaged in whole
  • upgrade-<version-old>-to-<version-new> ==> Upgrade from one version to the next version

Which file to use

The file we will like to use is hive-schema-### files; as they cover a new install.

Create SQL Modules

 

Here are the steps.

Connect to database



Syntax:
         mysql -h <machine> -u <user> -p 

Sample:
         mysql -h localhost -u root -p 

Create database



Syntax:
         create database if not exists <database>;

Sample:
         create database if not exists metastore;

Output:

MySQL - Create Database

Change database context:



Syntax:
         use <database>;

Sample:
         use metastore;

Output:

MySQL - Change Database

Create Hive MetaStore database objects:



Syntax:
         source <filename>;

Sample:
         source /usr/lib/hive/scripts/metastore/upgrade/mysql/hive-schema-0.10.0.mysql.sql;

Create Hive User

Here are the steps.

Create Hive User



Syntax:
    create user <user>@machine identified by <password>;

Sample:

--there are so many variations of the create user sql statement
--here are some of them
    --create user (hive) and grant access when on DB Server
    create user 'hive'@'localhost' identified by 'sanda1e';

    --create user (hive) and grant access from specific host (rachel)
    create user 'hive'@'rachel' identified by 'sanda1e';

    --create user (hive) and grant access from hosts in specific domain
    create user 'hive'@<domain-name> identified by 'sanda1e'

    --create user and allow access from all hosts
    create user 'hive'@'%' identified by 'sanda1e'; 

Command:

Create User & Output.

Hadoop - Hive - User and Password

Example:

  • In the example above we created a new user that is usable from any hosts that resides on the specified domain
  • To determine a host’s domain and Full Qualified Domain Name (FQDN), connect to the host and issue the following commands “hostname –domain” and “hostname –fqdn”
  • The password for the created user (hive) is sanda1e

Review User List

Here are the steps.

Query system table (mysql.user) and make sure that user was indeed created.



Syntax:
    select user, host, password from mysql.user;

Sample:

    select user, host, password from mysql.user;

Command:

List Users.

MySQL -- mysql-user (v2)

Explanation:

  • We see that the hive user is indeed created and binded to a specific domain

 

Grant permission to Hive User

Grant permission to Hive User.



Syntax:

   REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'hive'@'metastorehost';

   GRANT SELECT,INSERT,UPDATE,DELETE,LOCK TABLES,EXECUTE 
       ON metastore.* TO 'hive'@'metastorehost';

   FLUSH PRIVILEGES;

Sample:

   REVOKE ALL PRIVILEGES, GRANT OPTION 
        FROM 'hive'@'%.<domain-name>';

   GRANT SELECT,INSERT,UPDATE,DELETE,LOCK TABLES,EXECUTE 
       ON metastore.* TO 'hive'@'%.<domain-name>';

   FLUSH PRIVILEGES;

Output:

Mysql - Hive - Grant permissions

Example:

  • Revoke all permissions from user
  • Grant DML permissions only

Review Permission Set

Review permission set.



Syntax:

  select User, Host, DB, Select_priv, Insert_priv, Delete_priv
             , Execute_Priv 
  from mysql.db;

Sample:

  Select User, Host, DB, Select_priv, Insert_priv, Delete_priv
             , Execute_priv 
  from   mysql.db;

 

Output:

Mysql - Hive -- Read permissions (v2)

Explanation:

  1. The record bearing the name “hive”.  This user is only valid when issued from a host in the domain specified in the Host column
  2. We have the privileges that we assigned

Validate User Access & permission set

Validate that user can connect to Mysql

Determine whether user can connect.  And, if so dig into connection ID.



Syntax:

  mysql -h <hostname> -u hive -p <password> -e <query>

Sample:

  mysql -h rachel -u hive -p "sanda1e" -e 'select current_user'

Output:

Mysql - Hive - Query -- Select curent_user

Explanation:

  • User hive is able to connect to host rachel and is being authenticated via its FQDN
  • The query ran is “select current_user”

Configure – Hive MetaSource – Configure MetaSource Service

Configuration File

File Name :- hive-site.xml

File Name (Full) :- /etc/hive/conf.dist/hive-site.xml

Configuration File – Deployment Host

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/4.2.0/CDH4-Installation-Guide/cdh4ig_topic_18_4.html

Though you can use the same hive-site.xml on all hosts (client, metastore, HiveServer), hive.metastore.uris is the only property that must be configured on all of them; the others are used only on the metastore host.

Configurable Elements

Configuration values – javax.jdo.option.ConnectionURL

There is a specific syntax for specifying the ConnectionURL.

Here is how it is defined for each Vendor\database:

Vendor Syntax
Apache Derby jdbc:derby:;databaseName=../build/test/junit_metastore_db;create=true
mysql jdbc:mysql://myhost/metastore
postgresql jdbc:postgresql://myhost/metastore
oracle jdbc:oracle:thin:@//myhost/xe

Configuration values – javax.jdo.option.ConnectionDriverName

Each Vendor provides Database applicable Java jar files; to gain connectivity to that database please specify the corresponding class name (that is defined specifically in the provided Jar file).

Vendor Value
Apache Derby org.apache.derby.jdbc.EmbeddedDriver
mysql com.mysql.jdbc.Driver
postgresql org.postgresql.Driver
oracle oracle.jdbc.OracleDriver

Configuration values – javax.jdo.option.ConnectionUsername

This is the username the Hive Service will use to connect to the database.  Conventionally, it will be hive.

For security conscious implementation consider changing.

Configuration values – javax.jdo.option.ConnectionPassword

This is the password for the defined username. 

Configuration values – datanucleus.autoCreateSchema

DataNucleus Community

http://www.datanucleus.org/products/accessplatform_2_2/rdbms/schema.html

DataNucleus can generate the schema for the developer as it encounters classes in the persistence process. Alternatively you can use the SchemaTool application before starting your application.

If you want to create the schema (tables+columns+constraints) during the persistence process, the property datanucleus.autoCreateSchema provides a way of telling DataNucleus to do this.

Configuration values – datanucleus.fixedDataStore

DataNucleus Community – Fixed Schema

http://www.datanucleus.org/products/accessplatform_2_2/jdo/datastore_control.html

Some datastores have a “schema” defining the structure of data stored within that datastore. In this case we don’t want to allow updates to the structure at all. You can set this when creating your PersistenceManagerFactory by setting the persistence property datanucleus.fixedDatastore to true .

Configuration values – datanucleus.hive.metastore.uris

This is the full URL to the metastore.

The syntax is thrift://<host>:<port>

Item Syntax Value
Protocol thrift thrift
Host host localhost
Port port 9083

Configuration values

Here are the pertinent values for us.

Item Syntax Value
javax.jdo.option.ConnectionURL jdbc:mysql://<host>/<database> jdbc:mysql://rachel/metastore
javax.jdo.option.ConnectionDriverName <jdbc-class-name> com.mysql.jdbc.Driver
javax.jdo.option.ConnectionUserName <hive-username> hive
javax.jdo.option.ConnectionPassword <hive-password> sanda1e
datanucleus.autoCreateSchema false or true false
datanucleus.fixedDatastore false or true false
hive.metastore.uris thrift://<host>:<port> thrift://localhost:9083

Configure – Hive -to- MetaSource – Avail *-connector-java.jar

Introduction

For Hive to be able to connect to the MetaSource, let us install and avail the Connector Jar files.

As our metastore is back-ended by mysql, we will install the mysql-connector-java.

Install mysql-connector-java

$ sudo yum install mysql-connector-java

Create OS File System  Soft Link

$ sudo ln -s /usr/share/java/mysql-connector-java.jar /usr/lib/hive/lib/mysql-connector-java.jar

Is Soft Link in place?

Query Hive Lib Folder and ensure that we have a connector (mySQL Connector) file symbolically linked to our original install location:

$ ls -ila /usr/lib/hive/lib/mysql-connector-java.jar

Explanation:

  • Issue ls query against soft link recipient folder

Output:

Linux - SymLinks - Hive - Lib file

HDFS – File System – Permissions

Introduction

File System

By default Hive stores the actual tables and data in HDFS under the /user/hive/warehouse folder.

Therefore, please grant all users and groups that will be using Hive access to that folder and its sub-folders.

Please keep in mind that this construct is directed at HDFS and OS File System.

Hive, itself, has built in support for more higher level permission sets.  We will use Hive tooling a bit later.

It is also important to grant HDFS Folder level permissions to the temp (/tmp) Folder.

Type Folder Port#
HDFS /user/hive/warehouse 1777
HDFS /tmp 1777

File System – HDFS – Create Folder ( /user/hive/warehouse)

  • Create HDFS Folder –> /user/hive/warehouse
sudo -u hdfs hadoop fs -mkdir /user/hive/warehouse

Explanation:

  • In the example above, we are assuming the identity of the HDFS owner and invoking hadoop fs module.  And, we specifically submitting a command (mkdir <folder>).  The folder we are creating is obviously /user/hive/warehouse.
  • Please keep in mind that is an HDFS Folder and not an OS Folder

Output:

Hadoop - HDFS -ls user--hive--warehouse

File System – Set HDFS Permissions ( /user/hive/warehouse)

  • Set HDFS Permissions –> /user/hive/warehouse
sudo -u hdfs hadoop fs -chmod -R 1777 /user/hive/warehouse

File System – Get HDFS Permissions ( /user/hive/warehouse)

  • Get HDFS Permissions –> /user/hive/warehouse
sudo -u hdfs hadoop fs -ls -d /user/hive/warehouse

Output:

Hadoop - HDFS -ls user--hive--warehouse (revised - v2)

File System – HDFS – Folder Permissions ( /tmp)

File System – HDFS – Set FS Permissions ( /tmp)

  • Set OS File Permissions –> /tmp
sudo -u hdfs hadoop fs -chmod -R 1777 /tmp

HDFS – Get FS Permissions ( /tmp)

  • Get HDFS Permissions –> /tmp
sudo -u hdfs hadoop fs -ls -d /tmp

Output:

Hadoop -- HDFS -- tmp

 

Services

Introduction

Here are Hive specific services; along with their Port#.

Service Name & Port#

Here are the pertinent values for us.

Service Service Name Port#
Hive MetaStore hive-metastore 9083
Hive Server (v2) hive-server2 10000

Start Command

Please start the services in the following order:

  • hive-server2
  • hive-metastore
sudo service hive-metastore start
sudo service hive-server2 start

Please refer and monitor the service specific log files for any errors and warnings.

Stop Command

Please stop the services in the following order:

  • hive-server2
  • hive-metastore
$ sudo service hive-server2 stop
$ sudo service hive-metastore stop

Log Files

Introduction

Here are the log files.

Here are the service specific log files.

Service Log File
Hive MetaStore /var/log/hive/hive-metastore.log
Hive Server (v2) /var/log/hive/hive-server2.log

Validate Install

Introduction

Here are some quick test that you can do to check connectivity between the components.

Hive Server -to- Meta Store Connectivity

As previously discussed, Hive stores schema data for each object in the metastore.

So one way of testing that the metastore is available and usable by Hive is to ask for all tables.

Launch Hive Client

Syntax:
        hive

Sample:
        hive

Ask for tables

Syntax:
        show tables;

Sample:
        show tables;

Output:

Hive -- Client -- Show Tables

Hive – Client Access Tools

Introduction

Hive is a database and so to test it out you will use any of the available database query tools that supports JDBC.

Why JDBC? Well, because it is a Java base tool.

Available tools

Here are some of the available tools:

Vendor\Repository Tool  URL
Sourceforge.net SQLLine http://sqlline.sourceforge.net/
Apache Hive http://archive.cloudera.com/redhat/cdh/unstable/RPMS/noarch/

Trouble Shooting

Trouble Shooting – Hive MetaStore- Server

Trouble Shooting – Hive MetaStore – Server – Configuration (/etc/hive/conf.dist/hive-site.xml) –>  hive.metastore.uris

Aforementioned, there is a lone configuration file and the name of that file is hive-site.xml.

If the hive-site.xml file is not fully configured, you will get the error message stated below:

Hive - MetaStore - Log - Error - MetaStore URL not configured

The error lines are:

  • org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:9083.
  • Exception in thread “main” org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:9083.
  • ERROR metastore.HiveMetaStore (HiveMetaStore.java:main(4150)) – Metastore Thrift Server threw an exception…

To remediate:

  • Please add the hive.metastore.uris entry.  And, specify the IP Address and FQDN for the thrift metastore
  • In earlier versions of Hive one can simply indicate locality (local or external metastore by simply employing the hive.metastore.local XML item). That is no longer the case and now the hive.metastore.uris is a required item
  • As we are in pseudo mode, we will simply use localhost and offer the default port# of 9083
<property>
  <name>hive.metastore.uris</name>
  <value>thrift://localhost:9083</value>
  <description>
IP address (or fully-qualified domain name) and port of the 
metastore host
</description>
</property>

Couple of other important items.

  • This error is encountered when starting the Hive MetaStore Service ( sudo service hive-metastore start)
  • As the hive-metastore service is left running, please correct the XML file and restart the service ( sudo service hive-metastore restart)

Trouble Shooting – Hive MetaStore – Server – Configuration (/etc/hive/conf.dist/hive-site.xml) –> hive user’s password

If the Hive’s user password is incorrect, you will get the error pasted below:

Hive - MetaStore - Log - Error - JDOFatalDataStoreException -- Access denied for user

The error lines are:

  • javax.jdo.JDOFatalDataStoreException: Access denied for user ‘hive’@’rachel.labDomain.org’ (using password: YES)
  • java.sql.SQLException: Access denied for user ‘hive’@’rachel.labDomain.org’ (using password: YES)
  • ERROR metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(4247)) – javax.jdo.JDOFatalDataStoreException: Access denied for user ‘hive’@’labDomain.org’ (using password: YES)
  • ERROR metastore.HiveMetaStore (HiveMetaStore.java:main(4150)) – Metastore Thrift Server threw an exception…javax.jdo.JDOFatalDataStoreException: Access denied for user ‘hive’@’rachel.labDomain.org’ (using password: YES)

To remediate:

  • Please check the hive user and password


<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>hive</value>
  <description>
  </description>
</property>

<property>
  <name>java.jdo.option.ConnectionPassword</name>
  <value>password</value>
  <description>
  </description>
</property>

Trouble Shooting – Hive Server2 – Server – Configuration (/etc/hive/conf.dist/hive-site.xml) –> unable to connect to hive metastore

Hive -- Server2 - Log (unable to connect to metastore)

  • hive.metastore (HiveMetaStoreClient.java:open(285)) – Failed to connect to the MetaStore Server…
  • service.CompositeService (CompositeService.java:start(74)) – Error starting services HiveServer2org.apache.hive.service.ServiceException: Unable to connect to MetaStore!
  • Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused

To remediate:

  • Check if Hive MetaStore is running
  • Determine what port is running under
  • Consider restarting service
  • Consider killing MetaStore process ID

Trouble Shooting – Hive MetaStore – Server – Missing DB Jar file

Depending on the DB Backend you have chosen – Remember it can be Derby, Mysql, PostgreSQL, or Oracle – you might experience connectivity issues.

The error lines are:

  • javax.jdo.JDOFatalDataStoreException: Access denied for user ‘hive’@’rachel.labDomain.org’ (using password: YES)
  • java.sql.SQLException: Access denied for user ‘hive’@’rachel.labDomain.org’ (using password: YES)
  • ERROR metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(4247)) – javax.jdo.JDOFatalDataStoreException: Access denied for user ‘hive’@’labDomain.org’ (using password: YES)
  • ERROR metastore.HiveMetaStore (HiveMetaStore.java:main(4150)) – Metastore Thrift Server threw an exception…javax.jdo.JDOFatalDataStoreException: Access denied for user ‘hive’@’rachel.labDomain.org’ (using password: YES)

To remediate:

  • Please check the hive user and password


<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>hive</value>
  <description>
  </description>
</property>

<property>
  <name>java.jdo.option.ConnectionPassword</name>
  <value>password</value>
  <description>
  </description>
</property>

Trouble Shooting – Hive Server2 – Server – Configuration (/etc/hive/conf.dist/hive-site.xml) –> unable to connect to hive metastore

Hive -- Server2 - Log (unable to connect to metastore)

  • hive.metastore (HiveMetaStoreClient.java:open(285)) – Failed to connect to the MetaStore Server…
  • service.CompositeService (CompositeService.java:start(74)) – Error starting services HiveServer2org.apache.hive.service.ServiceException: Unable to connect to MetaStore!
  • Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused

To remediate:

  • Check if Hive MetaStore is running
  • Determine what port is running under
  • Consider restarting service
  • Consider killing MetaStore process ID

Trouble Shooting – Hive MetaStore – Server – Missing JDBC Connector Jar files

Hive -- Server2 - Log (unable to connect to metastore)

  • javax.jdo.JDOFatalInternalException: Error creating transactional connection factory at org.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:425)
  • Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the “DBCP” plugin to create a ConnectionPool gave an error : The specified datastore driver (“com.mysql.jdbc.Driver”) was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
  • Caused by: org.datanucleus.store.rdbms.datasource.DatastoreDriverNotFoundException: The specified datastore driver (“com.mysql.jdbc.Driver”) was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
  • ERROR metastore.HiveMetaStore (HiveMetaStore.java:main(4150)) – Metastore Thrift Server threw an exception…javax.jdo.JDOFatalInternalException: Error creating transactional connection factoryat org.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:425)

To remediate:

  • Missing database connector file
  • In our case, we needed
    
    sudo ln -s /usr/share/java/mysql-connector-java.jar \
        /usr/lib/hive/lib/mysql-connector-java.jar
    

Trouble Shooting – Hive MetaStore – Server – Access Control Exception – Permission denied

Permission denied.

Screen Output:

Hive - Client - Permission denied

Text Output:



hive> create table xyzlogTable (dateC string);
FAILED: Error in metadata: MetaException(message:Got exception: org.apache.hadoop.security.AccessControlException Permission denied: user=hive, 
access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
 at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(
FSPermissionChecker.java:205)
 at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(
FSPermissionChecker.java:186)
	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:135)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4684)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:4655)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:2996)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:2960)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:2938)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:648)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:417)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44096)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1695)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1691)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1689)
)
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
hive> 

To remediate:

  • Review the File System & HDFS permissions

References

References – MetaStore Admin

References – HDFS – Shell

References – Files – Symbolic Links

References – Files – Unix Bash

7 thoughts on “Technical: Hadoop/Cloudera [CDH]/Hive v2 – Installation

  1. This procedure works fine for Hadoop Standalone mode + Hive. I am trying with Hadoop Pseudo distributed mode + Hive. Having issues, could u pls help ?

    13/09/07 10:37:23 ERROR exec.Task: FAILED: Error in metadata: MetaException(message:javax.jdo.JDOFatalDataStoreException: Communications link failure

    The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    at org.datanucleus.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:298)
    at org.datanucleus.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:601)
    at org.datanucleus.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:286)
    at org.datanucleus.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:182)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at javax.jdo.JDOHelper$16.run(JDOHelper.java:1958)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.jdo.JDOHelper.invoke(JDOHelper.java:1953)
    at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1159)
    at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803)
    at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:698)
    at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:262)
    at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:291)
    at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:224)
    at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:199)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
    at org.apache.hadoop.hive.metastore.RetryingRawStore.(RetryingRawStore.java:62)
    at org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:71)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:404)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:392)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database(HiveMetaStore.java:613)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
    at $Proxy9.get_database(Unknown Source)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:736)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)

    • Jose:

      I will love to help you. But, I am back working with MS SQL Server exclusively for the last couple of months and it will take a little while to contextually shift back to the Linux and Hadoop mind-space and be able to effectively make this top priority, reproduce your environment, and offer meaningful help & suggestions.

      Nevertheless, I am sorry and envious of you having the opportunity to learn and work on Hadoop.

      Stay blessed,

      Daniel

  2. Never Mind . I fixed the issue, It has todo with mysqld daemon which was configured to use FQDN instead of localhost. I reinitialized the database with FQDN instead of localhost which fixed the issue. Great documentation from your. Thanks a bunch !!!

  3. Hi
    Your documentation is so helpful,
    I have been facing this error
    org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:9083.
    Exception in thread “main” org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:9083.
    ERROR metastore.HiveMetaStore (HiveMetaStore.java:main(4150)) – Metastore Thrift Server threw an exception…

    Followed your solution for this, but cold not solve this exception…. your help is greatly helpfull.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s