Installing HIVE on Mac

Download the binary file

$ wget http://apache.arvixe.com/hive/hive–1.1.0/apache-hive–1.1.0-bin.tar.gz
$ tar xvf apache-hive–1.1.0-bin.tar.gz

If you already have hadoop installed with Brew, for consistency move the directory into Cellar.

$ mv apache-hive–1.1.0-bin /usr/local/Cellar/hive–1.1.0
$ cd /usr/local/Cellar/hive–1.1.0
$ export HIVE_HOME=$(pwd)

$ echo $HIVE_HOME
> /usr/local/Cellar/hive–1.1.0

Create the HDFS directories

$ hdfs dfs -mkdir /user/hive/warehouse
$ hdfs dfs -mkdir -p /user/hive/warehouse
$ hdfs dfs -chmod g+w /tmp
$ hdfs dfs -chmod g+w /user/hive/warehouse

Create hive-site.xml and hcat.out

$ mkdir -p hcatalog/var/log
$ touch hcatalog/var/log/hcat.out
$ vim conf/hive-site.xml
<configuration>
 <property>
  <name>hive.exec.scratchdir</name>
  <value>/tmp/hive-${user.name}</value>
  <description>Scratch space for Hive jobs</description>
 </property>
</configuration>
 

Start HCatalog Services (Optional)

HCatalog UsingHCat
Good to be aware of the service, but not necessary.

$ $HIVE_HOME/hcatalog/sbin/hcat_server.sh start
$ $HIVE_HOME/hcatalog/sbin/webhcat_server.sh start

Run Hive

Add bin to path

$ export PATH=$(pwd)/bin:$PATH
$ echo $PATH
> /usr/local/Cellar/hive–1.1.0/bin:/opt/local/lib/mysql56/bin:/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

To make the PATH more permanent edit your ~/.profile

$ vim ~/.profile
$ export PATH=/usr/local/Cellar/hive–1.1.0/bin:$PATH
$ source ~/.profile

Run hive

$ hive

> readlink: illegal option – f
> usage: readlink [-n][file ]

Logging initialized using configuration in jar:file:/Users/marekbejda/Desktop/HIVETEST/apache-hive–1.1.0-bin/lib/hive-common–1.1.0.jar!/hive-log4j.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hadoop/2.6.0/libexec/share/hadoop/common/lib/slf4j-log4j12–1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/marekbejda/Desktop/HIVETEST/apache-hive–1.1.0-bin/lib/hive-jdbc–1.1.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

hive> SET mapred.job.tracker=local;

Examples

Creating tables

Directly from Hive Getting Started-Installation and Configuration

hive> CREATE TABLE pokes (foo INT, bar STRING);
> OK
> Time taken: 0.098 seconds
hive> CREATE TABLE invites (foo INT, bar STRING) PARTITIONED BY (ds STRING);
> OK
> Time taken: 0.077 seconds
hive> SHOW TABLES;
> OK
> invites
> pokes
> u_data
> Time taken: 0.093 seconds, Fetched: 3 row(s)

hive> SHOW TABLES '.*s';

> OK
> invites
> pokes
> Time taken: 0.034 seconds, Fetched: 2 row(s)

hive> DESCRIBE invites;

> OK

> foo int
> bar string
> ds string
>

> # Partition Information
> # col_name data_type comment
>
> ds string

> Time taken: 0.08 seconds, Fetched: 8 row(s)

Using datasets MovieLens User Ratings

Download the dataset from MovieLens

$ wget http://files.grouplens.org/datasets/movielens/ml–100k.zip
$ unzip ml–100k.zip

Run Hive and load the data

./hive
hive> CREATE TABLE u_data (
  userid INT,
  movieid INT,
  rating INT,
  unixtime STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE;

hive> LOAD DATA LOCAL INPATH '../ml–100k/u.data' OVERWRITE INTO TABLE u_data;

> Loading data to table default.u_data
> Table default.u_data stats: [numFiles=1, numRows=0, totalSize=1979173, rawDataSize=0]
> OK
> Time taken: 0.498 seconds

hive> SELECT COUNT(*) FROM u_data;

> Query ID = marekbejda_20150329143434_59ae5f17-bb06–4a6e-a029–4a03da1a4be0
> Total jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks determined at compile time:

> In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=number

> In order to limit the maximum number of reducers:
set hive.exec.reducers.max=number

> In order to set a constant number of reducers:
set mapreduce.job.reduces=number

> Job running in-process (local Hadoop)
> 2015–03–29 14:34:55,170 Stage–1 map = 100%, reduce = 100
> Ended Job = job_local1972666714_0001

> MapReduce Jobs Launched:
> Stage-Stage–1: HDFS Read: 3958346 HDFS Write: 3958346 SUCCESS
> Total MapReduce CPU Time Spent: 0 msec
> OK
> 100000
> Time taken: 2.228 seconds, Fetched: 1 row(s)

Errors

1.Error

ERROR] Terminal initialization failed; falling back to unsupported java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected at jline.TerminalFactory.create(TerminalFactory.java:101) at jline.TerminalFactory.get(TerminalFactory.java:158) at org.apache.hive.beeline.BeeLineOpts.(BeeLineOpts.java:73) at org.apache.hive.beeline.BeeLine.(BeeLine.java:117)

1. Solution

https://issues.apache.org/jira/browse/HIVE–8918

Remove “$HADOOP_HOME/share/hadoop/yarn/lib/jline–0.9.94.jar”

Additional references

https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration

http://apache.arvixe.com/hive/hive–1.1.0/

https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-InstallationandConfiguration

Advertisements

10 thoughts on “Installing HIVE on Mac”

  1. Thanks for the great tutorial! One thing that I had to correct on my machine was HIVE_HOME: it should point to `/usr/local/Cellar/hive–x.x.x/libexec` for hive to start

  2. This tutorial is really well done! One help request – when I do the “CREATE TABLE u_data…” step, it works fine except for the “STORED AS TEXTFILE” option – when I try that, I get an error. It looks somewhat like this:

    “Exception in thread “main” java.lang.AssertionError: Unknown token: [@-1,0:0=’TOK_FILEFORMAT_GENERIC’,,0:-1]” followed by a stack trace.

    Any ideas?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s