« Statistical analysis of Oracle performance data using R | Main | Correlation probability in Oracle »
Wednesday
Feb022011

Using Toad with Hive in Amazon Elastic Map Reduce

The Toad for Cloud Databases eclipse client has support for Hive queries which makes it really easy for me to run queries against our test hadoop clusters.  It also supports Hive running on top of Amazon Elastic Map Reduce (EMR), but you do need to be aware that in EMR the default ports are different from what we have come to expect.

Firstly, if you have started an EMR cluster with Hive 0.5 support, then the Hive server will be running on port 10001, not port 10000.  The second difference is that the JobTracker is running on port 9100, rather than 50030.  So when attaching to EMR, you would set up your hive connection something like this:

1-02-2011 5-56-03 PM

Once you’ve done that, the Hive connection will show all the Hive tables and you can enter HQL queries in the SQL editor.  You can drag table and column names into the editor as well:

1-02-2011 5-57-29 PM

One of the simple, but really useful things about the hive client is that you can jump to the jobtracker web page while the HQL is running to see how it is going:

1-02-2011 5-59-19 PM

Here’s the resulting JobTracker console.  We can see the job running and – if we scroll to the right or maximize the window – we can see how the Map and reduce phases of the Hive job are progressing:

1-02-2011 8-11-07 PM

References (5)

References allow you to track sources for this article, as well as articles that were written in response to this article.
  • Response
    Since my first post on connecting to Amazon Elastic Map Reduce with TOAD, we’ve added quite a few
  • Response
    Since my first post on connecting to Amazon Elastic Map Reduce with TOAD, we’ve added quite a few
  • Response
    Response: Niagara therapy
    Guy Harrison - Yet Another Database Blog - Using Toad with Hive in Amazon Elastic Map Reduce
  • Response
    Guy Harrison - Yet Another Database Blog - Using Toad with Hive in Amazon Elastic Map Reduce
  • Response
    Guy Harrison - Yet Another Database Blog - Using Toad with Hive in Amazon Elastic Map Reduce

Reader Comments (3)

Nice post,
I am trying to use the toad for cloud, eclipse plugin in my own environment,

But I am facing a problem, once I got results for my query, I can't do anything with them,
I couldn't figure out, how to copy paste them to excel or export them to excel, or even an ordinary text file.

Maybe you know if I can?

February 23, 2011 | Unregistered CommenterGuy

Hi, Sorry for not noticing this comment immediately.

We are putting this feature into the product at the moment - you might have been in contact with the development team? Should have a patch for you in a few days

Regards,
Guy

March 7, 2011 | Registered CommenterGuy Harrison

It really is very valuable information given to us, come the December article in the Amazon Elastic Map Reduce (EMR), Hive, and TOAD and it pays to be aware of your blog!

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>