« Statistical analysis of Oracle performance data using R | Main | Correlation probability in Oracle »
Wednesday
Feb022011

Using Toad with Hive in Amazon Elastic Map Reduce

The Toad for Cloud Databases eclipse client has support for Hive queries which makes it really easy for me to run queries against our test hadoop clusters.  It also supports Hive running on top of Amazon Elastic Map Reduce (EMR), but you do need to be aware that in EMR the default ports are different from what we have come to expect.

Firstly, if you have started an EMR cluster with Hive 0.5 support, then the Hive server will be running on port 10001, not port 10000.  The second difference is that the JobTracker is running on port 9100, rather than 50030.  So when attaching to EMR, you would set up your hive connection something like this:

1-02-2011 5-56-03 PM

Once you’ve done that, the Hive connection will show all the Hive tables and you can enter HQL queries in the SQL editor.  You can drag table and column names into the editor as well:

1-02-2011 5-57-29 PM

One of the simple, but really useful things about the hive client is that you can jump to the jobtracker web page while the HQL is running to see how it is going:

1-02-2011 5-59-19 PM

Here’s the resulting JobTracker console.  We can see the job running and – if we scroll to the right or maximize the window – we can see how the Map and reduce phases of the Hive job are progressing:

1-02-2011 8-11-07 PM

References (15)

References allow you to track sources for this article, as well as articles that were written in response to this article.
  • Response
    Since my first post on connecting to Amazon Elastic Map Reduce with TOAD, we’ve added quite a few
  • Response
    Since my first post on connecting to Amazon Elastic Map Reduce with TOAD, we’ve added quite a few
  • Response
    Response: titanfall
    Guy Harrison - Yet Another Database Blog - Using Toad with Hive in Amazon Elastic Map Reduce
  • Response
    Response: pizza hut coupon
    Guy Harrison - Yet Another Database Blog - Using Toad with Hive in Amazon Elastic Map Reduce
  • Response
    I've been looking for blogs where blogger covers penis enlargement or at least delves into the topic of penis enlargement with an independent and serious voice. What are the leading blogs of this type?.
  • Response
    I know anything you write is technically copyrighted, but I'm not sure how you could prove that it was your original material..
  • Response
    Is there a way to forward other people's blog posts to my site?
  • Response
    Response: medical insurance
    Most of the files in the Wordpress theme I have end in .php. I would like to be able to make the text changes on screen in the Wordpress theme, also, how do you add additional pages, etc?.
  • Response
    Guy Harrison - Yet Another Database Blog - Using Toad with Hive in Amazon Elastic Map Reduce
  • Response
    go here for greatest seo españa anywhere
  • Response
    Guy Harrison - Yet Another Database Blog - Using Toad with Hive in Amazon Elastic Map Reduce
  • Response
    Response: seovolución
    I hosted my new site from powweb, and did a little work on it, then I downloaded joomla and I am navagating through joomla trying to find how to upload my material from powweb? I tried to get into powweb website maker and it doesn't work... obviously because I downloaded joomla. ...
  • Response
    click here for the top cheap auto insurance for teenagers around
  • Response
    Response: search engines
    Guy Harrison - Yet Another Database Blog - Using Toad with Hive in Amazon Elastic Map Reduce
  • Response
    A computer starts somewhere. It starts with a bios and a CPU and a motherboard ect... When we code that computer to pump out a random number, what makes it pick the number it picks? Exactly the same computer sits beside it's identical and they both pick different 1st random numbers. ...

Reader Comments (3)

Nice post,
I am trying to use the toad for cloud, eclipse plugin in my own environment,

But I am facing a problem, once I got results for my query, I can't do anything with them,
I couldn't figure out, how to copy paste them to excel or export them to excel, or even an ordinary text file.

Maybe you know if I can?

February 23, 2011 | Unregistered CommenterGuy

Hi, Sorry for not noticing this comment immediately.

We are putting this feature into the product at the moment - you might have been in contact with the development team? Should have a patch for you in a few days

Regards,
Guy

March 7, 2011 | Registered CommenterGuy Harrison

It really is very valuable information given to us, come the December article in the Amazon Elastic Map Reduce (EMR), Hive, and TOAD and it pays to be aware of your blog!

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>