Re: Decomposing nested Hive statements with views

2015-09-15 Thread Gopal Vijayaraghavan
> My goal is to create separate independent processing units (queries) >that can then be composed into larger queries. Effectively something like: Ah, modularization beyond a single statement. Views would be the right thing there as a permanent abstraction. I like CTEs during dev is because I ca

Re: Decomposing nested Hive statements with views

2015-09-15 Thread Elliot West
On 15 September 2015 at 00:09, Gopal Vijayaraghavan wrote: > CTE Thank you for the in depth reply Gopal. I've just had a quick try out of CTEs but can't see how they address my original problem of decomposing a query into separate independent units. It seems that the CTE definition (' with' cla

Re: Decomposing nested Hive statements with views

2015-09-14 Thread Gopal Vijayaraghavan
> We have many HQL scripts that select from nested sub-selects. In many >cases the nesting can be a few levels deep: ... > Such queries are difficult to test as they are complex monoliths. While >we can decompose and modularise them at the column level with UDFs and >macros, it is not as evident t

Decomposing nested Hive statements with views

2015-09-14 Thread Elliot West
Hello, We have many HQL scripts that select from nested sub-selects. In many cases the nesting can be a few levels deep: SELECT ... FROM ( SELECT ... FROM ( SELECT ... FROM ( SELECT ... FROM a WHERE ... ) A LEFT JOIN ( SELECT ... FROM b ) B ON (