[
https://issues.apache.org/jira/browse/HIVE-24563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shubham Chaurasia updated HIVE-24563:
-------------------------------------
Description:
Currently we open three tcp connections when llap external client communicates
to llap.
{noformat}
llap-ext-client ............... llap
connection1: client .......>>....... server
(RPC for submitting fragments - say t1, t2, t3. llap-ext-client initiates
connection)
connection2: client .......>>....... server
(for reading the output of t1, t2, t3. llap-ext-client initiates connection)
connection3: umbilical server .......<<....... client
(RPC for status updates/heartbeat of t1, t2, t3. llap Daemon initiates
connection)
{noformat}
connection3 starts a umbilical(RPC) server at the client side to which llap
daemon keeps sending the task statuses / heartbeats and node heartbeats.
*The Problem*
In cloud based deployment, we need to open tcp traffic.
1. For connection1 and connection2, we need to open incoming tcp traffic on the
machines running llap from client.
2. For connection3, we need to open incoming tcp traffic on the machines where
llap-ext-client is running, from llap daemon.
Here clients also need to worry about opening traffic(from llap) at their end.
*Possible Solution*
This jira is to evaluate the possibility of interchanging Umbilical server and
client sides i.e. umbilical server will run in llap only and llap-ext-client
will act as client and initiate the connection.
We can have umbilical address in llap splits (when get_splits is called by
external client) which the client can later connect to.
cc [~prasanth_j] [~harishjp]
was:
Currently we open three tcp connections when llap external client communicates
to llap.
{noformat}
llap-ext-client ............... llap
connection1: client .......>>....... server
(RPC for submitting fragments - say t1, t2, t3. llap-ext-client initiates
connection)
connection2: client .......>>....... server
(for reading the output of t1, t2, t3. llap-ext-client initiates connection)
connection3: umbilical server .......<<....... client
(RPC for status updates/heartbeat of t1, t2, t3. llap Daemon initiates
connection)
{noformat}
connection3 starts a umbilical(RPC) server at the client side to which llap
daemon keeps sending the task statuses / heartbeats and node heartbeats.
*The Problem*
In cloud based deployment, we need to open tcp traffic.
1. For connection1 and connection2, we need to open incoming tcp traffic on the
machines running llap from client.
2. For connection3, we need to open incoming tcp traffic on the machines where
llap-ext-client is running, from llap daemon.
Here clients also need to worry about opening traffic(from llap) at their end.
*Possible Solution*
This jira is to evaluate the possibility of interchanging Umbilical server and
client sides i.e. umbilical server will run in llap only and llap-ext-client
will act as client and initiate the connection.
We can have umbilical address in llap splits (when get_splits is called by
external client) which the client can later connect to.
cc [~prasanth_j]
> Check if we can interchange client and server sides for umbilical for
> external client flow
> ------------------------------------------------------------------------------------------
>
> Key: HIVE-24563
> URL: https://issues.apache.org/jira/browse/HIVE-24563
> Project: Hive
> Issue Type: Sub-task
> Components: Hive, llap
> Reporter: Shubham Chaurasia
> Assignee: Shubham Chaurasia
> Priority: Major
>
> Currently we open three tcp connections when llap external client
> communicates to llap.
> {noformat}
> llap-ext-client ............... llap
> connection1: client .......>>....... server
> (RPC for submitting fragments - say t1, t2, t3. llap-ext-client initiates
> connection)
> connection2: client .......>>....... server
> (for reading the output of t1, t2, t3. llap-ext-client initiates connection)
> connection3: umbilical server .......<<....... client
> (RPC for status updates/heartbeat of t1, t2, t3. llap Daemon initiates
> connection)
> {noformat}
> connection3 starts a umbilical(RPC) server at the client side to which llap
> daemon keeps sending the task statuses / heartbeats and node heartbeats.
> *The Problem*
> In cloud based deployment, we need to open tcp traffic.
> 1. For connection1 and connection2, we need to open incoming tcp traffic on
> the machines running llap from client.
> 2. For connection3, we need to open incoming tcp traffic on the machines
> where llap-ext-client is running, from llap daemon.
> Here clients also need to worry about opening traffic(from llap) at their
> end.
> *Possible Solution*
> This jira is to evaluate the possibility of interchanging Umbilical server
> and client sides i.e. umbilical server will run in llap only and
> llap-ext-client will act as client and initiate the connection.
> We can have umbilical address in llap splits (when get_splits is called by
> external client) which the client can later connect to.
> cc [~prasanth_j] [~harishjp]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
