Re: [DISCUSSION] Spark Data Frame through Thin Client

IGNITE-10325 created. ср, 7 нояб. 2018 г., 11:42 Ray ray...@cisco.com: > From my past experience with Spark Data Frame API, the thick client > approach > leads to many usability problems. > > Ex. > > http://apache-ignite-users.70518.x6.nabble.com/Local-node-SEGMENTED-error-causing-node-goes-down-

Re: [DISCUSSION] Spark Data Frame through Thin Client

>From my past experience with Spark Data Frame API, the thick client approach leads to many usability problems. Ex. http://apache-ignite-users.70518.x6.nabble.com/Local-node-SEGMENTED-error-causing-node-goes-down-for-no-obvious-reason-td25061.html I think it makes a lot of sense to change to thin

Re: [DISCUSSION] Spark Data Frame through Thin Client

Hello, Valentin. > What I don't agree with is that replacing thick client with thin client is a > way to fix usability issues. I think it will fix some of them. > will potentially compromise the performance As I mentioned earlier, I want to provide easy way to play with integration. For maxim

Re: [DISCUSSION] Spark Data Frame through Thin Client

Nikolay, Please see my comments below. Actually, I haven't made most of the assumptions that you mentioned, and I generally agree with you. What I don't agree with is that replacing thick client with thin client is a way to fix usability issues. Thin client is not going to be issue-free either, bu

Re: [DISCUSSION] Spark Data Frame through Thin Client

Ignite doesn’t currently support Spark Structured Streaming: https://issues.apache.org/jira/browse/IGNITE-9357 There’s a working patch associated with it. Regards, Stephen > On 22 Oct 2018, at 10:43, Nikolay Izhikov wrote: > > Hello, Stephe

Re: [DISCUSSION] Spark Data Frame through Thin Client

Hello, Stephen. I suggest thin client deployment as a second option together with existing integration that use Client Node. > I’m thinking specifically about better support for Spark Streaming, where the > lack of continuous query support in thin clients removes a significant > optimisation

Re: [DISCUSSION] Spark Data Frame through Thin Client

Are you suggesting making the Thin Client deployment an option or as a replacement for the thick-client? If the latter, do we risk making future desirable changes more difficult (or impossible)? I’m thinking specifically about better support for Spark Streaming, where the lack of continuous que

Re: [DISCUSSION] Spark Data Frame through Thin Client

Valentin. Seems, You made several suggestions, which is not always true, from my point of view: 1. "We have access to Spark cluster installation to perform deployment steps" - this is not true in cloud or enterprise environment. 2. "Spark cluster is used only for Ignite integration". From what

Re: [DISCUSSION] Spark Data Frame through Thin Client

Guys, >From my experience, Ignite and Spark clusters typically run in the same environment, which makes client node a more preferable option. Mainly, because of performance. BTW, I doubt partition-awareness on thin client will help either, because in dataframes we only run SQL queries and I believ

Re: [DISCUSSION] Spark Data Frame through Thin Client

Hello Nikolay, Your proposal sounds reasonable. However, I would suggest us to wait while partition-awareness is supported for Java thin client first. With that feature, the client can connect to any node directly while presently all the communication goes through a proxy (a node the client is con