Max, The point of Thin Client k8s discovery feature is to make k8s deployment simple - similar to KubernetesIPFinder for thick clients and servers.
- I don't think that enabling partition awareness is related to a StatefulSet in any way - Not everyone is able to / wants to define a StatefulSet - Even with a StatefulSet you'll have to manually provide a set of all server nodes IPs in the thin client config, which is extra upfront work and extra maintenance With k8s discovery I can set a single property and the whole thing just works, servers join the cluster, thin clients connect to them. This is especially important for beginners. Ignite should be easy to get started with. On Thu, Aug 13, 2020 at 11:35 AM Max Timonin <timonin.ma...@gmail.com> wrote: > In case of an enabled partition-awareness feature or sessions I suppose > that a state appears. And as we have a state we should use the stateful > deployment of kubernetes with StatefulSet [1]. In that case addresses of > nodes are predictable and aren't changed so one can configure a client with > them. I think it's a better solution than implementing specific discovery > by self. Point is to rely on k8s discovery mechanisms as much as possible > otherwise why use it? What do you think? > > I will prepare a demo with an enabled partition-awareness feature and > configured StatefulSet. If you see any issues with it please let me know. > > [1] https://apacheignite.readme.io/docs/stateful-deployment > > On Thu, Aug 13, 2020 at 11:25 AM Pavel Tupitsyn <ptupit...@apache.org> > wrote: > > > Max, Denis, > > > > Partition Awareness [1] is our main reason to have a specialized k8s > > discovery mechanism in Thin Clients. > > > > k8s ingress mechanism is fine for a single client connection, but in > > Partition Aware mode > > thin clients need to connect to every server node, and track nodes as > they > > enter or leave the topology. > > > > Denis is right, current tickets [2] [3] are about Thin Clients being > > deployed in the same k8s cluster as server nodes. > > Using k8s APIs we can get every pod address and establish connections. > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients > > [2] https://issues.apache.org/jira/browse/IGNITE-13011 > > <https://issues.apache.org/jira/browse/IGNITE-13204> > > [3] https://issues.apache.org/jira/browse/IGNITE-13204 > > > > On Thu, Aug 13, 2020 at 10:48 AM Denis Magda <dma...@apache.org> wrote: > > > > > We're discussing the case when the thin client and server nodes are a > > part > > > of a single Kubernetes cluster. > > > > > > Ignite thick client uses KubernetesIPFinder > > > <https://apacheignite.readme.io/docs/kubernetes-ip-finder> to > > > auto-discover > > > other Ignite pods. The IP finder gets IP addresses of other Ignite pods > > via > > > a special Service instance, and then the thick client uses those > > addresses > > > to communicate to the rest of the cluster bypassing the Service. I > > > believe the ultimate goal of the JIRA task is to support a similar > > > capability for the thin client. > > > > > > However, you solution might be the way to go. Still, I wonder how the > > thin > > > client will handle cases when a server pod, the client was connected > to, > > > goes down, and the Service connects the client to another pod > > (considering > > > the session affinity option). It might break some internal state of the > > > connection on the client-side and lead to unpredictable behavior. Also, > > if > > > the partition-awareness feature is enabled, then the thin client will > > start > > > bypassing the Service for most of the key-value requests connecting to > > the > > > nodes directly using their IP addresses. > > > > > > - > > > Denis > > > > > > > > > On Wed, Aug 12, 2020 at 11:07 PM Max Timonin <timonin.ma...@gmail.com> > > > wrote: > > > > > > > I'm not sure what you mean by inside/outside of kubernetes. Service > > name > > > is > > > > visible within k8s environment. I've described a case with connection > > > from > > > > another pod that is part of k8s cluster. > > > > > > > > To provide connection outside of Kubernetes one should configure > > Ingress > > > > [1]. It will have a fixed address assigned by a cluster > administrator. > > > Any > > > > request outside of k8s to the Ingress will be proxied inside k8s env > to > > > the > > > > Service and then to a pod automatically without need to provide any > IP > > > > addresses in a client configuration. > > > > > > > > [1] https://kubernetes.io/docs/concepts/services-networking/ingress/ > > > > > > > > On Thu, Aug 13, 2020 at 1:39 AM Denis Magda <dma...@apache.org> > wrote: > > > > > > > > > Max, > > > > > > > > > > That improvement automates the cluster discovery if both the > cluster > > > and > > > > > thin clients are deployed *inside* of Kubernetes. That's my reading > > of > > > > the > > > > > ticket's description. While you're referring to the case when the > > > client > > > > is > > > > > deployed outside of a K8S environment. > > > > > > > > > > @Vladimir Pligin <vpli...@gridgain.com>, could you please join the > > > > thread. > > > > > I've seen you joined the discussion in JIRA. > > > > > > > > > > - > > > > > Denis > > > > > > > > > > > > > > > On Wed, Aug 12, 2020 at 12:06 PM Max Timonin < > > timonin.ma...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi, Igniters > > > > > > > > > > > > I'm now discovering the issue > > > > > > https://issues.apache.org/jira/browse/IGNITE-13204 and wondering > > > what > > > > > is a > > > > > > case that requires a client to be able to discover a cluster? > > > > > > > > > > > > I believe that discovery is a goal of kubernetes itself. We could > > > > assign > > > > > a > > > > > > DNS name for a Service [1] and then any request to the Service > will > > > be > > > > > > forwarded to any of Ignite pods behind it. So there is no need to > > > > extract > > > > > > pods IP to provide the connection. For example, I've configured a > > > k8s > > > > > > service with name "ignite" and port mapping 10800:9888. Then just > > > > connect > > > > > > to it with sqlline by a string "ignite:9888". > > > > > > > > > > > > I think that it's OK that clients should be configured with those > > > > > settings > > > > > > as they aren't changed during cluster live. > > > > > > > > > > > > Also k8s provides SessionAffinity [2] to guarantee forwarding > > traffic > > > > > from > > > > > > one client to the same pod for every request. So there is no need > > to > > > > > > provide IP for a specific pod if a state is required. > > > > > > > > > > > > Maybe I'm missing something about this feature? Could somebody > > > provide > > > > > more > > > > > > details for this task? > > > > > > > > > > > > Below is the example config and connection string: > > > > > > > > > > > > apiVersion: v1 > > > > > > kind: Service > > > > > > metadata: > > > > > > name: ignite > > > > > > spec: > > > > > > ports: > > > > > > - port: 9888 > > > > > > targetPort: 10800 > > > > > > selector: > > > > > > app: ignite > > > > > > > > > > > > ./bin/sqlline.sh --verbose=true -u jdbc:ignite:thin://ignite:9888 > > > > > > 0: jdbc:ignite:thin://ignite:9888> > > > > > > > > > > > > [1] > > https://kubernetes.io/docs/concepts/services-networking/service/ > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-userspace > > > > > > > > > > > > > > > > > > > > >
