In case of an enabled partition-awareness feature or sessions I suppose that a state appears. And as we have a state we should use the stateful deployment of kubernetes with StatefulSet [1]. In that case addresses of nodes are predictable and aren't changed so one can configure a client with them. I think it's a better solution than implementing specific discovery by self. Point is to rely on k8s discovery mechanisms as much as possible otherwise why use it? What do you think?
I will prepare a demo with an enabled partition-awareness feature and configured StatefulSet. If you see any issues with it please let me know. [1] https://apacheignite.readme.io/docs/stateful-deployment On Thu, Aug 13, 2020 at 11:25 AM Pavel Tupitsyn <ptupit...@apache.org> wrote: > Max, Denis, > > Partition Awareness [1] is our main reason to have a specialized k8s > discovery mechanism in Thin Clients. > > k8s ingress mechanism is fine for a single client connection, but in > Partition Aware mode > thin clients need to connect to every server node, and track nodes as they > enter or leave the topology. > > Denis is right, current tickets [2] [3] are about Thin Clients being > deployed in the same k8s cluster as server nodes. > Using k8s APIs we can get every pod address and establish connections. > > [1] > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients > [2] https://issues.apache.org/jira/browse/IGNITE-13011 > <https://issues.apache.org/jira/browse/IGNITE-13204> > [3] https://issues.apache.org/jira/browse/IGNITE-13204 > > On Thu, Aug 13, 2020 at 10:48 AM Denis Magda <dma...@apache.org> wrote: > > > We're discussing the case when the thin client and server nodes are a > part > > of a single Kubernetes cluster. > > > > Ignite thick client uses KubernetesIPFinder > > <https://apacheignite.readme.io/docs/kubernetes-ip-finder> to > > auto-discover > > other Ignite pods. The IP finder gets IP addresses of other Ignite pods > via > > a special Service instance, and then the thick client uses those > addresses > > to communicate to the rest of the cluster bypassing the Service. I > > believe the ultimate goal of the JIRA task is to support a similar > > capability for the thin client. > > > > However, you solution might be the way to go. Still, I wonder how the > thin > > client will handle cases when a server pod, the client was connected to, > > goes down, and the Service connects the client to another pod > (considering > > the session affinity option). It might break some internal state of the > > connection on the client-side and lead to unpredictable behavior. Also, > if > > the partition-awareness feature is enabled, then the thin client will > start > > bypassing the Service for most of the key-value requests connecting to > the > > nodes directly using their IP addresses. > > > > - > > Denis > > > > > > On Wed, Aug 12, 2020 at 11:07 PM Max Timonin <timonin.ma...@gmail.com> > > wrote: > > > > > I'm not sure what you mean by inside/outside of kubernetes. Service > name > > is > > > visible within k8s environment. I've described a case with connection > > from > > > another pod that is part of k8s cluster. > > > > > > To provide connection outside of Kubernetes one should configure > Ingress > > > [1]. It will have a fixed address assigned by a cluster administrator. > > Any > > > request outside of k8s to the Ingress will be proxied inside k8s env to > > the > > > Service and then to a pod automatically without need to provide any IP > > > addresses in a client configuration. > > > > > > [1] https://kubernetes.io/docs/concepts/services-networking/ingress/ > > > > > > On Thu, Aug 13, 2020 at 1:39 AM Denis Magda <dma...@apache.org> wrote: > > > > > > > Max, > > > > > > > > That improvement automates the cluster discovery if both the cluster > > and > > > > thin clients are deployed *inside* of Kubernetes. That's my reading > of > > > the > > > > ticket's description. While you're referring to the case when the > > client > > > is > > > > deployed outside of a K8S environment. > > > > > > > > @Vladimir Pligin <vpli...@gridgain.com>, could you please join the > > > thread. > > > > I've seen you joined the discussion in JIRA. > > > > > > > > - > > > > Denis > > > > > > > > > > > > On Wed, Aug 12, 2020 at 12:06 PM Max Timonin < > timonin.ma...@gmail.com> > > > > wrote: > > > > > > > > > Hi, Igniters > > > > > > > > > > I'm now discovering the issue > > > > > https://issues.apache.org/jira/browse/IGNITE-13204 and wondering > > what > > > > is a > > > > > case that requires a client to be able to discover a cluster? > > > > > > > > > > I believe that discovery is a goal of kubernetes itself. We could > > > assign > > > > a > > > > > DNS name for a Service [1] and then any request to the Service will > > be > > > > > forwarded to any of Ignite pods behind it. So there is no need to > > > extract > > > > > pods IP to provide the connection. For example, I've configured a > > k8s > > > > > service with name "ignite" and port mapping 10800:9888. Then just > > > connect > > > > > to it with sqlline by a string "ignite:9888". > > > > > > > > > > I think that it's OK that clients should be configured with those > > > > settings > > > > > as they aren't changed during cluster live. > > > > > > > > > > Also k8s provides SessionAffinity [2] to guarantee forwarding > traffic > > > > from > > > > > one client to the same pod for every request. So there is no need > to > > > > > provide IP for a specific pod if a state is required. > > > > > > > > > > Maybe I'm missing something about this feature? Could somebody > > provide > > > > more > > > > > details for this task? > > > > > > > > > > Below is the example config and connection string: > > > > > > > > > > apiVersion: v1 > > > > > kind: Service > > > > > metadata: > > > > > name: ignite > > > > > spec: > > > > > ports: > > > > > - port: 9888 > > > > > targetPort: 10800 > > > > > selector: > > > > > app: ignite > > > > > > > > > > ./bin/sqlline.sh --verbose=true -u jdbc:ignite:thin://ignite:9888 > > > > > 0: jdbc:ignite:thin://ignite:9888> > > > > > > > > > > [1] > https://kubernetes.io/docs/concepts/services-networking/service/ > > > > > [2] > > > > > > > > > > > > > > > > > > > > https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-userspace > > > > > > > > > > > > > > >
