Hi David,
I tried similar config with 4 nodes:
> cas.ticket.registry.hazelcast.cluster.members=${HAZELCAST_CLUSTER_MEMBERS}
> cas.ticket.registry.hazelcast.cluster.asyncBackupCount=4
> cas.ticket.registry.hazelcast.cluster.backupCount=0
> cas.ticket.registry.hazelcast.cluster.port=5701
> cas.ticket.registry.hazelcast.cluster.portAutoIncrement=false
> cas.ticket.registry.hazelcast.cluster.instanceName=localhost
>
> cas.ticket.registry.hazelcast.cluster.publicAddress=${HAZELCAST_PUBLIC_ADDRESS}
> cas.ticket.registry.hazelcast.cluster.tcpipEnabled=true
> cas.ticket.registry.hazelcast.crypto.enabled=false
I see this output on each node i.e. hazelcast creates a cluster and sees
all nodes:
2020-02-06 21:20:49,235 INFO
> [com.hazelcast.internal.cluster.ClusterService] -
> <[ecdc-rant-affiliateidp-dev-1]:5701 [dev] [3.12.4]
> Members {size:4, ver:4} [
> Member [wcdc-rant-affiliateidp-dev-1]:5701 -
> a245c93b-beb0-4929-b831-e40a323cad8b
> Member [ecdc-rant-affiliateidp-dev-2]:5701 -
> bcbcd799-8cb8-4e5d-8802-5d95d4015ffd
> Member [wcdc-rant-affiliateidp-dev-2]:5701 -
> 9d3f52c9-1475-462e-844a-1b534efdca73
> Member [ecdc-rant-affiliateidp-dev-1]:5701 -
> e9f81f52-7a99-4428-a402-5a2f48cba838 this
> ]
> >
However tickets distribution doesn't work. Nodes 1, 2, 3 don't know about
session on Node 4.
I don't see any errors in the logs related to hazelcast but this one
appears time to time
> 2020-02-06 17:31:56,248 ERROR
> [org.apereo.cas.web.flow.executor.EncryptedTranscoder] - <Null input buffer>
> java.lang.IllegalArgumentException: Null input buffer
> at javax.crypto.Cipher.doFinal(Unknown Source) ~[?:?]
> at
> org.apereo.cas.util.cipher.BaseBinaryCipherExecutor.decode(BaseBinaryCipherExecutor.java:92)
>
> ~[cas-server-core-util-api-6.1.3.jar!/:6.1.3]
On Wednesday, February 5, 2020 at 9:28:43 PM UTC+2, David Curry wrote:
>
> Maksim,
>
> If you don't want to ever lose tickets, then you would want all nodes to
> back up all other nodes. So if you have 3 member nodes, you would want 2
> async backup nodes (asyncBackupCount) and also you'd probably want to
> disable the default sync backup (backupCount) node since it will block.
> Here are the settings we're running with in production (although this is
> CAS 5):
>
> cas.ticket.registry.hazelcast.cluster.members:
> cas01.newschool.edu,cas02.newschool.edu,cas03.newschool.edu,
> cas04.newschool.edu,cas05.newschool.edu
> cas.ticket.registry.hazelcast.cluster.asyncBackupCount: 4
> cas.ticket.registry.hazelcast.cluster.backupCount: 0
> cas.ticket.registry.hazelcast.cluster.port: 5701
> cas.ticket.registry.hazelcast.cluster.portAutoIncrement: false
> cas.ticket.registry.hazelcast.crypto.encryption.key:
> xxxIoXN6SBU5bF+iAVTKgw==
> cas.ticket.registry.hazelcast.crypto.signing.key:
>
> xxxmEbPGT_MXg0JWYLTe4oFaOaklocCqlY2VuHBdAHuh0V6-PdQxmgi4tTA3CZZos8TUbzg-L9nYHJpA5RqcvA
> cas.ticket.registry.hazelcast.crypto.enabled: true
>
> This works well for us behind an F5 load balancer; we do not use sticky
> sessions. We can (and do) reboot servers in the pool without anyone getting
> re-prompted to log in (just don't reboot them all at once).
>
> The crypto stuff (last three lines) is not needed for this to work, but
> you (arguably) might want it in production. You can leave it off while
> getting things to work and enable it later.
>
> One other thing -- did you remember to open 5701 in the firewall on all
> the servers?
>
> --
>
> DAVID A. CURRY, CISSP
> *DIRECTOR • INFORMATION SECURITY & PRIVACY*
> THE NEW SCHOOL • INFORMATION TECHNOLOGY
>
> 71 FIFTH AVE., 9TH FL., NEW YORK, NY 10003
> +1 646 909-4728 • [email protected] <javascript:>
>
>
> On Wed, Feb 5, 2020 at 1:40 PM Ray Bon <[email protected] <javascript:>>
> wrote:
>
>> Maksim,
>>
>> There is this config setting
>> cas.ticket.registry.hazelcast.cluster.members=
>> Add IPs of all members to the list.
>> https://apereo.github.io/cas/6.1.x/configuration/Configuration-Properties.html#hazelcast-ticket-registry
>>
>> and the link to common settings.
>>
>> There are some hazelcast loggers in log4j2.xml
>>
>> <AsyncLogger name="com.hazelcast"
>> level="${sys:hazelcast.log.level}" includeLocation="true" />
>>
>> Ray
>>
>> On Wed, 2020-02-05 at 09:16 -0800, 'Maksim Kopeyka' via CAS Community
>> wrote:
>>
>> Ray,
>>
>> I asked about CAS functionality to distribute tickets across nodes. I
>> need specific CAS functionality based on Hazelcast and seems to me this
>> functionality doesn't work as expected so I need to check it somehow. Maybe
>> with some debug logging.
>> I have a cluster with several nodes of CAS with hazelcast ticket registry
>> and I have a load balancer. If I turn off sticky sessions CAS asks me about
>> username/password every time. This cluster works in the same way without
>> hazelcast ticket registry. So seems to me this functionality doesn't work
>> and I don't see any errors in the logs.
>>
>> On Wednesday, February 5, 2020 at 7:03:55 PM UTC+2, rbon wrote:
>>
>> Maksim,
>>
>> I do not know if there is a stand alone client for accessing hazelcast
>> data. In the docs,
>> https://docs.hazelcast.org/docs/4.0/manual/html-single/index.html, there
>> is a section on clients and one on management. It looks like you would have
>> to create an application yourself but someone has probably done that
>> already.
>>
>> Ray
>>
>> On Wed, 2020-02-05 at 08:11 -0800, 'Maksim Kopeyka' via CAS Community
>> wrote:
>>
>> Hi Ray,
>>
>> Seems to me Hazelcast doesn't distribute data across all nodes because
>> each node doesn't have information about sessions on other nodes.
>> How to check data distribution?
>>
>> On Friday, January 31, 2020 at 11:02:05 PM UTC+2, rbon wrote:
>>
>> Maksim,
>>
>> Hazelcast is distributed but not replicated. Thus, when a server goes
>> down, the tickets on that server are lost. You have to relogin only if your
>> ticket was on that server.
>> Hazelcast has some mechanism of determining which node has which ticket.
>> It may also be possible to make hazelcast replicated but I have not tried.
>>
>> Ray
>>
>> P.S. you have two node3s in your config.
>>
>> On Fri, 2020-01-31 at 11:51 -0800, 'Maksim Kopeyka' via CAS Community
>> wrote:
>>
>> Hi Andy,
>>
>> Your example is very helpful. Thank you.
>> I see how hazelcast tickets registry works on my local env. I turned off
>> active container and another container continues to work with my active
>> session without relogin.
>>
>> However on my remote env. with CAS 6.1.3 it doesn't work in this way.
>> Load balancer ask me to re-login If I turn off active container.
>> Each node with hazelcast sees other nodes. I see such messages on all
>> nodes in case node1 is turned off:
>>
>> WARN [com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler] - <[node2]:5701
>> [dev] [3.12.4] Removing connection to endpoint [node1]:5701 Cause => java
>> .net.SocketException {Connection refused to address node1/xx.xx.xx.xx:
>> 5701}, Error-Count: 5>
>> WARN [com.hazelcast.internal.cluster.impl.MembershipManager] - <[node2]:
>> 5701 [dev] [3.12.4] Member [node1]:5701 -
>> b1fba639-dfff-4536-b5f4-a8681920594d
>> is suspected to be dead for reason: No connection>
>> WARN [com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler] - <[node2]:5701
>> [dev] [3.12.4] Removing connection to endpoint [node1]:5701 Cause => java
>> .net.SocketException {Connection refused to address node1/xx.xx.xx.xx:
>> 5701}, Error-Count: 6>
>> WARN [com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler] - <[node2]:5701
>> [dev] [3.12.4] Removing connection to endpoint [node1]:5701 Cause => java
>> .net.SocketException {Connection refused to address node1/xx.xx.xx.xx:
>> 5701}, Error-Count: 7>
>> WARN [com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler] - <[node2]:5701
>> [dev] [3.12.4] Removing connection to endpoint [node1]:5701 Cause => java
>> .net.SocketException {Connection refused to address node1/xx.xx.xx.xx:
>> 57001}, Error-Count: 8>
>>
>>
>>
>> This is my hazelcast settings:
>>
>> cas.ticket.registry.hazelcast.cluster.members=node1:5701,node2:5701,node3
>> :5701,node3:5701
>> cas.ticket.registry.hazelcast.cluster.asyncBackupCount=3
>> cas.ticket.registry.hazelcast.cluster.port=5701
>> cas.ticket.registry.hazelcast.cluster.portAutoIncrement=false
>> cas.ticket.registry.hazelcast.cluster.instanceName=localhost
>> cas.ticket.registry.hazelcast.cluster.publicAddress=node1:5701
>> cas.ticket.registry.hazelcast.cluster.tcpipEnabled=true
>>
>>
>> Why hazelcast doesn't share data across the cluster?
>>
>> These messages I see on startup
>>
>> WARN [com.hazelcast.instance.AddressPicker] - <[LOCAL] [dev] [3.12.4] You
>> configured your member address as host name. Please be aware of that
>> your dns can be spoofed. Make sure that your dns configurations are
>> correct.>
>> WARN [com.hazelcast.instance.AddressPicker] - <[LOCAL] [dev] [3.12.4] You
>> configured your member address as host name. Please be aware of that
>> your dns can be spoofed. Make sure that your dns configurations are
>> correct.>
>> WARN [com.hazelcast.instance.AddressPicker] - <[LOCAL] [dev] [3.12.4] You
>> configured your member address as host name. Please be aware of that
>> your dns can be spoofed. Make sure that your dns configurations are
>> correct.>
>> WARN [com.hazelcast.instance.AddressPicker] - <[LOCAL] [dev] [3.12.4] You
>> configured your member address as host name. Please be aware of that
>> your dns can be spoofed. Make sure that your dns configurations are
>> correct.>
>> WARN [com.hazelcast.instance.AddressPicker] - <[LOCAL] [dev] [3.12.4]
>> Could not find a matching address to start with! Picking one of non-loopback
>> addresses.>
>> INFO [org.apereo.cas.util.CoreTicketUtils] - <Ticket registry
>> encryption/signing
>> is turned off. This MAY NOT be safe in a clustered production environment
>> . Consider using other choices to handle encryption, signing and
>> verification of ticket registry tickets, and verify the chosen ticket
>> registry does support this behavior.>
>>
>>
>> On Wednesday, January 22, 2020 at 3:18:34 AM UTC+2, Andy Ng wrote:
>>
>> Hi Maksim,
>>
>> Pretty sure:
>> cas.ticket.registry.hazelcast.cluster.public-address
>> and
>> cas.ticket.registry.hazelcast.cluster.publicAddress
>>
>> Both works the same, since spring property allows both camelCase and
>> kebak-case.
>>
>>
>> And I did successfully use docker CAS and use Hazelcast as ticketing
>> system, however I am using it for demo so I just included a whole bunch of
>> private IP so it works......
>>
>> here's my CAS properties:
>>
>>
>> cas.ticket.registry.hazelcast.cluster.members=172.20.0.1,172.20.0.2,172.20.0.3,172.20.0.4,172.20.0.5,172.20.0.6,172.20.0.7,172.20.0.8,172.20.0.9,172.20.0.10
>> cas.ticket.registry.hazelcast.cluster.instanceName=localhost
>> my project link as well so you can reference if want to:
>> https://github.com/NgSekLong/SelectUrCAS/blob/master/source/ticket-registry/hazelcast/cas.yml
>>
>>
>> Cheers!
>> - Andy
>>
>> --
>>
>>
>> Ray Bon
>> Programmer Analyst
>> Development Services, University Systems
>> 2507218831 | CLE 019 | [email protected]
>>
>> I respectfully acknowledge that my place of work is located within the
>> ancestral, traditional and unceded territory of the Songhees, Esquimalt and
>> WSÁNEĆ Nations.
>>
>> --
>>
>>
>> Ray Bon
>> Programmer Analyst
>> Development Services, University Systems
>> 2507218831 | CLE 019 | [email protected]
>>
>> I respectfully acknowledge that my place of work is located within the
>> ancestral, traditional and unceded territory of the Songhees, Esquimalt and
>> WSÁNEĆ Nations.
>>
>> --
>>
>> Ray Bon
>> Programmer Analyst
>> Development Services, University Systems
>> 2507218831 | CLE 019 | [email protected] <javascript:>
>>
>> I respectfully acknowledge that my place of work is located within the
>> ancestral, traditional and unceded territory of the Songhees, Esquimalt and
>> WSÁNEĆ Nations.
>>
>> --
>> - Website: https://apereo.github.io/cas
>> - Gitter Chatroom: https://gitter.im/apereo/cas
>> - List Guidelines: https://goo.gl/1VRrw7
>> - Contributions: https://goo.gl/mh7qDG
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "CAS Community" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> To view this discussion on the web visit
>> https://groups.google.com/a/apereo.org/d/msgid/cas-user/cfe74ef4f3aeb0c1dfc506a7257e659418e41f0e.camel%40uvic.ca
>>
>> <https://groups.google.com/a/apereo.org/d/msgid/cas-user/cfe74ef4f3aeb0c1dfc506a7257e659418e41f0e.camel%40uvic.ca?utm_medium=email&utm_source=footer>
>> .
>>
>
--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS
Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/a/apereo.org/d/msgid/cas-user/8d82a0af-84cb-4e9e-96e9-7e1e688eba1d%40apereo.org.
