Hello Carlos, Thanks for the advice. Have made the change but it still does not seem to be helping. I cleared all of my browsing data and tried again. This time the logs are a bit different: I log into the cas-dev successfully, the logs on that node show: 017-07-19 15:55:12,149 INFO [org.apereo.inspektr.audit.support.Slf4jLoggingAuditTrailManager] - <Audit trail record BEGIN ============================================================= WHO: f282c439 WHAT: Supplied credentials: [f282c439] ACTION: AUTHENTICATION_SUCCESS APPLICATION: CAS WHEN: Wed Jul 19 15:55:12 CDT 2017 CLIENT IP ADDRESS: 156.26.183.205 SERVER IP ADDRESS: unknown =============================================================
> 2017-07-19 15:55:12,149 INFO [org.apereo.inspektr.audit.support.Slf4jLoggingAuditTrailManager] - <Audit trail record BEGIN ============================================================= WHO: f282c439 WHAT: Supplied credentials: [f282c439] ACTION: AUTHENTICATION_SUCCESS APPLICATION: CAS WHEN: Wed Jul 19 15:55:12 CDT 2017 CLIENT IP ADDRESS: 156.26.183.205 SERVER IP ADDRESS: unknown ============================================================= > 2017-07-19 15:55:12,255 INFO [org.apereo.inspektr.audit.support.Slf4jLoggingAuditTrailManager] - <Audit trail record BEGIN ============================================================= WHO: f282c439 WHAT: TGT-**********************************************Vn7zvmKG6w-cas-dev.wichita.edu ACTION: TICKET_GRANTING_TICKET_CREATED APPLICATION: CAS WHEN: Wed Jul 19 15:55:12 CDT 2017 CLIENT IP ADDRESS: 156.26.183.205 SERVER IP ADDRESS: unknown ============================================================= > 2017-07-19 15:55:12,255 INFO [org.apereo.inspektr.audit.support.Slf4jLoggingAuditTrailManager] - <Audit trail record BEGIN ============================================================= WHO: f282c439 WHAT: TGT-**********************************************Vn7zvmKG6w-cas-dev.wichita.edu ACTION: TICKET_GRANTING_TICKET_CREATED APPLICATION: CAS WHEN: Wed Jul 19 15:55:12 CDT 2017 CLIENT IP ADDRESS: 156.26.183.205 SERVER IP ADDRESS: unknown ============================================================= > 2017-07-19 15:55:12,321 INFO [org.apereo.inspektr.audit.support.Slf4jLoggingAuditTrailManager] - <Audit trail record BEGIN ============================================================= WHO: f282c439 WHAT: ST-1-wacoVmE2DgaKj4WZqGmx-cas-dev.wichita.edu for https://cas-dev.wichita.edu/cas-management/manage.html ACTION: SERVICE_TICKET_CREATED APPLICATION: CAS WHEN: Wed Jul 19 15:55:12 CDT 2017 CLIENT IP ADDRESS: 156.26.183.205 SERVER IP ADDRESS: unknown ============================================================= > 2017-07-19 15:55:12,321 INFO [org.apereo.inspektr.audit.support.Slf4jLoggingAuditTrailManager] - <Audit trail record BEGIN ============================================================= WHO: f282c439 WHAT: ST-1-wacoVmE2DgaKj4WZqGmx-cas-dev.wichita.edu for https://cas-dev.wichita.edu/cas-management/manage.html ACTION: SERVICE_TICKET_CREATED APPLICATION: CAS WHEN: Wed Jul 19 15:55:12 CDT 2017 CLIENT IP ADDRESS: 156.26.183.205 SERVER IP ADDRESS: unknown Then I shut down access to the first node (shut down http) and refresh and here are the logs from the second node 2017-07-19 15:56:06,157 DEBUG [com.hazelcast.internal.partition.InternalPartitionService] - <[156.26.183.186]:5701 [dev] [3.8.1] Master version should be greater than ours! Local version: 543, Master version: 543 Master: [156.26.183.187]:5701> 2017-07-19 15:56:06,157 DEBUG [com.hazelcast.internal.partition.operation.PartitionStateOperation] - <[156.26.183.186]:5701 [dev] [3.8.1] Applied new partition state. Version: 543, caller: [156.26.183.187]:5701> 2017-07-19 15:56:06,162 DEBUG [com.hazelcast.internal.cluster.impl.ClusterHeartbeatManager] - <[156.26.183.186]:5701 [dev] [3.8.1] Received heartbeat from Member [156.26.183.187]:5701 - ea6ee5d9-f6f2-433e-b11a-93df21a99a05 (now: 2017-07-19 15:56:06.161, timestamp: 2017-07-19 15:56:06.162)> 2017-07-19 15:56:06,162 DEBUG [com.hazelcast.internal.cluster.ClusterService] - <[156.26.183.186]:5701 [dev] [3.8.1] Setting cluster time diff to 0ms.> 2017-07-19 15:56:10,970 INFO [org.apereo.inspektr.audit.support.Slf4jLoggingAuditTrailManager] - <Audit trail record BEGIN ============================================================= WHO: audit:unknown WHAT: [event=success,timestamp=Wed Jul 19 15:56:10 CDT 2017,source=RankedAuthenticationProviderWebflowEventResolver] ACTION: AUTHENTICATION_EVENT_TRIGGERED APPLICATION: CAS WHEN: Wed Jul 19 15:56:10 CDT 2017 CLIENT IP ADDRESS: 156.26.183.205 SERVER IP ADDRESS: unknown ============================================================= > 2017-07-19 15:56:10,970 INFO [org.apereo.inspektr.audit.support.Slf4jLoggingAuditTrailManager] - <Audit trail record BEGIN ============================================================= WHO: audit:unknown WHAT: [event=success,timestamp=Wed Jul 19 15:56:10 CDT 2017,source=RankedAuthenticationProviderWebflowEventResolver] ACTION: AUTHENTICATION_EVENT_TRIGGERED APPLICATION: CAS WHEN: Wed Jul 19 15:56:10 CDT 2017 CLIENT IP ADDRESS: 156.26.183.205 SERVER IP ADDRESS: unknown ============================================================= > 2017-07-19 15:56:11,162 DEBUG [com.hazelcast.internal.cluster.impl.ClusterHeartbeatManager] - <[156.26.183.186]:5701 [dev] [3.8.1] Received heartbeat from Member [156.26.183.187]:5701 - ea6ee5d9-f6f2-433e-b11a-93df21a99a05 (now: 2017-07-19 15:56:11.162, timestamp: 2017-07-19 15:56:11.162)> 2017-07-19 15:56:11,162 DEBUG [com.hazelcast.internal.cluster.ClusterService] - <[156.26.183.186]:5701 [dev] [3.8.1] Setting cluster time diff to 0ms.> 2017-07-19 15:56:15,466 DEBUG [com.hazelcast.internal.cluster.impl.ClusterHeartbeatManager] - <[156.26.183.186]:5701 [dev] [3.8.1] Sending MasterConfirmation to Member [156.26.183.187]:5701 - ea6ee5d9-f6f2-433e-b11a-93df21a99a05> 2017-07-19 15:56:16,162 DEBUG [com.hazelcast.internal.cluster.impl.ClusterHeartbeatManager] - <[156.26.183.186]:5701 [dev] [3.8.1] Received heartbeat from Member [156.26.183.187]:5701 - ea6ee5d9-f6f2-433e-b11a-93df21a99a05 (now: 2017-07-19 15:56:16.162, timestamp: 2017-07-19 15:56:16.162)> 2017-07-19 15:56:16,162 DEBUG [com.hazelcast.internal.cluster.ClusterService] - <[156.26.183.186]:5701 [dev] [3.8.1] Setting cluster time diff to 0ms.> I’m not sure if it’s normal for the WHO: to be audit:unknown. It makes sense because I’m already authenticated and I assume CAS would be checking for a ticket in hazelcast not requiring authenication, nonetheless I’m kicked to the login page on the second node. If any one has any ideas I’m all ears. Thanks, Erik Mallory Server Analyst Wichita State University 316.978.3502 From: <[email protected]> on behalf of Carlos Fernandez <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Wednesday, July 19, 2017 at 2:45 PM To: "[email protected]" <[email protected]> Subject: Re: [cas-user] Hazelcast is not replicating tickets. Hi, Erik, I had a similar problem using Hazelcast as the storage service for a Shibboleth installation. I found that the member list should only include *other* servers, not the local host. In your case, on appdev-523 should have cas.ticket.registry.hazelcast.cluster.members set to appdev-524.wichita.edu<http://appdev-524.wichita.edu>, and vice versa. Once I set it up that way Hazelcast stopped logging weird partition errors. Best regards, -- Carlos M. Fernández Enterprise Systems Manager Saint Joseph’s University Philadelphia PA 19131 T: +1 610 660 1501 On Wed, Jul 19, 2017 at 11:17 AM, Mallory, Erik <[email protected]<mailto:[email protected]>> wrote: Hello, I have configured two CAS 5.1 nodes and I am using a netscaler to load balance them. I was doing some resiliency testing by logging into the cas service, then shutting down the http service on node I am connected to, then refreshing the connection. When I do this I get rerouted to the cas login screen again. In the logs I can see no Errors or failures. Hazelcast logging is set to debug. I see heartbeats and communication between the two nodes. But the service is not behaving as I expect. Here is my configuration for hazelcast cas.ticket.registry.hazelcast.pageSize=500 cas.ticket.registry.hazelcast.mapName=tickets #cas.ticket.registry.hazelcast.configLocation=/data/cas/config/cas5/hazelcast.xml cas.ticket.registry.hazelcast.cluster.evictionPolicy=LRU cas.ticket.registry.hazelcast.cluster.maxNoHeartbeatSeconds=300 cas.ticket.registry.hazelcast.cluster.multicastEnabled=false cas.ticket.registry.hazelcast.cluster.evictionPercentage=10 cas.ticket.registry.hazelcast.cluster.tcpipEnabled=true cas.ticket.registry.hazelcast.cluster.members=appdev-523.wichita.edu<http://appdev-523.wichita.edu>,appdev-524.wichita.edu<http://appdev-524.wichita.edu> cas.ticket.registry.hazelcast.cluster.loggingType=slf4j cas.ticket.registry.hazelcast.cluster.instanceName=cas-dev cas.ticket.registry.hazelcast.cluster.port=5701 cas.ticket.registry.hazelcast.cluster.portAutoIncrement=true cas.ticket.registry.hazelcast.cluster.maxHeapSizePercentage=85 cas.ticket.registry.hazelcast.cluster.backupCount=1 cas.ticket.registry.hazelcast.cluster.asyncBackupCount=0 cas.ticket.registry.hazelcast.cluster.maxSizePolicy=USED_HEAP_PERCENTAGE cas.ticket.registry.hazelcast.cluster.timeout=5 The logs generally look like this. 3.wichita.edu<http://3.wichita.edu>]:5701 [dev] [3.8.1] Applied new partition state. Version: 543, caller: [appdev-524.wichita.edu<http://appdev-524.wichita.edu>]:5701> cas-2017-07-18-11-1.log:2017-07-18 11:12:54,087 DEBUG [com.hazelcast.internal.cluster.impl.ClusterHeartbeatManager] - <[appdev-523.wichita.edu<http://appdev-523.wichita.edu>]:5701 [dev] [3.8.1] Received heartbeat from Member [appdev-524.wichita.edu<http://appdev-524.wichita.edu>]:5701 - 5a3577aa-4567-4629-a162-92d2cab86bd2 (now: 2017-07-18 11:12:54.085, timestamp: 2017-07-18 11:12:54.085)> cas-2017-07-18-11-1.log:2017-07-18 11:12:54,087 DEBUG [com.hazelcast.internal.cluster.ClusterService] - <[appdev-523.wichita.edu<http://appdev-523.wichita.edu>]:5701 [dev] [3.8.1] Setting cluster time diff to -2ms.> cas-2017-07-18-11-1.log:2017-07-18 11:12:59,087 DEBUG [com.hazelcast.internal.cluster.impl.ClusterHeartbeatManager] - <[appdev-523.wichita.edu<http://appdev-523.wichita.edu>]:5701 [dev] [3.8.1] Received heartbeat from Member [appdev-524.wichita.edu<http://appdev-524.wichita.edu>]:5701 - 5a3577aa-4567-4629-a162-92d2cab86bd2 (now: 2017-07-18 11:12:59.085, timestamp: 2017-07-18 11:12:59.085)> cas-2017-07-18-11-1.log:2017-07-18 11:12:59,087 DEBUG [com.hazelcast.internal.cluster.ClusterService] - <[appdev-523.wichita.edu<http://appdev-523.wichita.edu>]:5701 [dev] [3.8.1] Setting cluster time diff to -2ms.> cas-2017-07-18-11-1.log:2017-07-18 11:13:04,087 DEBUG [com.hazelcast.internal.cluster.impl.ClusterHeartbeatManager] - <[appdev-523.wichita.edu<http://appdev-523.wichita.edu>]:5701 [dev] [3.8.1] Received heartbeat from Member [appdev-524.wichita.edu<http://appdev-524.wichita.edu>]:5701 - 5a3577aa-4567-4629-a162-92d2cab86bd2 (now: 2017-07-18 11:13:04.085, timestamp: 2017-07-18 11:13:04.085)> cas-2017-07-18-11-1.log:2017-07-18 11:13:04,087 DEBUG [com.hazelcast.internal.cluster.ClusterService] - <[appdev-523.wichita.edu<http://appdev-523.wichita.edu>]:5701 [dev] [3.8.1] Setting cluster time diff to -2ms.> cas-2017-07-18-11-1.log:2017-07-18 11:13:09,081 DEBUG [com.hazelcast.internal.partition.InternalPartitionService] - <[appdev-523.wichita.edu<http://appdev-523.wichita.edu>]:5701 [dev] [3.8.1] Master version should be greater than ours! Local version: 543, Master version: 543 Master: [appdev-524.wichita.edu<http://appdev-524.wichita.edu>]:5701> Any help you can give would be greatly appreciated. Thanks, Erik Mallory Server Analyst Wichita State University 316.978.3502 -- - CAS gitter chatroom: https://gitter.im/apereo/cas - CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html<http://spamburger.sju.edu/canit/urlproxy.php?_q=aHR0cHM6Ly9hcGVyZW8uZ2l0aHViLmlvL2Nhcy9NYWlsaW5nLUxpc3RzLmh0bWw%3D&_s=Y2Zlcm5hbmQ%3D&_c=77a03ca9> - CAS documentation website: https://apereo.github.io/cas<http://spamburger.sju.edu/canit/urlproxy.php?_q=aHR0cHM6Ly9hcGVyZW8uZ2l0aHViLmlvL2Nhcw%3D%3D&_s=Y2Zlcm5hbmQ%3D&_c=9d7960c0> - CAS project website: https://github.com/apereo/cas --- You received this message because you are subscribed to the Google Groups "CAS Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/24FACB15-FA65-42F0-9595-F4D1353B7C63%40wichita.edu<https://groups.google.com/a/apereo.org/d/msgid/cas-user/24FACB15-FA65-42F0-9595-F4D1353B7C63%40wichita.edu?utm_medium=email&utm_source=footer>. -- - CAS gitter chatroom: https://gitter.im/apereo/cas - CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html - CAS documentation website: https://apereo.github.io/cas - CAS project website: https://github.com/apereo/cas --- You received this message because you are subscribed to the Google Groups "CAS Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAE7KU842VecUe4Wuy2aStukkVeuxuMvTUsX8Om6K1%3DxvwwhdMg%40mail.gmail.com<https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAE7KU842VecUe4Wuy2aStukkVeuxuMvTUsX8Om6K1%3DxvwwhdMg%40mail.gmail.com?utm_medium=email&utm_source=footer>. -- - CAS gitter chatroom: https://gitter.im/apereo/cas - CAS mailing list guidelines: https://apereo.github.io/cas/Mailing-Lists.html - CAS documentation website: https://apereo.github.io/cas - CAS project website: https://github.com/apereo/cas --- You received this message because you are subscribed to the Google Groups "CAS Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/84A8B65D-A8BC-4320-B9E4-6432E625D695%40wichita.edu.
