I made a few changes and did some more testing this evening. 1. I changed the /etc/ipsec.conf to bring it in line with the Greenbow default transforms that Hans-Joerg recommened.
# cat /etc/ipsec.conf ike dynamic esp tunnel from any to 192.168.1.0/24 \ main auth hmac-sha1 enc 3des group modp1024 \ quick auth hmac-sha1 enc 3des \ psk abc123 2. I created the basic polciy file: # cat /etc/isakmpd/isakmpd.policy KeyNote-Version: 2 Authorizer: "POLICY" 3. Being lazy I rebooted the server and tried starting isakmpd manually without the "-K". It would not start. When I tried starting it with "-dLv" I got the message: 180252.969043 Default check_file_secrecy_fd: not loading /etc/isakmpd/isakmpd.policy - too open permissions 180252.970281 Default policy_init: cannot read /etc/isakmpd/isakmpd.policy: Operation not permitted So I went back and started it with "-K". 4. I then turned on packet tracing as Stuart suggested, tried logging in, turned packet tracing off and ran tcpdump on the file: # echo "p on" > /var/run/isakmpd.fifo # echo "p off" > /var/run/isakmpd.fifo # tcpdump -r /var/run/isakmpd.pcap -vvn tcpdump: WARNING: snaplen raised from 96 to 65536 18:08:57.938430 64.119.40.170.500 > 64.119.37.74.500: [udp sum ok] isakmp v1.0 exchange ID_PROT cookie: ed67c89ed96545fb->0000000000000000 msgid: 00000000 len: 160 payload: SA len: 52 DOI: 1(IPSEC) situation: IDENTITY_ONLY payload: PROPOSAL len: 40 proposal: 1 proto: ISAKMP spisz: 0 xforms: 1 payload: TRANSFORM len: 32 transform: 0 ID: ISAKMP attribute ENCRYPTION_ALGORITHM = 3DES_CBC attribute HASH_ALGORITHM = SHA attribute AUTHENTICATION_METHOD = PRE_SHARED attribute GROUP_DESCRIPTION = MODP_1024 attribute LIFE_TYPE = SECONDS attribute LIFE_DURATION = 3600 payload: VENDOR len: 20 (supports v1 NAT-T, draft-ietf-ipsec-nat-t-ike-00) payload: VENDOR len: 20 (supports v2 NAT-T, draft-ietf-ipsec-nat-t-ike-02) payload: VENDOR len: 20 (supports v3 NAT-T, draft-ietf-ipsec-nat-t-ike-03) payload: VENDOR len: 20 (supports DPD v1.0) [ttl 0] (id 1, len 188) 18:08:57.944015 64.119.37.74.500 > 64.119.40.170.500: [udp sum ok] isakmp v1.0 exchange INFO cookie: cfef30980a709fe2->0000000000000000 msgid: 00000000 len: 40 payload: NOTIFICATION len: 12 notification: NO PROPOSAL CHOSEN [ttl 0] (id 1, len 68) 5. OK, no good. Nothing jumped out at me in the tcpdump so I changed from dynamic to passive, and tried again: # cat /etc/ipsec.conf ike passive esp tunnel from any to 192.168.1.0/24 \ main auth hmac-sha1 enc 3des group modp1024 \ quick auth hmac-sha1 enc 3des \ psk abc123 # ipsecctl -f /etc/ipsec.conf killed the isakmpd daemon and restarted it with -K", turned packet tracing back on and tried everything again. Got more detail but nothing jumps out at me. # tcpdump -r /var/run/isakmpd.pcap -vvn tcpdump: WARNING: snaplen raised from 96 to 65536 18:08:57.938430 64.119.40.170.500 > 64.119.37.74.500: [udp sum ok] isakmp v1.0 exchange ID_PROT cookie: ed67c89ed96545fb->0000000000000000 msgid: 00000000 len: 160 payload: SA len: 52 DOI: 1(IPSEC) situation: IDENTITY_ONLY payload: PROPOSAL len: 40 proposal: 1 proto: ISAKMP spisz: 0 xforms: 1 payload: TRANSFORM len: 32 transform: 0 ID: ISAKMP attribute ENCRYPTION_ALGORITHM = 3DES_CBC attribute HASH_ALGORITHM = SHA attribute AUTHENTICATION_METHOD = PRE_SHARED attribute GROUP_DESCRIPTION = MODP_1024 attribute LIFE_TYPE = SECONDS attribute LIFE_DURATION = 3600 payload: VENDOR len: 20 (supports v1 NAT-T, draft-ietf-ipsec-nat-t-ike-00) payload: VENDOR len: 20 (supports v2 NAT-T, draft-ietf-ipsec-nat-t-ike-02) payload: VENDOR len: 20 (supports v3 NAT-T, draft-ietf-ipsec-nat-t-ike-03) payload: VENDOR len: 20 (supports DPD v1.0) [ttl 0] (id 1, len 188) 18:08:57.944015 64.119.37.74.500 > 64.119.40.170.500: [udp sum ok] isakmp v1.0 exchange INFO cookie: cfef30980a709fe2->0000000000000000 msgid: 00000000 len: 40 payload: NOTIFICATION len: 12 notification: NO PROPOSAL CHOSEN [ttl 0] (id 1, len 68) 18:24:12.441476 64.119.40.170.500 > 64.119.37.74.500: [udp sum ok] isakmp v1.0 exchange ID_PROT cookie: 7c923ecb8d9a90f0->0000000000000000 msgid: 00000000 len: 160 payload: SA len: 52 DOI: 1(IPSEC) situation: IDENTITY_ONLY payload: PROPOSAL len: 40 proposal: 1 proto: ISAKMP spisz: 0 xforms: 1 payload: TRANSFORM len: 32 transform: 0 ID: ISAKMP attribute ENCRYPTION_ALGORITHM = 3DES_CBC attribute HASH_ALGORITHM = SHA attribute AUTHENTICATION_METHOD = PRE_SHARED attribute GROUP_DESCRIPTION = MODP_1024 attribute LIFE_TYPE = SECONDS attribute LIFE_DURATION = 3600 payload: VENDOR len: 20 (supports v1 NAT-T, draft-ietf-ipsec-nat-t-ike-00) payload: VENDOR len: 20 (supports v2 NAT-T, draft-ietf-ipsec-nat-t-ike-02) payload: VENDOR len: 20 (supports v3 NAT-T, draft-ietf-ipsec-nat-t-ike-03) payload: VENDOR len: 20 (supports DPD v1.0) [ttl 0] (id 1, len 188) 18:24:12.442839 64.119.37.74.500 > 64.119.40.170.500: [udp sum ok] isakmp v1.0 exchange INFO cookie: 1c466525c4ed2062->0000000000000000 msgid: 00000000 len: 40 payload: NOTIFICATION len: 12 notification: NO PROPOSAL CHOSEN [ttl 0] (id 1, len 68) 6. Went back to my initial /etc/ipsec.conf that seemed to complete the Phase 1 transform, but hung on Phase 2 and tried everything all over again # cat /etc/ipsec.conf ike dynamic esp tunnel from any to 192.168.1.0/24 \ main auth hmac-sha1 enc 3des group modp1024 \ quick auth hmac-sha2-256 enc 3des \ psk abc123 # echo "p on" > /var/run/isakmpd.fifo # echo "p off" > /var/run/isakmpd.fifo # tcpdump -r /var/run/isakmpd.pcap -vvn tcpdump: WARNING: snaplen raised from 96 to 65536 18:32:48.599289 64.119.40.170.500 > 64.119.37.74.500: [udp sum ok] isakmp v1.0 exchange ID _PROT cookie: 006a1301618e50c2->0000000000000000 msgid: 00000000 len: 160 payload: SA len: 52 DOI: 1(IPSEC) situation: IDENTITY_ONLY payload: PROPOSAL len: 40 proposal: 1 proto: ISAKMP spisz: 0 xforms: 1 payload: TRANSFORM len: 32 transform: 0 ID: ISAKMP attribute ENCRYPTION_ALGORITHM = 3DES_CBC attribute HASH_ALGORITHM = SHA attribute AUTHENTICATION_METHOD = PRE_SHARED attribute GROUP_DESCRIPTION = MODP_1024 attribute LIFE_TYPE = SECONDS attribute LIFE_DURATION = 3600 payload: VENDOR len: 20 (supports v1 NAT-T, draft-ietf-ipsec-nat-t-ike-00) payload: VENDOR len: 20 (supports v2 NAT-T, draft-ietf-ipsec-nat-t-ike-02) payload: VENDOR len: 20 (supports v3 NAT-T, draft-ietf-ipsec-nat-t-ike-03) payload: VENDOR len: 20 (supports DPD v1.0) [ttl 0] (id 1, len 188) 18:32:48.605289 64.119.37.74.500 > 64.119.40.170.500: [udp sum ok] isakmp v1.0 exchange ID _PROT cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: 00000000 len: 180 payload: SA len: 52 DOI: 1(IPSEC) situation: IDENTITY_ONLY payload: PROPOSAL len: 40 proposal: 1 proto: ISAKMP spisz: 0 xforms: 1 payload: TRANSFORM len: 32 transform: 0 ID: ISAKMP attribute ENCRYPTION_ALGORITHM = 3DES_CBC attribute HASH_ALGORITHM = SHA attribute AUTHENTICATION_METHOD = PRE_SHARED attribute GROUP_DESCRIPTION = MODP_1024 attribute LIFE_TYPE = SECONDS attribute LIFE_DURATION = 3600 payload: VENDOR len: 20 (supports OpenBSD-4.0) payload: VENDOR len: 20 (supports v2 NAT-T, draft-ietf-ipsec-nat-t-ike-02) payload: VENDOR len: 20 (supports v3 NAT-T, draft-ietf-ipsec-nat-t-ike-03) payload: VENDOR len: 20 (supports NAT-T, RFC 3947) payload: VENDOR len: 20 (supports DPD v1.0) [ttl 0] (id 1, len 208) 18:32:49.089864 64.119.40.170.500 > 64.119.37.74.500: [udp sum ok] isakmp v1.0 exchange ID _PROT cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: 00000000 len: 228 payload: KEY_EXCH len: 132 payload: NONCE len: 20 payload: NAT-D-DRAFT len: 24 payload: NAT-D-DRAFT len: 24 [ttl 0] (id 1, len 256) 18:32:49.143210 64.119.37.74.4500 > 64.119.40.170.4500: [udp sum ok] udpencap: isakmp v1.0 exchange ID_PROT cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: 00000000 len: 228 payload: KEY_EXCH len: 132 payload: NONCE len: 20 payload: NAT-D-DRAFT len: 24 payload: NAT-D-DRAFT len: 24 [ttl 0] (id 1, len 260) 18:32:49.300718 64.119.40.170.4500 > 64.119.37.74.4500: [bad udp cksum 729f!] udpencap: is akmp v1.0 exchange ID_PROT cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: 00000000 len: 92 payload: ID len: 12 type: IPV4_ADDR = 192.168.11.200 payload: HASH len: 24 payload: NOTIFICATION len: 28 notification: INITIAL CONTACT (006a1301618e50c2->72ac6fa4514a7a00) [ttl 0] (id 1, len 124) 18:32:49.302016 64.119.37.74.4500 > 64.119.40.170.4500: [bad udp cksum 2cab!] udpencap: is akmp v1.0 exchange ID_PROT cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: 00000000 len: 104 payload: ID len: 24 type: FQDN = "gateway.home.lan" payload: HASH len: 24 payload: NOTIFICATION len: 28 notification: INITIAL CONTACT (006a1301618e50c2->72ac6fa4514a7a00) [ttl 0] (id 1, len 136) 18:32:49.373070 64.119.40.170.4500 > 64.119.37.74.500: [bad udp cksum e6c7!] udpencap: isa kmp v1.0 exchange QUICK_MODE cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: acc5e032 len: 148 payload: HASH len: 24 payload: SA len: 48 DOI: 1(IPSEC) situation: IDENTITY_ONLY payload: PROPOSAL len: 36 proposal: 1 proto: IPSEC_ESP spisz: 4 xforms: 1 SPI: 0xfc34a42d payload: TRANSFORM len: 24 transform: 1 ID: 3DES attribute LIFE_TYPE = SECONDS attribute LIFE_DURATION = 3600 attribute ENCAPSULATION_MODE = 61443 (unknown) attribute AUTHENTICATION_ALGORITHM = HMAC_SHA payload: NONCE len: 20 payload: ID len: 12 type: IPV4_ADDR = 192.168.11.200 payload: ID len: 16 type: IPV4_ADDR_SUBNET = 192.168.1.0/255.255.255.0 [ttl 0] (id 1, len 180) 18:32:49.374171 64.119.37.74.4500 > 64.119.40.170.4500: [bad udp cksum 2400!] udpencap: is akmp v1.0 exchange INFO cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: 231354db len: 64 payload: HASH len: 24 payload: NOTIFICATION len: 12 notification: NO PROPOSAL CHOSEN [ttl 0] (id 1, len 96) 18:32:51.141486 64.119.37.74.4500 > 64.119.40.170.4500: [udp sum ok] udpencap: isakmp v1.0 exchange QUICK_MODE cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: 73463ff9 len: 288 payload: HASH len: 24 payload: SA len: 52 DOI: 1(IPSEC) situation: IDENTITY_ONLY payload: PROPOSAL len: 40 proposal: 1 proto: IPSEC_ESP spisz: 4 xforms: 1 SPI: 0x893a7df9 payload: TRANSFORM len: 28 transform: 1 ID: 3DES attribute LIFE_TYPE = SECONDS attribute LIFE_DURATION = 1200 attribute ENCAPSULATION_MODE = TUNNEL attribute AUTHENTICATION_ALGORITHM = HMAC_SHA2_256 attribute GROUP_DESCRIPTION = 2 payload: NONCE len: 20 payload: KEY_EXCH len: 132 payload: ID len: 16 type: IPV4_ADDR_SUBNET = 0.0.0.0/0.0.0.0 payload: ID len: 16 type: IPV4_ADDR_SUBNET = 192.168.1.0/255.255.255.0 [ttl 0] (id 1, len 320) 18:32:54.322013 64.119.37.74.4500 > 64.119.40.170.4500: [bad udp cksum 280!] udpencap: isa kmp v1.0 exchange INFO cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: 0f474bcf len: 84 payload: HASH len: 24 payload: NOTIFICATION len: 32 notification: STATUS_DPD_R_U_THERE seq 28733 [ttl 0] (id 1, len 116) 18:32:54.375588 64.119.40.170.4500 > 64.119.37.74.500: [bad udp cksum 280!] udpencap: isak mp v1.0 exchange INFO cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: 74a59589 len: 84 payload: HASH len: 24 payload: NOTIFICATION len: 32 notification: STATUS_DPD_R_U_THERE_ACK seq 28733 [ttl 0] (id 1, len 116) 18:32:55.377184 64.119.40.170.4500 > 64.119.37.74.4500: [bad udp cksum e947!] udpencap: is akmp v1.0 exchange QUICK_MODE cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: acc5e032 len: 148 payload: HASH len: 24 payload: SA len: 48 DOI: 1(IPSEC) situation: IDENTITY_ONLY payload: PROPOSAL len: 36 proposal: 1 proto: IPSEC_ESP spisz: 4 xforms: 1 SPI: 0xfc34a42d payload: TRANSFORM len: 24 transform: 1 ID: 3DES attribute LIFE_TYPE = SECONDS attribute LIFE_DURATION = 3600 attribute ENCAPSULATION_MODE = 61443 (unknown) attribute AUTHENTICATION_ALGORITHM = HMAC_SHA payload: NONCE len: 20 payload: ID len: 12 type: IPV4_ADDR = 192.168.11.200 payload: ID len: 16 type: IPV4_ADDR_SUBNET = 192.168.1.0/255.255.255.0 [ttl 0] (id 1, len 180) 18:32:55.378120 64.119.37.74.4500 > 64.119.40.170.4500: [bad udp cksum 2400!] udpencap: is akmp v1.0 exchange INFO cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: aa6eb0be len: 64 payload: HASH len: 24 payload: NOTIFICATION len: 12 notification: NO PROPOSAL CHOSEN [ttl 0] (id 1, len 96) 18:32:59.391983 64.119.37.74.4500 > 64.119.40.170.4500: [bad udp cksum 280!] udpencap: isa kmp v1.0 exchange INFO cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: bad5ee9a len: 84 payload: HASH len: 24 payload: NOTIFICATION len: 32 notification: STATUS_DPD_R_U_THERE seq 28734 [ttl 0] (id 1, len 116) 18:32:59.456695 64.119.40.170.4500 > 64.119.37.74.500: [bad udp cksum 280!] udpencap: isak mp v1.0 exchange INFO cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: f27b7e00 len: 84 payload: HASH len: 24 payload: NOTIFICATION len: 32 notification: STATUS_DPD_R_U_THERE_ACK seq 28734 [ttl 0] (id 1, len 116) 18:33:04.462001 64.119.37.74.4500 > 64.119.40.170.4500: [bad udp cksum 280!] udpencap: isa kmp v1.0 exchange INFO cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: dc50c200 len: 84 payload: HASH len: 24 payload: NOTIFICATION len: 32 notification: STATUS_DPD_R_U_THERE seq 28735 [ttl 0] (id 1, len 116) 18:33:04.515614 64.119.40.170.4500 > 64.119.37.74.500: [bad udp cksum 280!] udpencap: isak mp v1.0 exchange INFO cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: a1a604bd len: 84 payload: HASH len: 24 payload: NOTIFICATION len: 32 notification: STATUS_DPD_R_U_THERE_ACK seq 28735 [ttl 0] (id 1, len 116) 18:33:09.532010 64.119.37.74.4500 > 64.119.40.170.4500: [bad udp cksum 280!] udpencap: isa kmp v1.0 exchange INFO cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: eb853807 len: 84 payload: HASH len: 24 payload: NOTIFICATION len: 32 notification: STATUS_DPD_R_U_THERE seq 28736 [ttl 0] (id 1, len 116) 18:33:09.584642 64.119.40.170.4500 > 64.119.37.74.500: [bad udp cksum 280!] udpencap: isak mp v1.0 exchange INFO cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: 3b39de48 len: 84 payload: HASH len: 24 payload: NOTIFICATION len: 32 notification: STATUS_DPD_R_U_THERE_ACK seq 28736 [ttl 0] (id 1, len 116) 18:33:14.592010 64.119.37.74.4500 > 64.119.40.170.4500: [bad udp cksum 280!] udpencap: isa kmp v1.0 exchange INFO cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: d8469136 len: 84 payload: HASH len: 24 payload: NOTIFICATION len: 32 notification: STATUS_DPD_R_U_THERE seq 28737 [ttl 0] (id 1, len 116) 18:33:14.645623 64.119.40.170.4500 > 64.119.37.74.500: [bad udp cksum 280!] udpencap: isak mp v1.0 exchange INFO cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: 46548fd7 len: 84 payload: HASH len: 24 payload: NOTIFICATION len: 32 notification: STATUS_DPD_R_U_THERE_ACK seq 28737 [ttl 0] (id 1, len 116) 18:33:19.661985 64.119.37.74.4500 > 64.119.40.170.4500: [bad udp cksum 280!] udpencap: isa kmp v1.0 exchange INFO cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: a5453e27 len: 84 payload: HASH len: 24 payload: NOTIFICATION len: 32 notification: STATUS_DPD_R_U_THERE seq 28738 [ttl 0] (id 1, len 116) 18:33:19.714866 64.119.40.170.4500 > 64.119.37.74.500: [bad udp cksum 280!] udpencap: isak mp v1.0 exchange INFO cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: b0ce6ed1 len: 84 payload: HASH len: 24 payload: NOTIFICATION len: 32 notification: STATUS_DPD_R_U_THERE_ACK seq 28738 [ttl 0] (id 1, len 116) 18:33:24.732016 64.119.37.74.4500 > 64.119.40.170.4500: [bad udp cksum 280!] udpencap: isa kmp v1.0 exchange INFO cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: 88a5d405 len: 84 payload: HASH len: 24 payload: NOTIFICATION len: 32 notification: STATUS_DPD_R_U_THERE seq 28739 [ttl 0] (id 1, len 116) 18:33:24.787445 64.119.40.170.4500 > 64.119.37.74.500: [bad udp cksum 280!] udpencap: isak mp v1.0 exchange INFO cookie: 006a1301618e50c2->72ac6fa4514a7a00 msgid: b6391410 len: 84 payload: HASH len: 24 payload: NOTIFICATION len: 32 notification: STATUS_DPD_R_U_THERE_ACK seq 28739 [ttl 0] (id 1, len 116)