Hi All,
Does everyone knows if is it possible to connect on Kafka brokers remotely
using IPSec VPN ?
If I connect through Internet it works fine, but inside an IPSec tunnel
does not work.
I´m implementing a scenario that the producers and consumers will be
located on a Datacenter A, and the cluster of Kafka and Zookeeper will be
located on Datacenter B. I know that is not recommended to run nodes of
Kafka and zookeeper over the WAN , but in this case they will run on the
same location (Datacenter A), only the producers and consumers will run
outside ( Datacenter B)
*Infrastructure Detais :*
VPN IPSec Device : *Openswan* on both side using proposals below :
ike=3des-sha1-modp1024,aes128-sha1-modp1024
esp=3des-sha1,aes128-sha1
Latency between Datacenters : *< 10ms*
When I start the producer, I got a disconnection after produce the first
message.
./kafka-console-producer.sh --broker-list server-kafka01:9092 --topic test:
[2017-03-01 17:13:44,146] WARN Bootstrap broker server-kafka01:9092
*disconnected* (org.apache.kafka.clients.NetworkClient)
*See a part of tcpdump collection*
*172.31.10.154 = producer*
*172.17.9.84 = *
*kafka broker*
17:15:12.645984 IP 172.31.10.154.49140 > 172.17.9.84.9092: Flags [S], seq
4072723614, win 26883, options [mss 8919,sackOK,TS val 501901970 ecr
0,nop,wscale 0], length 0
17:15:12.646010 IP 172.17.9.84.9092 > 172.31.10.154.49140: Flags [S.], seq
1316762982, ack 4072723615, win 28960, options [mss 1460,sackOK,TS val
527499997 ecr 501901970,nop,wscale 1], length 0
17:15:12.656237 IP 172.31.10.154.49140 > 172.17.9.84.9092: Flags [.], ack
1, win 26883, options [nop,nop,TS val 501901981 ecr 527499997], length 0
17:15:12.792223 IP 172.31.10.154.49140 > 172.17.9.84.9092: Flags [P.], seq
1:51, ack 1, win 26883, options [nop,nop,TS val 501902117 ecr 527499997],
length 50
17:15:12.792247 IP 172.17.9.84.9092 > 172.31.10.154.49140: Flags [.], ack
51, win 14480, options [nop,nop,TS val 527500143 ecr 501902117], length 0
17:15:12.792659 IP 172.17.9.84.9092 > 172.31.10.154.49140: Flags [P.], seq
1:2303, ack 51, win 14480, options [nop,nop,TS val 527500144 ecr
501902117], length 2302
17:15:12.802064 IP 172.31.10.154.49140 > 172.17.9.84.9092: Flags [.], ack
1, win 26883, options [nop,nop,TS val 501902127 ecr 527500143,nop,nop,sack
1 {1449:2303}], length 0
17:15:12.804304 IP 172.17.9.84.9092 > 172.31.10.154.49140: Flags [.], seq
1:1449, ack 51, win 14480, options [nop,nop,TS val 527500156 ecr
501902127], length 1448
17:15:13.014320 IP 172.17.9.84.9092 > 172.31.10.154.49140: Flags [.], seq
1:1449, ack 51, win 14480, options [nop,nop,TS val 527500366 ecr
501902127], length 1448
17:15:13.435338 IP 172.17.9.84.9092 > 172.31.10.154.49140: Flags [.], seq
1:1449, ack 51, win 14480, options [nop,nop,TS val 527500787 ecr
501902127], length 1448
17:15:14.276334 IP 172.17.9.84.9092 > 172.31.10.154.49140: Flags [.], seq
1:1449, ack 51, win 14480, options [nop,nop,TS val 527501628 ecr
501902127], length 1448
17:15:14.294642 IP 172.31.10.154.49140 > 172.17.9.84.9092: *Flags [F.]*,
seq 51, ack 1, win 26883, options [nop,nop,TS val 501903619 ecr
527500143,nop,nop,sack 1 {1449:2303}], length 0
17:15:14.294851 IP 172.17.9.84.9092 > 172.31.10.154.49140: *Flags [F.]*,
seq 2303, ack 52, win 14480, options [nop,nop,TS val 527501646 ecr
501903619], length 0
17:15:14.309289 IP 172.31.10.154.49140 > 172.17.9.84.9092: *Flags [R]*, seq
4072723666, win 0, length 0
17:15:14.410441 IP 172.31.10.154.49142 > 172.17.9.84.9092: Flags [S], seq
1240554369, win 26883, options [mss 8919,sackOK,TS val 501903735 ecr
0,nop,wscale 0], length 0
17:15:14.410486 IP 172.17.9.84.9092 > 172.31.10.154.49142: Flags [S.], seq
3005945570, ack 1240554370, win 28960, options [mss 1460,sackOK,TS val
527501762 ecr 501903735,nop,wscale 1], length 0
17:15:14.416284 IP 172.31.10.154.49142 > 172.17.9.84.9092: Flags [.], ack
1, win 26883, options [nop,nop,TS val 501903741 ecr 527501762], length 0
17:15:14.511220 IP 172.31.10.154.49142 > 172.17.9.84.9092: Flags [P.], seq
1:51, ack 1, win 26883, options [nop,nop,TS val 501903836 ecr 527501762],
length 50
17:15:14.511245 IP 172.17.9.84.9092 > 172.31.10.154.49142: Flags [.], ack
51, win 14480, options [nop,nop,TS val 527501862 ecr 501903836], length 0
17:15:14.511659 IP 172.17.9.84.9092 > 172.31.10.154.49142: Flags [P.], seq
1:2303, ack 51, win 14480, options [nop,nop,TS val 527501863 ecr
501903836], length 2302
17:15:14.517670 IP 172.31.10.154.49142 > 172.17.9.84.9092: Flags [.], ack
1, win 26883, options [nop,nop,TS val 501903842 ecr 527501862,nop,nop,sack
1 {1449:2303}], length 0
17:15:14.519290 IP 172.17.9.84.9092 > 172.31.10.154.49142: Flags [.], seq
1:1449, ack 51, win 14480, options [nop,nop,TS val 527501871 ecr
501903842], length 1448
17:15:14.726295 IP 172.17.9.84.9092 > 172.31.10.154.49142: Flags [.], seq
1:1449, ack 51, win 14480, options [nop,nop,TS val 527502078 ecr
501903842], length 1448
17:15:15.141294 IP 172.17.9.84.9092 > 172.31.10.154.49142: Flags [.], seq
1:1449, ack 51, win 14480, options [nop,nop,TS val 527502493 ecr
501903842], length 1448
17:15:15.972325 IP 172.17.9.84.9092 > 172.31.10.154.49142: Flags [.], seq
1:1449, ack 51, win 14480, options [nop,nop,TS val 527503324 ecr
501903842], length 1448
17:15:16.012950 IP 172.31.10.154.49142 > 172.17.9.84.9092: *Flags [F.]*,
seq 51, ack 1, win 26883, options [nop,nop,TS val 501905337 ecr
527501862,nop,nop,sack 1 {1449:2303}], length 0
17:15:16.013134 IP 172.17.9.84.9092 > 172.31.10.154.49142: *Flags [F.]*,
seq 2303, ack 52, win 14480, options [nop,nop,TS val 527503364 ecr
501905337], length 0
17:15:16.019160 IP 172.31.10.154.49142 > 172.17.9.84.9092:* Flags [R]*, seq
1240554421, win 0, length 0
17:15:16.113187 IP 172.31.10.154.49144 > 172.17.9.84.9092: Flags [S], seq
362441987, win 26883, options [mss 8919,sackOK,TS val 501905437 ecr
0,nop,wscale 0], length 0
17:15:16.113215 IP 172.17.9.84.9092 > 172.31.10.154.49144: Flags [S.], seq
3563709477, ack 362441988, win 28960, options [mss 1460,sackOK,TS val
527503464 ecr 501905437,nop,wscale 1], length 0
17:15:16.119323 IP 172.31.10.154.49144 > 172.17.9.84.9092: Flags [.], ack
1, win 26883, options [nop,nop,TS val 501905444 ecr 527503464], length 0
17:15:16.213260 IP 172.31.10.154.49144 > 172.17.9.84.9092: Flags [P.], seq
1:51, ack 1, win 26883, options [nop,nop,TS val 501905538 ecr 527503464],
length 50
17:15:16.213283 IP 172.17.9.84.9092 > 172.31.10.154.49144: Flags [.], ack
51, win 14480, options [nop,nop,TS val 527503564 ecr 501905538], length 0
17:15:16.213713 IP 172.17.9.84.9092 > 172.31.10.154.49144: Flags [P.], seq
1:2303, ack 51, win 14480, options [nop,nop,TS val 527503565 ecr
501905538], length 2302
17:15:16.219685 IP 172.31.10.154.49144 > 172.17.9.84.9092: Flags [.], ack
1, win 26883, options [nop,nop,TS val 501905544 ecr 527503564,nop,nop,sack
1 {1449:2303}], length 0
17:15:16.221307 IP 172.17.9.84.9092 > 172.31.10.154.49144: Flags [.], seq
1:1449, ack 51, win 14480, options [nop,nop,TS val 527503573 ecr
501905544], length 1448
17:15:16.428313 IP 172.17.9.84.9092 > 172.31.10.154.49144: Flags [.], seq
1:1449, ack 51, win 14480, options [nop,nop,TS val 527503780 ecr
501905544], length 1448
17:15:16.843312 IP 172.17.9.84.9092 > 172.31.10.154.49144: Flags [.], seq
1:1449, ack 51, win 14480, options [nop,nop,TS val 527504195 ecr
501905544], length 1448
17:15:17.672301 IP 172.17.9.84.9092 > 172.31.10.154.49144: Flags [.], seq
1:1449, ack 51, win 14480, options [nop,nop,TS val 527505024 ecr
501905544], length 1448
17:15:17.714982 IP 172.31.10.154.49144 > 172.17.9.84.9092: *Flags [F.],*
seq 51, ack 1, win 26883, options [nop,nop,TS val 501907039 ecr
527503564,nop,nop,sack 1 {1449:2303}], length 0
17:15:17.715098 IP 172.17.9.84.9092 > 172.31.10.154.49144: *Flags [F.],*
seq 2303, ack 52, win 14480, options [nop,nop,TS val 527505066 ecr
501907039], length 0
17:15:17.721467 IP 172.31.10.154.49144 > 172.17.9.84.9092: *Flags [R]*, seq
362442039, win 0, length 0
17:15:17.816131 IP 172.31.10.154.49146 > 172.17.9.84.9092: Flags [S], seq
481333162, win 26883, options [mss 8919,sackOK,TS val 501907140 ecr
0,nop,wscale 0], length 0
17:15:17.816152 IP 172.17.9.84.9092 > 172.31.10.154.49146: Flags [S.], seq
501906802, ack 481333163, win 28960, options [mss 1460,sackOK,TS val
527505167 ecr 501907140,nop,wscale 1], length 0
17:15:17.822033 IP 172.31.10.154.49146 > 172.17.9.84.9092: Flags [.], ack
1, win 26883, options [nop,nop,TS val 501907147 ecr 527505167], length 0
17:15:17.915938 IP 172.31.10.154.49146 > 172.17.9.84.9092: Flags [P.], seq
1:51, ack 1, win 26883, options [nop,nop,TS val 501907240 ecr 527505167],
length 50
17:15:17.915963 IP 172.17.9.84.9092 > 172.31.10.154.49146: Flags [.], ack
51, win 14480, options [nop,nop,TS val 527505267 ecr 501907240], length 0
17:15:17.916477 IP 172.17.9.84.9092 > 172.31.10.154.49146: Flags [P.], seq
1:2303, ack 51, win 14480, options [nop,nop,TS val 527505268 ecr
501907240], length 2302
17:15:17.922261 IP 172.31.10.154.49146 > 172.17.9.84.9092: Flags [.], ack
1, win 26883, options [nop,nop,TS val 501907247 ecr 527505267,nop,nop,sack
1 {1449:2303}], length 0
17:15:17.924307 IP 172.17.9.84.9092 > 172.31.10.154.49146: Flags [.], seq
1:1449, ack 51, win 14480, options [nop,nop,TS val 527505276 ecr
501907247], length 1448
17:15:18.130323 IP 172.17.9.84.9092 > 172.31.10.154.49146: Flags [.], seq
1:1449, ack 51, win 14480, options [nop,nop,TS val 527505482 ecr
501907247], length 1448
17:15:18.543284 IP 172.17.9.84.9092 > 172.31.10.154.49146: Flags [.], seq
1:1449, ack 51, win 14480, options [nop,nop,TS val 527505895 ecr
501907247], length 1448
Thanks in Advance,
Daniel Rosa
