Hi, Thank you. I know its kafka related question but thread started with issue in golang but later we are suspecting issue in go kafka library configuration. FYI, I am interested in kafka client or producer properties. log.retention.hours or log.retention.bytes are kafka broker related configuration. so I am confused now
On Friday, March 11, 2022 at 1:34:23 AM UTC+5:30 ren...@ix.netcom.com wrote: > Look at log.retention.hours and log.retention.bytes > > You should post this in the Kafka forums not the Go ones. > > On Mar 10, 2022, at 11:04 AM, Rakesh K R <rakesh...@gmail.com> wrote: > > Hi, > > Sorry I am not sure which kafka configuration are you referring here. Can > you please point me to the right configuration responsible for retaining > the message for replay. > I see following properties which might be related but not sure: > queued.min.messages > queued.max.messages.kbytes > queue.buffering.max.messages > queue.buffering.max.kbytes > linger.ms ---> this is currently set to 1000 > message.timeout.ms > > Thank you > > On Thursday, March 10, 2022 at 9:50:50 PM UTC+5:30 ren...@ix.netcom.com > wrote: > >> You need to configure Kafka for how long it retains messages for replay - >> or some other option to store on disk. >> >> On Mar 10, 2022, at 10:07 AM, Rakesh K R <rakesh...@gmail.com> wrote: >> >> Tamas, >> >> Thanks you. So any suggestion on how to make application release this >> 900MiB memory back to OS so that pod will not end up in OOMKilled state? >> >> On Thursday, March 10, 2022 at 1:45:18 PM UTC+5:30 Tamás Gulácsi wrote: >> >>> gopkg.in/confluentinc/confluent-kafka-go.v1/kafka._Cfunc_GoBytes >>> >>> says it uses cgo, hiding it's memory usage from Go. I bet that 900MiB of >>> memory is there... >>> >>> >>> Rakesh K R a következőt írta (2022. március 10., csütörtök, 7:26:57 >>> UTC+1): >>> >>>> HI, >>>> I have a micro service application deployed on kubernetes cluster(with >>>> 1gb of pod memory limit). This app receive continuous messages(500 message >>>> per second) from producer app over kafka interface(these messages are >>>> encoded in protobuf format.) >>>> >>>> *Basic application flow:* >>>> 1. Get the message one by one from kafka >>>> 2. unmarshal proto message >>>> 3. apply business logic >>>> 4. write the message to redis cache(in []byte format) >>>> >>>> When pod starts memory will be around 50mb and memory starts increasing >>>> as traffic flows into the application. It is never released back to OS. As >>>> a result pod restarts with error code *OOMKilled*. >>>> I have integrated grafana to see memory usage like RSS, heap, stack. >>>> During this traffic flow, in-use heap size is 80mb, idle heap is 80mb >>>> where as process resident memory is at 800-1000MB. Stopping the traffic >>>> completely for hours did not help and RSS continue to remain in 1000mb. >>>> Tried to analyze this with pprof and it reports only 80mb are in in-use >>>> section. So I am wondering where these remaining 800-1000mb of pods memory >>>> went. Also application allocates memory like slices/maps/strings to >>>> perform >>>> business logic(see alloc_space pprof output below) >>>> >>>> I tried couple of experiments: >>>> 1. Calling FreeOsMemory() in the app but that did not help >>>> 2. invoking my app with GODEBUG=madvdontneed=1 my_app_executable and >>>> did not help >>>> 3. Leaving the application for 5-6hrs without any traffic to see >>>> whether memory comes down. It did not help >>>> 4. pprof shows only 80mb of heap in use >>>> 5. Tried upgrading golang version from 1.13 to 1.16 as there were some >>>> improvements in runtime. It did not help >>>> >>>> pprof output for *alloc_space*: >>>> >>>> (pprof) top20 >>>> Showing nodes accounting for 481.98GB, 91.57% of 526.37GB total >>>> Dropped 566 nodes (cum <= 2.63GB) >>>> Showing top 20 nodes out of 114 >>>> flat flat% sum% cum cum% >>>> 78.89GB 14.99% 14.99% 78.89GB 14.99% >>>> github.com/go-redis/redis/v7/internal/proto.(*Reader).readStringReply >>>> 67.01GB 12.73% 27.72% 285.33GB 54.21% >>>> airgroup/internal/wrapper/agrediswrapper.GetAllConfigurationForGroups >>>> 58.75GB 11.16% 38.88% 58.75GB 11.16% >>>> google.golang.org/protobuf/internal/impl.(*MessageInfo).MessageOf >>>> 52.26GB 9.93% 48.81% 52.26GB 9.93% reflect.unsafe_NewArray >>>> 45.78GB 8.70% 57.50% 46.38GB 8.81% >>>> encoding/json.(*decodeState).literalStore >>>> 36.98GB 7.02% 64.53% 36.98GB 7.02% reflect.New >>>> 28.20GB 5.36% 69.89% 28.20GB 5.36% >>>> gopkg.in/confluentinc/confluent-kafka-go.v1/kafka._Cfunc_GoBytes >>>> 25.60GB 4.86% 74.75% 63.62GB 12.09% >>>> google.golang.org/protobuf/proto.MarshalOptions.marshal >>>> 12.79GB 2.43% 77.18% 165.56GB 31.45% >>>> encoding/json.(*decodeState).object >>>> 12.73GB 2.42% 79.60% 12.73GB 2.42% reflect.mapassign >>>> 11.05GB 2.10% 81.70% 63.31GB 12.03% reflect.MakeSlice >>>> 10.06GB 1.91% 83.61% 12.36GB 2.35% >>>> filterServersForDestinationDevicesAndSendToDistributionChan >>>> 6.92GB 1.32% 84.92% 309.45GB 58.79% >>>> groupAndSendToConfigPolicyChannel >>>> 6.79GB 1.29% 86.21% 48.85GB 9.28% >>>> publishInternalMsgToDistributionService >>>> 6.79GB 1.29% 87.50% 174.81GB 33.21% encoding/json.Unmarshal >>>> 6.14GB 1.17% 88.67% 6.14GB 1.17% >>>> google.golang.org/protobuf/internal/impl.consumeBytes >>>> 4.64GB 0.88% 89.55% 14.39GB 2.73% >>>> GetAllDevDataFromGlobalDevDataDb >>>> 4.11GB 0.78% 90.33% 18.47GB 3.51% >>>> GetAllServersFromServerRecordDb >>>> 3.27GB 0.62% 90.95% 3.27GB 0.62% net.HardwareAddr.String >>>> 3.23GB 0.61% 91.57% 3.23GB 0.61% reflect.makemap >>>> (pprof) >>>> >>>> >>>> Need experts help in analyzing this issue. >>>> >>>> Thanks in advance!! >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/e9b91937-7bf5-4526-940f-2a60b2989ddfn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/golang-nuts/e9b91937-7bf5-4526-940f-2a60b2989ddfn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts...@googlegroups.com. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/0b7a3c46-00f6-43ff-9db4-b0520282610an%40googlegroups.com > > <https://groups.google.com/d/msgid/golang-nuts/0b7a3c46-00f6-43ff-9db4-b0520282610an%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/70cbff4e-464f-4d4a-96da-e9c82ca89978n%40googlegroups.com.
