No problems. Feel free to come back and share your results with us :).
On 17 July 2017 at 19:37, harish jadhav <harishjadhav1...@yahoo.com> wrote:
> Thanks KR. We don't use Kafka connect( I have seen this getting used in in
> ETL CDC capture such as Maxwell CDC, debezium CDC), we have our own
> consumers and own logic to place the data into SQL.
>
> As you suggested, we will do some basic analysis by running Kafka for
> longer duration continuously along with log cleanup activities based on SLA.
>
>
> Thanks
> Harish
>
> On Mon, 7/17/17, M. Manna <manme...@gmail.com> wrote:
>
> Subject: Re: Windows OS platform support
> To: "harish jadhav" <harishjadhav1...@yahoo.com>
> Cc: "Kafka Users" <users@kafka.apache.org>
> Date: Monday, July 17, 2017, 6:20 PM
>
> Okay and are you using Kafka
> Connect (with JDBC connectors) or using plain
> Producer Consumers (and your own library calls
> to perform this SQL
> operations) ?
>
> If you have a maintenance
> window (and some SLA) before which all the
> messages are capture in SQL db - you can go
> ahead and clean up the logs
> yourself - Just
> need to delete the contents of version-2 (zookeeper) and
> Kafka log folders (specified by log.dirs in
> prop file). That's your
> challenge to
> take home :)
>
> The issue is
> the same for server logs (it cannot rename/delete) but
> same
> suggestion follows here - just
> clean/collect them manually in the
> maintenance window.
>
> I would not dismiss the trial straight away,
> Please try and emulate some
> basic scenario
> with the workaround/manual log cleanup. What you would
> really like to see is the following:
>
> 1) Can kafka recover from
> failures as expected? On your Windows platform
> 2) Can Kafka reliably deliver messages over a
> certain period (I would say,
> keep pumping
> data over 2 days, don't turn the machine off)
> 3) Some more tests if you have in mind.
>
> As long as those are answered,
> the Disk/File operation issue can be handled
> by your maintenance window anyway. But it's
> up to you whether you want to
> try this or
> not.
>
> KR,
>
> On 17 July 2017 at 18:05, harish jadhav <harishjadhav1...@yahoo.com>
> wrote:
>
> > Hi KR,
> >
> > Thanks for
> clarifyication and thank you for spending your valuable time
> to
> > assist on our issues.
> >
> > Our requirement is to
> have Kafka in Windows environment. The messages in
> > Kafka gets pushed from external
> entities; we have consumers which consume
> > those records and insert to mySQL
> database. Also we want to keep the
> >
> messages on Kafka for maximum one week for making sure we
> won't loose any
> > data when
> connectivity is not there to DB.
> >
> > We don't use Linux as our setups will
> be purely windows house (not even
> > Linux
> dockers or VM ).
> >
> >
> Indeed we have not faced any other issues apart from the
> file
> > deletion/renaming issue which I
> reported to you in my previous mail. As you
> > suggested, we can cleanup the logs from
> sepearte utility.
> >
> >
> My only concern is that - as document suggests running kafka
> on windows
> > has some issues hence not
> able to decide on using it on windows production
> > bed.
> >
> > Thanks
> > Harish
> >
> >
> >
> > On Mon, 7/17/17, M.
> Manna <manme...@gmail.com>
> wrote:
> >
> > Subject:
> Re: Windows OS platform support
> > To:
> "harish jadhav" <harishjadhav1...@yahoo.com>
> > Cc: "Kafka Users" <users@kafka.apache.org>
> > Date: Monday, July 17, 2017, 4:04 PM
> >
> > I have
> > never said that that we shouldn't
> use Kafka officially
> > for Windows
> Production Environment. I am running a major PoC
> > which involves developing solution on
> Windows 10 - but my
> > production docker
> containers are using Alpine Linux so I
> > won't have any issues for the final
> integration
> > pipeline.
> > What I would like
> >
> to clarify is that if you are going for a full Windows
> > pipeline for your implementation (i.e.
> dev+delivery both on
> > Windows
> platoform) - you might want to do more research.
> > This user group is awesome and everyone
> is very active in
> > getting all the
> issues resolved asap.
> > We have seen
> good performance on
> > Windows and they
> are more or less consistent with all the
> > metrics published in Kafka - if you
> don't have a strict
> > requirement
> to do log cleanup you can use the workaround I
> > mentioned, or just clean the logs
> yourself.
> > What is your
> implementation
> > requirement, exactly?
> Do you need to preserve all the
> >
> messages (i.e. ability to replay)? Or would you be happy
> to
> > delete them during a scheduled
> maintenance window
> > ?
> > KR,
> > On 17 July
> 2017 at 16:19,
> > harish jadhav <harishjadhav1...@yahoo.com>
> > wrote:
> > Hello
> > M.Manna,
> > Do you
> mean
> > we should not use Kafka in
> windows production until the disk
> > I/O
> issues(For which you have given workaround) are resolved
> > officilally?
> >
> > ThanksHarish
> >
> > On Monday, July 17, 2017,
> > 5:31:11 PM GMT+5:30, M. Manna <manme...@gmail.com>
> > wrote:
> >
> > The workaround is to
> > download the sources and create a
> wrapper around method
> > forceUnmap in
> LogSegment.scala file. Please do not take it a
> > solution as no one has approved this yet
> on KAFKA-1194
> > ticket. but it
> works.
> > I
> > tried
> to update the scala sources to run the unit tests -
> > but it fails all the time. Howerver, the
> fix works. Also,
> > the log files are
> actually okay, it's the index and
> >
> timeIndex files which throws RTE.
> > Do
> the following (Prerequisite - you
> >
> have to follow the src download and setup instruction
> from
> > Kafka website - Also use
> 0.10.2.1 or higher):
> > 1) Create a new
> wrapper in
> > AbstractIndex.scala
> directly under def
> > forceUnmap
> > //
> > Forces
> unmapping of multibyte map files on
> >
> Windowsdef forceUnmapOnWindows() {if
> >
> (OperatingSystem.IS_WINDOWS)
> >
> forceUnmap(mmap)}
> > 2. Find the
> method
> > changeFileSuffixes and invoke
> the above method (i.e.
> >
> forceUnmapOnWindows() ) for index and timeIndex files -
> > just before:
> >
> > i) The try catch block with index
> file
> > ii) The try catch block with
> timeIndex file.
> > So it will become
> the
> > following:
> >
> def
> > changeFileSuffixes(oldSuffix:
> String, newSuffix: String)
> > {
> > def kafkaStorageException(
> fileType:
> > String, e: IOException)
> =
> > new
> >
> KafkaStorageException(s"Failed to change the
> $fileType
> > file suffix from $oldSuffix
> to $newSuffix for log segment
> >
> $baseOffset", e)
> > try
> > log.renameTo(new
> File(CoreUtils.replaceSuffix(
> >
> log.file.getPath, oldSuffix, newSuffix)))
> >
> > catch {
> > case e: IOException
> > => throw
> kafkaStorageException("log", e)
> > }
> >
> > index.forceUnmapOnWindows()
> > try
> >
> index.renameTo(new File(CoreUtils.replaceSuffix(
> > index.file.getPath, oldSuffix,
> newSuffix)))
> > catch {
> > case e:
> >
> IOException => throw
> >
> kafkaStorageException("index", e)
> > }
> >
> > timeIndex.forceUnmapOnWindows( )
> > try
> >
> timeIndex.renameTo(new File(CoreUtils.replaceSuffix(
> > timeIndex.file.getPath, oldSuffix,
> newSuffix)))
> > catch {
> > case e:
> >
> IOException => throw kafkaStorageException("
> > timeindex", e)
> > }
> > }
> 3.
> > Execute gradlew -jar --debug
> (REMEMBER - you need to setup
> >
> everything with the project beforehand, otherwise it
> > won't work).
> >
> Once
> > the build is complete - get the
> core jar and dump it in your
> > kafka
> broker lib folder (it will replace the old
> > one).
> > 4.
> Redeploy
> > and check if the cleanup is
> working.
> > KR,
> >
> On 17 July 2017 at 12:21, harish
> >
> jadhav <harishjadhav1...@yahoo.com>
> > wrote:
> > Hello
> M.Manna,
> >
> >
> Thanks
> > for your quick and valuable
> response.
> > Actually I am getting
> following
> > error -
> > Caused by:
> >
> java.nio.file. FileSystemException:
> >
> \tmp\kafka-logs\TopicName-2\
> >
> 00000000000000007262.log
> > ->
> \tmp\kafka-logs\TopicName
> >
> -2\00000000000000007262.log. deleted: The
> > process cannot access the file because
> it is being used by
> > another
> process.
> >
> >
> > Uncaught exception in
> > scheduled task
> 'kafka-log-retention'
> >
> (kafka.utils.KafkaScheduler)
> >
> > kafka.common.
> >
> KafkaStorageException: Failed to change the log file
> suffix
> > from to .deleted for log
> segment 7262
> >
> >
> >
> > There must be
> > some workaround available for above
> issues. However I
> > don't know what
> other potential issues can come in
> >
> future. Hence need your suggestion on whether to continue
> to
> > use Kafka in windows production
> system.
> > Please guide us on
> > this.
> >
> ThanksHarish
> >
> >
> > On Monday,
> > July
> 17, 2017, 2:15:15 PM GMT+5:30, M. Manna <manme...@gmail.com>
> > wrote:
> >
> > Hi,
> > First I
> > apologise and should just correct
> myself. I should use
> > "Disk/File
> level operations" instead of using
> > "I/O" since that refers to
> different
> > things.
> > What I mean to
> >
> say that file deletion/renaming doesn't work even
> when
> > you provide full permissions in
> Windows. But Unix/Linux
> > based system
> it's okay. This can be called as
> >
> "Deletion/Renaming of shared files" i.e. which
> are
> > already opened for
> appending/editing.
> > As an example try
> to run the uni
> > tests for Kafka on
> Windows where file renaming/deletion
> >
> occurs as part of the tests.They will fail. But same
> process
> > works all good on Linux.
> >
> > From the official
> documentation,
> > Kafka has already made
> a disclaimer that Windows
> >
> Implementation is not fully vetted (in progress). I
> believe
> > all the top examples from
> LinkedIn are based on Unix/Linux.
> > You
> are actually okay to do PoC activities on windows - log
> > retention/rename/deletion will work
> fine.
> > I hope this helps you
> > understand.
> >
> KR,
> > On
> > 17 July
> 2017 at 09:22, harish jadhav <harishjadhav1...@yahoo.com>
> > wrote:
> > Hi
> > M.Manna,
> >
> > Thanks for the update.
> >
> > If possible, can
> you
> > please share which admin I/O
> level functionality will
> > impact? Do
> you mean something related to log files
> > renaming?
> > In
> > documentation, there is no pointer with
> respect to
> > I/O.
> > Kindly advice.
> >
> > ThanksHarish
> >
> >
> >
> > On
> > Saturday, July
> 15, 2017, 1:59:58 PM GMT+5:30, M. Manna
> > <manme...@gmail.com>
> > wrote:
> >
> > Kafka
> > has some
> issues wuth IO level admin functionalities e.g.
> > file deletion/renaming doesn't work
> out of the box like
> > Linux. However,
> user base is growing and there's a
> > momentum to support Windows OS
> seamlessly like
> > Linux.
> > I have been running
> > some stress tests on Windows and the
> performance is quite
> > consistent. If
> you have any findings you would like to
> > share, please publish them.
> >
> > Please read Kafka
> documentation
> > where there are some
> disclaimers regarding OS
> > support.
> > On
> > 15 Jul 2017
> 1:27 am, "Eno Thereska" <eno.there...@gmail.com>
> > wrote:
> > Hi
> > Harish,
> >
> >
> >
> > I believe many people/orgs use it on
> Windows. We rely on the
> > community to
> test/fix/answer any Windows questions, same as
> > with Linux or MacOS. However, based on
> what I've
> > observed, perhaps there
> are more people answering
> >
> Linux-related questions.
> >
> >
> >
> > Eno
> >
> >
> >
> > > On 14 Jul 2017, at 13:24, harish
> jadhav <harishjadhav1...@yahoo.com.
> > INVALID> wrote:
> >
> > >
> >
> > > Hello Team,
> >
> > >
> >
> > > I am exploring
> Apache Kafka and found that one of the
> > best MQ I have encountered. I was
> exploring option to use it
> > in Windows
> machine and started some kind of proof of concept
> > work referring installation section on
> windows and it work
> > perfectly. Later
> realized that Kafka documentation says
> > under Hardware and OS section " We
> have seen a few
> > issues running on
> Windows and Windows is not currently a
> > well supported platform though we would
> be happy to change
> > that. "
> >
> > >
> >
> > > I am curios to
> know is there any actual issue running
> > Kafka in windows OS as whichever feature
> I am using with
> > single instance (
> Producer-Consumer) works perfectly in
> >
> windows test bed. Can I use it production windows
> > machines?
> >
> > >
> >
> > > Please advice.
> >
> > >
> >
> > > Thanks
> >
> > > Harish
> >
> >
> >
> >
> >
> >
> >
> >
>
>
