Okay and are you using Kafka Connect (with JDBC connectors) or using plain
Producer Consumers (and your own library calls to perform this SQL
operations) ?
If you have a maintenance window (and some SLA) before which all the
messages are capture in SQL db - you can go ahead and clean up the logs
yourself - Just need to delete the contents of version-2 (zookeeper) and
Kafka log folders (specified by log.dirs in prop file). That's your
challenge to take home :)
The issue is the same for server logs (it cannot rename/delete) but same
suggestion follows here - just clean/collect them manually in the
maintenance window.
I would not dismiss the trial straight away, Please try and emulate some
basic scenario with the workaround/manual log cleanup. What you would
really like to see is the following:
1) Can kafka recover from failures as expected? On your Windows platform
2) Can Kafka reliably deliver messages over a certain period (I would say,
keep pumping data over 2 days, don't turn the machine off)
3) Some more tests if you have in mind.
As long as those are answered, the Disk/File operation issue can be handled
by your maintenance window anyway. But it's up to you whether you want to
try this or not.
KR,
On 17 July 2017 at 18:05, harish jadhav <harishjadhav1...@yahoo.com> wrote:
> Hi KR,
>
> Thanks for clarifyication and thank you for spending your valuable time to
> assist on our issues.
>
> Our requirement is to have Kafka in Windows environment. The messages in
> Kafka gets pushed from external entities; we have consumers which consume
> those records and insert to mySQL database. Also we want to keep the
> messages on Kafka for maximum one week for making sure we won't loose any
> data when connectivity is not there to DB.
>
> We don't use Linux as our setups will be purely windows house (not even
> Linux dockers or VM ).
>
> Indeed we have not faced any other issues apart from the file
> deletion/renaming issue which I reported to you in my previous mail. As you
> suggested, we can cleanup the logs from sepearte utility.
>
> My only concern is that - as document suggests running kafka on windows
> has some issues hence not able to decide on using it on windows production
> bed.
>
> Thanks
> Harish
>
>
>
> On Mon, 7/17/17, M. Manna <manme...@gmail.com> wrote:
>
> Subject: Re: Windows OS platform support
> To: "harish jadhav" <harishjadhav1...@yahoo.com>
> Cc: "Kafka Users" <users@kafka.apache.org>
> Date: Monday, July 17, 2017, 4:04 PM
>
> I have
> never said that that we shouldn't use Kafka officially
> for Windows Production Environment. I am running a major PoC
> which involves developing solution on Windows 10 - but my
> production docker containers are using Alpine Linux so I
> won't have any issues for the final integration
> pipeline.
> What I would like
> to clarify is that if you are going for a full Windows
> pipeline for your implementation (i.e. dev+delivery both on
> Windows platoform) - you might want to do more research.
> This user group is awesome and everyone is very active in
> getting all the issues resolved asap.
> We have seen good performance on
> Windows and they are more or less consistent with all the
> metrics published in Kafka - if you don't have a strict
> requirement to do log cleanup you can use the workaround I
> mentioned, or just clean the logs yourself.
> What is your implementation
> requirement, exactly? Do you need to preserve all the
> messages (i.e. ability to replay)? Or would you be happy to
> delete them during a scheduled maintenance window
> ?
> KR,
> On 17 July 2017 at 16:19,
> harish jadhav <harishjadhav1...@yahoo.com>
> wrote:
> Hello
> M.Manna,
> Do you mean
> we should not use Kafka in windows production until the disk
> I/O issues(For which you have given workaround) are resolved
> officilally?
>
> ThanksHarish
>
> On Monday, July 17, 2017,
> 5:31:11 PM GMT+5:30, M. Manna <manme...@gmail.com>
> wrote:
>
> The workaround is to
> download the sources and create a wrapper around method
> forceUnmap in LogSegment.scala file. Please do not take it a
> solution as no one has approved this yet on KAFKA-1194
> ticket. but it works.
> I
> tried to update the scala sources to run the unit tests -
> but it fails all the time. Howerver, the fix works. Also,
> the log files are actually okay, it's the index and
> timeIndex files which throws RTE.
> Do the following (Prerequisite - you
> have to follow the src download and setup instruction from
> Kafka website - Also use 0.10.2.1 or higher):
> 1) Create a new wrapper in
> AbstractIndex.scala directly under def
> forceUnmap
> //
> Forces unmapping of multibyte map files on
> Windowsdef forceUnmapOnWindows() {if
> (OperatingSystem.IS_WINDOWS)
> forceUnmap(mmap)}
> 2. Find the method
> changeFileSuffixes and invoke the above method (i.e.
> forceUnmapOnWindows() ) for index and timeIndex files -
> just before:
>
> i) The try catch block with index file
> ii) The try catch block with timeIndex file.
> So it will become the
> following:
> def
> changeFileSuffixes(oldSuffix: String, newSuffix: String)
> {
> def kafkaStorageException( fileType:
> String, e: IOException) =
> new
> KafkaStorageException(s"Failed to change the $fileType
> file suffix from $oldSuffix to $newSuffix for log segment
> $baseOffset", e)
> try
> log.renameTo(new File(CoreUtils.replaceSuffix(
> log.file.getPath, oldSuffix, newSuffix)))
>
> catch {
> case e: IOException
> => throw kafkaStorageException("log", e)
> }
>
> index.forceUnmapOnWindows()
> try
> index.renameTo(new File(CoreUtils.replaceSuffix(
> index.file.getPath, oldSuffix, newSuffix)))
> catch {
> case e:
> IOException => throw
> kafkaStorageException("index", e)
> }
>
> timeIndex.forceUnmapOnWindows( )
> try
> timeIndex.renameTo(new File(CoreUtils.replaceSuffix(
> timeIndex.file.getPath, oldSuffix, newSuffix)))
> catch {
> case e:
> IOException => throw kafkaStorageException("
> timeindex", e)
> }
> } 3.
> Execute gradlew -jar --debug (REMEMBER - you need to setup
> everything with the project beforehand, otherwise it
> won't work).
> Once
> the build is complete - get the core jar and dump it in your
> kafka broker lib folder (it will replace the old
> one).
> 4. Redeploy
> and check if the cleanup is working.
> KR,
> On 17 July 2017 at 12:21, harish
> jadhav <harishjadhav1...@yahoo.com>
> wrote:
> Hello M.Manna,
>
> Thanks
> for your quick and valuable response.
> Actually I am getting following
> error -
> Caused by:
> java.nio.file. FileSystemException:
> \tmp\kafka-logs\TopicName-2\
> 00000000000000007262.log
> -> \tmp\kafka-logs\TopicName
> -2\00000000000000007262.log. deleted: The
> process cannot access the file because it is being used by
> another process.
>
>
> Uncaught exception in
> scheduled task 'kafka-log-retention'
> (kafka.utils.KafkaScheduler)
>
> kafka.common.
> KafkaStorageException: Failed to change the log file suffix
> from to .deleted for log segment 7262
>
>
>
> There must be
> some workaround available for above issues. However I
> don't know what other potential issues can come in
> future. Hence need your suggestion on whether to continue to
> use Kafka in windows production system.
> Please guide us on
> this.
> ThanksHarish
>
>
> On Monday,
> July 17, 2017, 2:15:15 PM GMT+5:30, M. Manna <manme...@gmail.com>
> wrote:
>
> Hi,
> First I
> apologise and should just correct myself. I should use
> "Disk/File level operations" instead of using
> "I/O" since that refers to different
> things.
> What I mean to
> say that file deletion/renaming doesn't work even when
> you provide full permissions in Windows. But Unix/Linux
> based system it's okay. This can be called as
> "Deletion/Renaming of shared files" i.e. which are
> already opened for appending/editing.
> As an example try to run the uni
> tests for Kafka on Windows where file renaming/deletion
> occurs as part of the tests.They will fail. But same process
> works all good on Linux.
>
> From the official documentation,
> Kafka has already made a disclaimer that Windows
> Implementation is not fully vetted (in progress). I believe
> all the top examples from LinkedIn are based on Unix/Linux.
> You are actually okay to do PoC activities on windows - log
> retention/rename/deletion will work fine.
> I hope this helps you
> understand.
> KR,
> On
> 17 July 2017 at 09:22, harish jadhav <harishjadhav1...@yahoo.com>
> wrote:
> Hi
> M.Manna,
>
> Thanks for the update.
>
> If possible, can you
> please share which admin I/O level functionality will
> impact? Do you mean something related to log files
> renaming?
> In
> documentation, there is no pointer with respect to
> I/O.
> Kindly advice.
>
> ThanksHarish
>
>
>
> On
> Saturday, July 15, 2017, 1:59:58 PM GMT+5:30, M. Manna
> <manme...@gmail.com>
> wrote:
>
> Kafka
> has some issues wuth IO level admin functionalities e.g.
> file deletion/renaming doesn't work out of the box like
> Linux. However, user base is growing and there's a
> momentum to support Windows OS seamlessly like
> Linux.
> I have been running
> some stress tests on Windows and the performance is quite
> consistent. If you have any findings you would like to
> share, please publish them.
>
> Please read Kafka documentation
> where there are some disclaimers regarding OS
> support.
> On
> 15 Jul 2017 1:27 am, "Eno Thereska" <eno.there...@gmail.com>
> wrote:
> Hi
> Harish,
>
>
>
> I believe many people/orgs use it on Windows. We rely on the
> community to test/fix/answer any Windows questions, same as
> with Linux or MacOS. However, based on what I've
> observed, perhaps there are more people answering
> Linux-related questions.
>
>
>
> Eno
>
>
>
> > On 14 Jul 2017, at 13:24, harish jadhav <harishjadhav1...@yahoo.com.
> INVALID> wrote:
>
> >
>
> > Hello Team,
>
> >
>
> > I am exploring Apache Kafka and found that one of the
> best MQ I have encountered. I was exploring option to use it
> in Windows machine and started some kind of proof of concept
> work referring installation section on windows and it work
> perfectly. Later realized that Kafka documentation says
> under Hardware and OS section " We have seen a few
> issues running on Windows and Windows is not currently a
> well supported platform though we would be happy to change
> that. "
>
> >
>
> > I am curios to know is there any actual issue running
> Kafka in windows OS as whichever feature I am using with
> single instance ( Producer-Consumer) works perfectly in
> windows test bed. Can I use it production windows
> machines?
>
> >
>
> > Please advice.
>
> >
>
> > Thanks
>
> > Harish
>
>
>
>
>
>
>
>
