Thanks KR. We don't use Kafka connect( I have seen this getting used in in ETL
CDC capture such as Maxwell CDC, debezium CDC), we have our own consumers and
own logic to place the data into SQL.
As you suggested, we will do some basic analysis by running Kafka for longer
duration continuously along with log cleanup activities based on SLA.
Thanks
Harish
On Mon, 7/17/17, M. Manna <manme...@gmail.com> wrote:
Subject: Re: Windows OS platform support
To: "harish jadhav" <harishjadhav1...@yahoo.com>
Cc: "Kafka Users" <users@kafka.apache.org>
Date: Monday, July 17, 2017, 6:20 PM
Okay and are you using Kafka
Connect (with JDBC connectors) or using plain
Producer Consumers (and your own library calls
to perform this SQL
operations) ?
If you have a maintenance
window (and some SLA) before which all the
messages are capture in SQL db - you can go
ahead and clean up the logs
yourself - Just
need to delete the contents of version-2 (zookeeper) and
Kafka log folders (specified by log.dirs in
prop file). That's your
challenge to
take home :)
The issue is
the same for server logs (it cannot rename/delete) but
same
suggestion follows here - just
clean/collect them manually in the
maintenance window.
I would not dismiss the trial straight away,
Please try and emulate some
basic scenario
with the workaround/manual log cleanup. What you would
really like to see is the following:
1) Can kafka recover from
failures as expected? On your Windows platform
2) Can Kafka reliably deliver messages over a
certain period (I would say,
keep pumping
data over 2 days, don't turn the machine off)
3) Some more tests if you have in mind.
As long as those are answered,
the Disk/File operation issue can be handled
by your maintenance window anyway. But it's
up to you whether you want to
try this or
not.
KR,
On 17 July 2017 at 18:05, harish jadhav <harishjadhav1...@yahoo.com>
wrote:
> Hi KR,
>
> Thanks for
clarifyication and thank you for spending your valuable time
to
> assist on our issues.
>
> Our requirement is to
have Kafka in Windows environment. The messages in
> Kafka gets pushed from external
entities; we have consumers which consume
> those records and insert to mySQL
database. Also we want to keep the
>
messages on Kafka for maximum one week for making sure we
won't loose any
> data when
connectivity is not there to DB.
>
> We don't use Linux as our setups will
be purely windows house (not even
> Linux
dockers or VM ).
>
>
Indeed we have not faced any other issues apart from the
file
> deletion/renaming issue which I
reported to you in my previous mail. As you
> suggested, we can cleanup the logs from
sepearte utility.
>
>
My only concern is that - as document suggests running kafka
on windows
> has some issues hence not
able to decide on using it on windows production
> bed.
>
> Thanks
> Harish
>
>
>
> On Mon, 7/17/17, M.
Manna <manme...@gmail.com>
wrote:
>
> Subject:
Re: Windows OS platform support
> To:
"harish jadhav" <harishjadhav1...@yahoo.com>
> Cc: "Kafka Users" <users@kafka.apache.org>
> Date: Monday, July 17, 2017, 4:04 PM
>
> I have
> never said that that we shouldn't
use Kafka officially
> for Windows
Production Environment. I am running a major PoC
> which involves developing solution on
Windows 10 - but my
> production docker
containers are using Alpine Linux so I
> won't have any issues for the final
integration
> pipeline.
> What I would like
>
to clarify is that if you are going for a full Windows
> pipeline for your implementation (i.e.
dev+delivery both on
> Windows
platoform) - you might want to do more research.
> This user group is awesome and everyone
is very active in
> getting all the
issues resolved asap.
> We have seen
good performance on
> Windows and they
are more or less consistent with all the
> metrics published in Kafka - if you
don't have a strict
> requirement
to do log cleanup you can use the workaround I
> mentioned, or just clean the logs
yourself.
> What is your
implementation
> requirement, exactly?
Do you need to preserve all the
>
messages (i.e. ability to replay)? Or would you be happy
to
> delete them during a scheduled
maintenance window
> ?
> KR,
> On 17 July
2017 at 16:19,
> harish jadhav <harishjadhav1...@yahoo.com>
> wrote:
> Hello
> M.Manna,
> Do you
mean
> we should not use Kafka in
windows production until the disk
> I/O
issues(For which you have given workaround) are resolved
> officilally?
>
> ThanksHarish
>
> On Monday, July 17, 2017,
> 5:31:11 PM GMT+5:30, M. Manna <manme...@gmail.com>
> wrote:
>
> The workaround is to
> download the sources and create a
wrapper around method
> forceUnmap in
LogSegment.scala file. Please do not take it a
> solution as no one has approved this yet
on KAFKA-1194
> ticket. but it
works.
> I
> tried
to update the scala sources to run the unit tests -
> but it fails all the time. Howerver, the
fix works. Also,
> the log files are
actually okay, it's the index and
>
timeIndex files which throws RTE.
> Do
the following (Prerequisite - you
>
have to follow the src download and setup instruction
from
> Kafka website - Also use
0.10.2.1 or higher):
> 1) Create a new
wrapper in
> AbstractIndex.scala
directly under def
> forceUnmap
> //
> Forces
unmapping of multibyte map files on
>
Windowsdef forceUnmapOnWindows() {if
>
(OperatingSystem.IS_WINDOWS)
>
forceUnmap(mmap)}
> 2. Find the
method
> changeFileSuffixes and invoke
the above method (i.e.
>
forceUnmapOnWindows() ) for index and timeIndex files -
> just before:
>
> i) The try catch block with index
file
> ii) The try catch block with
timeIndex file.
> So it will become
the
> following:
>
def
> changeFileSuffixes(oldSuffix:
String, newSuffix: String)
> {
> def kafkaStorageException(
fileType:
> String, e: IOException)
=
> new
>
KafkaStorageException(s"Failed to change the
$fileType
> file suffix from $oldSuffix
to $newSuffix for log segment
>
$baseOffset", e)
> try
> log.renameTo(new
File(CoreUtils.replaceSuffix(
>
log.file.getPath, oldSuffix, newSuffix)))
>
> catch {
> case e: IOException
> => throw
kafkaStorageException("log", e)
> }
>
> index.forceUnmapOnWindows()
> try
>
index.renameTo(new File(CoreUtils.replaceSuffix(
> index.file.getPath, oldSuffix,
newSuffix)))
> catch {
> case e:
>
IOException => throw
>
kafkaStorageException("index", e)
> }
>
> timeIndex.forceUnmapOnWindows( )
> try
>
timeIndex.renameTo(new File(CoreUtils.replaceSuffix(
> timeIndex.file.getPath, oldSuffix,
newSuffix)))
> catch {
> case e:
>
IOException => throw kafkaStorageException("
> timeindex", e)
> }
> }
3.
> Execute gradlew -jar --debug
(REMEMBER - you need to setup
>
everything with the project beforehand, otherwise it
> won't work).
>
Once
> the build is complete - get the
core jar and dump it in your
> kafka
broker lib folder (it will replace the old
> one).
> 4.
Redeploy
> and check if the cleanup is
working.
> KR,
>
On 17 July 2017 at 12:21, harish
>
jadhav <harishjadhav1...@yahoo.com>
> wrote:
> Hello
M.Manna,
>
>
Thanks
> for your quick and valuable
response.
> Actually I am getting
following
> error -
> Caused by:
>
java.nio.file. FileSystemException:
>
\tmp\kafka-logs\TopicName-2\
>
00000000000000007262.log
> ->
\tmp\kafka-logs\TopicName
>
-2\00000000000000007262.log. deleted: The
> process cannot access the file because
it is being used by
> another
process.
>
>
> Uncaught exception in
> scheduled task
'kafka-log-retention'
>
(kafka.utils.KafkaScheduler)
>
> kafka.common.
>
KafkaStorageException: Failed to change the log file
suffix
> from to .deleted for log
segment 7262
>
>
>
> There must be
> some workaround available for above
issues. However I
> don't know what
other potential issues can come in
>
future. Hence need your suggestion on whether to continue
to
> use Kafka in windows production
system.
> Please guide us on
> this.
>
ThanksHarish
>
>
> On Monday,
> July
17, 2017, 2:15:15 PM GMT+5:30, M. Manna <manme...@gmail.com>
> wrote:
>
> Hi,
> First I
> apologise and should just correct
myself. I should use
> "Disk/File
level operations" instead of using
> "I/O" since that refers to
different
> things.
> What I mean to
>
say that file deletion/renaming doesn't work even
when
> you provide full permissions in
Windows. But Unix/Linux
> based system
it's okay. This can be called as
>
"Deletion/Renaming of shared files" i.e. which
are
> already opened for
appending/editing.
> As an example try
to run the uni
> tests for Kafka on
Windows where file renaming/deletion
>
occurs as part of the tests.They will fail. But same
process
> works all good on Linux.
>
> From the official
documentation,
> Kafka has already made
a disclaimer that Windows
>
Implementation is not fully vetted (in progress). I
believe
> all the top examples from
LinkedIn are based on Unix/Linux.
> You
are actually okay to do PoC activities on windows - log
> retention/rename/deletion will work
fine.
> I hope this helps you
> understand.
>
KR,
> On
> 17 July
2017 at 09:22, harish jadhav <harishjadhav1...@yahoo.com>
> wrote:
> Hi
> M.Manna,
>
> Thanks for the update.
>
> If possible, can
you
> please share which admin I/O
level functionality will
> impact? Do
you mean something related to log files
> renaming?
> In
> documentation, there is no pointer with
respect to
> I/O.
> Kindly advice.
>
> ThanksHarish
>
>
>
> On
> Saturday, July
15, 2017, 1:59:58 PM GMT+5:30, M. Manna
> <manme...@gmail.com>
> wrote:
>
> Kafka
> has some
issues wuth IO level admin functionalities e.g.
> file deletion/renaming doesn't work
out of the box like
> Linux. However,
user base is growing and there's a
> momentum to support Windows OS
seamlessly like
> Linux.
> I have been running
> some stress tests on Windows and the
performance is quite
> consistent. If
you have any findings you would like to
> share, please publish them.
>
> Please read Kafka
documentation
> where there are some
disclaimers regarding OS
> support.
> On
> 15 Jul 2017
1:27 am, "Eno Thereska" <eno.there...@gmail.com>
> wrote:
> Hi
> Harish,
>
>
>
> I believe many people/orgs use it on
Windows. We rely on the
> community to
test/fix/answer any Windows questions, same as
> with Linux or MacOS. However, based on
what I've
> observed, perhaps there
are more people answering
>
Linux-related questions.
>
>
>
> Eno
>
>
>
> > On 14 Jul 2017, at 13:24, harish
jadhav <harishjadhav1...@yahoo.com.
> INVALID> wrote:
>
> >
>
> > Hello Team,
>
> >
>
> > I am exploring
Apache Kafka and found that one of the
> best MQ I have encountered. I was
exploring option to use it
> in Windows
machine and started some kind of proof of concept
> work referring installation section on
windows and it work
> perfectly. Later
realized that Kafka documentation says
> under Hardware and OS section " We
have seen a few
> issues running on
Windows and Windows is not currently a
> well supported platform though we would
be happy to change
> that. "
>
> >
>
> > I am curios to
know is there any actual issue running
> Kafka in windows OS as whichever feature
I am using with
> single instance (
Producer-Consumer) works perfectly in
>
windows test bed. Can I use it production windows
> machines?
>
> >
>
> > Please advice.
>
> >
>
> > Thanks
>
> > Harish
>
>
>
>
>
>
>
>
