I took that approach. It was painful, but, ultimately did get me a working
Windows development environment.
To any who follow in my footsteps, here is my trail:
1. Upgrade to at least Kafka Streams 0.10.0.1 (currently only in RC).
- This is necessary because .1 bumps the rocksdb dependency to 4.8.0,
where the previous 4.4 dependency did not yet support loading a
Windows JNI
library.
2. Follow the instructions here (
https://github.com/facebook/rocksdb/blob/v4.8/CMakeLists.txt) to build
rocksdb.
- That link is for the v4.8 tag; be sure to match this with the
version of rocksdb that Kafka Streams depends upon in the
future. 0.10.0.1
-> v4.8, but future releases of Kafka Streams will likely depend on newer
versions.
- Be sure to include the "-DJNI=1" compile option to build the JNI
wrapper.
- None of the third-party dependencies (eg. snappy, jemalloc, etc.)
seem to be required to get something functional, but, it
probably isn't the
most efficient build.
- Ensure that "javac" and "javah" are on your path when running
cmake; if there are any errors in the cmake output your JNI wrapper
probably won't build.
- You may or may not need to make minor patches to make the project
build in Windows. It appears that the Windows build is often broken; for
example, I had to apply this patch:
https://github.com/facebook/rocksdb/pull/1223/files
- Phew, that should give you a build\java\Debug\rocksdbjni.dll. So
close to the summit... just a horrible hack left...
3. Copy rocksdbjni.dll to librocksdbjni-win64.dll.
4. Insert librocksdbjni-win64.dll into your rocksdbjni-4.8.0.jar.
- Ugh, this is the horrible hack. rocksdbjni seems to only look
inside its own jar for its native libraries, so, this is where
it needs to
be.
- If you're a gradle user on Windows, you'd find this jar file
in
C:\Users\...your-windows-user...\.gradle\caches\modules-2\files-2.1\org.rocksdb\rocksdbjni\4.8.0\b543fc4ea5b52ad790730dee376ba0df06d9f5f7.
And there you go, almost a working Kafka Streams app in Windows. One other
detail is that the default state storage directory doesn't seem to be
created on demand, so I had to mkdir C:\tmp\kafka-streams myself before my
app would work.
Mathieu
On Fri, Aug 5, 2016 at 3:13 AM, Eno Thereska <eno.there...@gmail.com> wrote:
> Hi Mathieu,
>
> It is true that the DSL currently does not support configuration of the
> stores.
>
> Sounds like it might be worth trying to build RocksDb and dropping into
> classpath for now.
>
> Eno
>
> > On 4 Aug 2016, at 17:42, Mathieu Fenniak <mathieu.fenn...@replicon.com>
> wrote:
> >
> > Hi Eno,
> >
> > Yes, I've looked at that. RocksDB can be built and run in Windows, but,
> > the JNI wrapper does not include Windows binarie (
> > https://github.com/facebook/rocksdb/issues/703). rocksdbjni-4.4.1.jar
> > includes librocksdbjni-linux32.so, librocksdbjni-linux64.so, and
> > librocksdbjni-osx.jnilib, so only supports Linux x86 & x64 and OS X. It
> is
> > probably possible for me to build it myself and drop it in my classpath,
> > but, I'm looking for a lower friction approach if one exists. :-)
> >
> > It looks like this was discussed recently on the Confluent Platform
> mailing
> > list (https://groups.google.com/forum/#!topic/confluent-
> platform/Z1rsfSNrVJk)
> > and the conclusion there was that high-level streams DSL doesn't support
> > configuration of the stores.
> >
> > Mathieu
> >
> >
> > On Thu, Aug 4, 2016 at 10:28 AM, Eno Thereska <eno.there...@gmail.com>
> > wrote:
> >
> >> Hi Mathieu,
> >>
> >> Have you had a chance to look at http://rocksdb.org/blog/2033/
> >> rocksdb-is-now-available-in-windows-platform/? <
> >> http://rocksdb.org/blog/2033/rocksdb-is-now-available-in-
> windows-platform/?>
> >> Curious to hear your and other's comments on whether that worked.
> >>
> >> It is possible to have Kafka Streams use an in-memory store (included
> with
> >> Kafka Streams) for development purposes. In that scenario RocksDb would
> not
> >> be needed.
> >>
> >> Eno
> >>
> >>
> >>> On 4 Aug 2016, at 16:14, Mathieu Fenniak <mathieu.fenn...@replicon.com
> >
> >> wrote:
> >>>
> >>> Hey all,
> >>>
> >>> Is it anyone developing Kafka Streams applications on Windows?
> >>>
> >>> It seems like the RocksDB Java library doesn't include a native JNI
> >> library
> >>> for Windows, which prevents a Kafka Streams app from running on
> >> Windows. I
> >>> was just wondering if others have run into this, and if so, what
> approach
> >>> you took to resolve it.
> >>>
> >>> I'm favouring the idea of running my applications in a Vagrant VM to
> >> avoid
> >>> the issue. It makes the Windows development environment a little less
> >>> pleasant, but, seems plausible.
> >>>
> >>> Other ideas that occurred to me:
> >>>
> >>> - RocksDB does support Windows, but, there don't seem to be any
> >> binaries
> >>> available or packaged for it or the jni library. I could probably
> >> build
> >>> these myself, but it sounds a little painful.
> >>> - Not developing on Windows. Works for me, but, won't work as well
> for
> >>> my colleagues.
> >>> - Is it possible to make Kafka Streams not use any local state storage
> >>> in some kind of development mode? Not sure...
> >>>
> >>> Thanks,
> >>>
> >>> Mathieu
> >>
> >>
>
>
