Thanks a lot for investing and also for sharing back your findings, Mathieu!
-Michael On Fri, Aug 5, 2016 at 3:10 PM, Mathieu Fenniak < mathieu.fenn...@replicon.com> wrote: > I took that approach. It was painful, but, ultimately did get me a working > Windows development environment. > > To any who follow in my footsteps, here is my trail: > > 1. Upgrade to at least Kafka Streams 0.10.0.1 (currently only in RC). > - This is necessary because .1 bumps the rocksdb dependency to 4.8.0, > where the previous 4.4 dependency did not yet support loading a > Windows JNI > library. > 2. Follow the instructions here ( > https://github.com/facebook/rocksdb/blob/v4.8/CMakeLists.txt) to build > rocksdb. > - That link is for the v4.8 tag; be sure to match this with the > version of rocksdb that Kafka Streams depends upon in the > future. 0.10.0.1 > -> v4.8, but future releases of Kafka Streams will likely depend on > newer > versions. > - Be sure to include the "-DJNI=1" compile option to build the JNI > wrapper. > - None of the third-party dependencies (eg. snappy, jemalloc, etc.) > seem to be required to get something functional, but, it > probably isn't the > most efficient build. > - Ensure that "javac" and "javah" are on your path when running > cmake; if there are any errors in the cmake output your JNI wrapper > probably won't build. > - You may or may not need to make minor patches to make the project > build in Windows. It appears that the Windows build is often > broken; for > example, I had to apply this patch: > https://github.com/facebook/rocksdb/pull/1223/files > - Phew, that should give you a build\java\Debug\rocksdbjni.dll. So > close to the summit... just a horrible hack left... > 3. Copy rocksdbjni.dll to librocksdbjni-win64.dll. > 4. Insert librocksdbjni-win64.dll into your rocksdbjni-4.8.0.jar. > - Ugh, this is the horrible hack. rocksdbjni seems to only look > inside its own jar for its native libraries, so, this is where > it needs to > be. > - If you're a gradle user on Windows, you'd find this jar file > in C:\Users\...your-windows-user...\.gradle\caches\modules-2\ > files-2.1\org.rocksdb\rocksdbjni\4.8.0\b543fc4ea5b52ad790730dee376ba0 > df06d9f5f7. > > And there you go, almost a working Kafka Streams app in Windows. One other > detail is that the default state storage directory doesn't seem to be > created on demand, so I had to mkdir C:\tmp\kafka-streams myself before my > app would work. > > Mathieu > > > On Fri, Aug 5, 2016 at 3:13 AM, Eno Thereska <eno.there...@gmail.com> > wrote: > > > Hi Mathieu, > > > > It is true that the DSL currently does not support configuration of the > > stores. > > > > Sounds like it might be worth trying to build RocksDb and dropping into > > classpath for now. > > > > Eno > > > > > On 4 Aug 2016, at 17:42, Mathieu Fenniak <mathieu.fenn...@replicon.com > > > > wrote: > > > > > > Hi Eno, > > > > > > Yes, I've looked at that. RocksDB can be built and run in Windows, > but, > > > the JNI wrapper does not include Windows binarie ( > > > https://github.com/facebook/rocksdb/issues/703). rocksdbjni-4.4.1.jar > > > includes librocksdbjni-linux32.so, librocksdbjni-linux64.so, and > > > librocksdbjni-osx.jnilib, so only supports Linux x86 & x64 and OS X. > It > > is > > > probably possible for me to build it myself and drop it in my > classpath, > > > but, I'm looking for a lower friction approach if one exists. :-) > > > > > > It looks like this was discussed recently on the Confluent Platform > > mailing > > > list (https://groups.google.com/forum/#!topic/confluent- > > platform/Z1rsfSNrVJk) > > > and the conclusion there was that high-level streams DSL doesn't > support > > > configuration of the stores. > > > > > > Mathieu > > > > > > > > > On Thu, Aug 4, 2016 at 10:28 AM, Eno Thereska <eno.there...@gmail.com> > > > wrote: > > > > > >> Hi Mathieu, > > >> > > >> Have you had a chance to look at http://rocksdb.org/blog/2033/ > > >> rocksdb-is-now-available-in-windows-platform/? < > > >> http://rocksdb.org/blog/2033/rocksdb-is-now-available-in- > > windows-platform/?> > > >> Curious to hear your and other's comments on whether that worked. > > >> > > >> It is possible to have Kafka Streams use an in-memory store (included > > with > > >> Kafka Streams) for development purposes. In that scenario RocksDb > would > > not > > >> be needed. > > >> > > >> Eno > > >> > > >> > > >>> On 4 Aug 2016, at 16:14, Mathieu Fenniak < > mathieu.fenn...@replicon.com > > > > > >> wrote: > > >>> > > >>> Hey all, > > >>> > > >>> Is it anyone developing Kafka Streams applications on Windows? > > >>> > > >>> It seems like the RocksDB Java library doesn't include a native JNI > > >> library > > >>> for Windows, which prevents a Kafka Streams app from running on > > >> Windows. I > > >>> was just wondering if others have run into this, and if so, what > > approach > > >>> you took to resolve it. > > >>> > > >>> I'm favouring the idea of running my applications in a Vagrant VM to > > >> avoid > > >>> the issue. It makes the Windows development environment a little > less > > >>> pleasant, but, seems plausible. > > >>> > > >>> Other ideas that occurred to me: > > >>> > > >>> - RocksDB does support Windows, but, there don't seem to be any > > >> binaries > > >>> available or packaged for it or the jni library. I could probably > > >> build > > >>> these myself, but it sounds a little painful. > > >>> - Not developing on Windows. Works for me, but, won't work as well > > for > > >>> my colleagues. > > >>> - Is it possible to make Kafka Streams not use any local state > storage > > >>> in some kind of development mode? Not sure... > > >>> > > >>> Thanks, > > >>> > > >>> Mathieu > > >> > > >> > > > > >
