Hello Mohit,
I'm copy-pasting Mathieu's previous email on making Streams to work on
Windows, note it was for 0.10.0.1 but I think the process should be very
similar.
--------
To any who follow in my footsteps, here is my trail:
1. Upgrade to at least Kafka Streams 0.10.0.1 (currently only in RC).
- This is necessary because .1 bumps the rocksdb dependency to 4.8.0,
where the previous 4.4 dependency did not yet support loading a
Windows JNI
library.
2. Follow the instructions here (
https://github.com/facebook/rocksdb/blob/v4.8/CMakeLists.txt) to build
rocksdb.
- That link is for the v4.8 tag; be sure to match this with the
version of rocksdb that Kafka Streams depends upon in the
future. 0.10.0.1
-> v4.8, but future releases of Kafka Streams will likely depend on
newer
versions.
- Be sure to include the "-DJNI=1" compile option to build the JNI
wrapper.
- None of the third-party dependencies (eg. snappy, jemalloc, etc.)
seem to be required to get something functional, but, it
probably isn't the
most efficient build.
- Ensure that "javac" and "javah" are on your path when running
cmake; if there are any errors in the cmake output your JNI wrapper
probably won't build.
- You may or may not need to make minor patches to make the project
build in Windows. It appears that the Windows build is often broken;
for
example, I had to apply this patch:
https://github.com/facebook/rocksdb/pull/1223/files
- Phew, that should give you a build\java\Debug\rocksdbjni.dll. So
close to the summit... just a horrible hack left...
3. Copy rocksdbjni.dll to librocksdbjni-win64.dll.
4. Insert librocksdbjni-win64.dll into your rocksdbjni-4.8.0.jar.
- Ugh, this is the horrible hack. rocksdbjni seems to only look
inside its own jar for its native libraries, so, this is where
it needs to
be.
- If you're a gradle user on Windows, you'd find this jar file
in C:\Users\...your-windows-user...\.gradle\caches\modules-2\
files-2.1\org.rocksdb\rocksdbjni\4.8.0\b543fc4ea5b52ad790730dee376ba0
df06d9f5f7.
And there you go, almost a working Kafka Streams app in Windows. One other
detail is that the default state storage directory doesn't seem to be
created on demand, so I had to mkdir C:\tmp\kafka-streams myself before my
app would work.
------------
Currently we have not spent too much time running and maintaining Kafka
Streams in Windows yet, but we'd love to hear your experience and if there
is any issues come up, we would like to help debugging and fixing them.
Guozhang
On Mon, Nov 28, 2016 at 5:28 PM, Mohit Anchlia <mohitanch...@gmail.com>
wrote:
> I just cloned 3.1x and tried to run a test. I am still seeing rocksdb
> error:
>
> Exception in thread "StreamThread-1" java.lang.UnsatisfiedLinkError:
> C:\Users\manchlia\AppData\Local\Temp\librocksdbjni1087890444431344273.dll:
> Can't find dependent libraries
>
>
> On Mon, Oct 24, 2016 at 11:26 AM, Matthias J. Sax <matth...@confluent.io>
> wrote:
>
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA512
> >
> > It's a client issues... But CP 3.1 should be our in about 2 weeks...
> > Of course, you can use Kafka 0.10.1.0 for now. It was released last
> > week and does contain the fix.
> >
> > - -Matthias
> >
> > On 10/24/16 9:19 AM, Mohit Anchlia wrote:
> > > Would this be an issue if I connect to a remote Kafka instance
> > > running on the Linux box? Or is this a client issue. What's rockdb
> > > used for to keep state?
> > >
> > > On Mon, Oct 24, 2016 at 12:08 AM, Matthias J. Sax
> > > <matth...@confluent.io> wrote:
> > >
> > > Kafka 0.10.1.0 which was release last week does contain the fix
> > > already. The fix will be in CP 3.1 coming up soon!
> > >
> > > (sorry that I did mix up versions in a previous email)
> > >
> > > -Matthias
> > >
> > > On 10/23/16 12:10 PM, Mohit Anchlia wrote:
> > >>>> So if I get it right I will not have this fix until 4
> > >>>> months? Should I just create my own example with the next
> > >>>> version of Kafka?
> > >>>>
> > >>>> On Sat, Oct 22, 2016 at 9:04 PM, Matthias J. Sax
> > >>>> <matth...@confluent.io> wrote:
> > >>>>
> > >>>> Current version is 3.0.1 CP 3.1 should be release the next
> > >>>> weeks
> > >>>>
> > >>>> So CP 3.2 should be there is about 4 month (Kafka follows a
> > >>>> time base release cycle of 4 month and CP usually aligns with
> > >>>> Kafka releases)
> > >>>>
> > >>>> -Matthias
> > >>>>
> > >>>>
> > >>>> On 10/20/16 5:10 PM, Mohit Anchlia wrote:
> > >>>>>>> Any idea of when 3.2 is coming?
> > >>>>>>>
> > >>>>>>> On Thu, Oct 20, 2016 at 4:53 PM, Matthias J. Sax
> > >>>>>>> <matth...@confluent.io> wrote:
> > >>>>>>>
> > >>>>>>> No problem. Asking questions is the purpose of mailing
> > >>>>>>> lists. :)
> > >>>>>>>
> > >>>>>>> The issue will be fixed in next version of examples
> > >>>>>>> branch.
> > >>>>>>>
> > >>>>>>> Examples branch is build with CP dependency and not
> > >>>>>>> with Kafka dependency. CP-3.2 is not available yet;
> > >>>>>>> only Kafka 0.10.1.0. Nevertheless, they should work
> > >>>>>>> with Kafka dependency, too. I never tried it, but you
> > >>>>>>> should give it a shot...
> > >>>>>>>
> > >>>>>>> But you should use example master branch because of
> > >>>>>>> API changes from 0.10.0.x to 0.10.1 (and thus, changing
> > >>>>>>> CP-3.1 to 0.10.1.0 will not be compatible and not
> > >>>>>>> compile, while changing CP-3.2-SNAPSHOT to 0.10.1.0
> > >>>>>>> should work -- hopefully ;) )
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> -Matthias
> > >>>>>>>
> > >>>>>>> On 10/20/16 4:02 PM, Mohit Anchlia wrote:
> > >>>>>>>>>> So this issue I am seeing is fixed in the next
> > >>>>>>>>>> version of example branch? Can I change my pom to
> > >>>>>>>>>> point it the higher version of Kafka if that is
> > >>>>>>>>>> the issue? Or do I need to wait until new branch
> > >>>>>>>>>> is made available? Sorry lot of questions :)
> > >>>>>>>>>>
> > >>>>>>>>>> On Thu, Oct 20, 2016 at 3:56 PM, Matthias J. Sax
> > >>>>>>>>>> <matth...@confluent.io> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> The branch is 0.10.0.1 and not 0.10.1.0 (sorry
> > >>>>>>>>>> for so many zeros and ones -- super easy to mix
> > >>>>>>>>>> up)
> > >>>>>>>>>>
> > >>>>>>>>>> However, examples master branch uses
> > >>>>>>>>>> CP-3.1-SNAPSHOT (ie, Kafka 0.10.1.0) -- there
> > >>>>>>>>>> will be a 0.10.1 examples branch, after CP-3.1
> > >>>>>>>>>> was released
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> -Matthias
> > >>>>>>>>>>
> > >>>>>>>>>> On 10/20/16 3:48 PM, Mohit Anchlia wrote:
> > >>>>>>>>>>>>> I just now cloned this repo. It seems to be
> > >>>>>>>>>>>>> using 10.1
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> https://github.com/confluentinc/examples
> > >>>>>>>>>>>>> and running examples in
> > >>>>>>>>>>>>> https://github.com/confluentinc/examples/tree/kafka-0.10.0
> > .1-
> > >
> > >>>>>>>>>>>>>
> > cp-
> > >>>>
> > >>>>>>>>>>>>>
> > > 3.0
> > >>>>>>>
> > >>>>>>>>>>>>>
> > >>>> .1/
> > >>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>> kafka-streams
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Thu, Oct 20, 2016 at 3:10 PM, Michael
> > >>>>>>>>>>>>> Noll <mich...@confluent.io> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I suspect you are running Kafka 0.10.0.x
> > >>>>>>>>>>>>>> on Windows? If so, this is a known issue
> > >>>>>>>>>>>>>> that is fixed in Kafka 0.10.1 that was
> > >>>>>>>>>>>>>> just released today.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Also: which examples are you referring
> > >>>>>>>>>>>>>> to? And, to confirm: which git branch /
> > >>>>>>>>>>>>>> Kafka version / OS in case my guess above
> > >>>>>>>>>>>>>> was wrong.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Thursday, October 20, 2016, Mohit
> > >>>>>>>>>>>>>> Anchlia <mohitanch...@gmail.com> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I am trying to run the examples from
> > >>>>>>>>>>>>>>> git. While running the wordcount
> > >>>>>>>>>>>>>>> example I see this error:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Caused by:
> > >>>>>>>>>>>>>>> *java.lang.RuntimeException*:
> > >>>>>>>>>>>>>>> librocksdbjni-win64.dll was not found
> > >>>>>>>>>>>>>>> inside JAR.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Am I expected to include this jar
> > >>>>>>>>>>>>>>> locally?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> -- *Michael G. Noll* Product Manager |
> > >>>>>>>>>>>>>> Confluent +1 650 453 5860 | @miguno
> > >>>>>>>>>>>>>> <https://twitter.com/miguno> Follow us:
> > >>>>>>>>>>>>>> Twitter
> > >>>>>>>>>>>>>> <https://twitter.com/ConfluentInc> |
> > >>>>>>>>>>>>>> Blog <http://www.confluent.io/blog>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>
> > >
> > -----BEGIN PGP SIGNATURE-----
> > Comment: GPGTools - https://gpgtools.org
> >
> > iQIcBAEBCgAGBQJYDlJxAAoJECnhiMLycopPvAsP/2MVs4HI+MKupJTzwJouTllZ
> > KStLTskEzb8ny2EPDJXxx20MqCxUYObbazcKgJx8PxWQPvBXgz0VuGkkrL0Tzj6q
> > vnE3If+38yeoAcmC5fORxJj2jito0i9FHd7o5LRwUfEnJmwniNwK+YI6/GA0Z15R
> > /JqiDCbmHhk9/79gF4dKevKscx8DFQ5+Bx22Q8XNxf1Ywk74ws3VFDTA59PqXJ46
> > F4VuC0Ws2bAaHA+L5SCY2FZvKdiPyoU4zuUm7SzqxVh6UyFti7YhkHy0vVmZEWLZ
> > eTaQ8o+6RAkBtzJKX8xNsIF0bK3d5edTn4tlmXknA2+ptwFSU6QiBNETYouNb+ep
> > Bw05ccvy6QDDpGS7dO4gSm8t1npEUkB7KZ/9uLL4Pwmt9O6D99D4qpGST+lAjFvn
> > g4MRNU5lZV3qgDcIjDDbaY8atNoSWV9LiwB1mRl6K9QCDwgojqLfp+mz+HNhDrfo
> > TvpVzLoZtAY6dd9RxY3OGf++OqmjV5SM+0dKbLpneRKCQDAPsi5Qqm1Qoikpmh3o
> > RhB26N4PFvCNjShMxcuRM8kaGiRHno3DA9kCgsIpKr+Dw3a3lpTfO4cFeHepmWyo
> > kwHxrCmiFZszQQCI81nJcC2uqjD8UkK7EoicSMZY4K9exZ4JeBe7BlnsjF6ZfLR/
> > 0lMh15VEwRFLpMewDqQ5
> > =0qhL
> > -----END PGP SIGNATURE-----
> >
>
--
-- Guozhang
