Ok so an update here, it's just a learning thing alongside the day job so it took a little longer than expected. I upgraded to 2.6 but the same thing happened, the reason is that rocksDB doesn't include the .so for the Raspberry Pi platform, hence the linker error. Solution was to build the JNI jar on the Raspberry PI.
Blogged it so there is a record in case someone else has the challenge. One 'interesting' piece was that although the application had never successfully run there was a record of the state somewhere in the broker so once the linker issue was the resolved it failed because it couldn't access the RocksDB database in the state that it thought existed, changed the application.id and it all worked fine. Not sure that counts as a defect on the Kafka side and I didn't have time to track down the root cause. http://service-architecture.blogspot.com/2020/08/getting-rocksdb-working-on-raspberry-pi.html On Mon, 24 Aug 2020 at 20:53, John Roesler <vvcep...@apache.org> wrote: > Ah, yes, before messing with rocksjni. As Sophie said, you'll probably > just want to try using 2.6.0. It should work "out of the box". > > Thanks, > -John > > On Mon, Aug 24, 2020, at 21:08, Steve Jones wrote: > > Version: *kafka_2.12-2.5.0* > > > > RocksDB compiles fine, Kafka compiles fine, I'm just trying to work out > > where the linking is done and if that can be changed to a static rather > > than a dynamic link. Any hints appreciated, and I'll document out what > > I've found. Its very stable on the Raspberry Pi as a broker, and for > > streams if the streams is run on another machine. > > > > Steve > > > > On Mon, 24 Aug 2020 at 17:49, Sophie Blee-Goldman <sop...@confluent.io> > > wrote: > > > > > Yeah, if you weren't already running it, try upgrading Streams to 2.6. > They > > > recently > > > added support for "all" platforms to rocksdb and we updated the > dependency > > > to > > > get this fix in 2.6. See KAFKA-9225 > > > <https://issues.apache.org/jira/browse/KAFKA-9225> > > > > > > If you already were running 2.6, then, that's unfortunate. You might > have > > > some luck > > > asking the rocksdb folks if all else fails > > > > > > On Mon, Aug 24, 2020 at 5:46 PM John Roesler <vvcep...@apache.org> > wrote: > > > > > > > Hi Steve, > > > > > > > > Which version of Streams is this? I vaguely recall that we updated > to a > > > > version of Rocks that’s compiled for ARM, and I think some people > have > > > used > > > > it on ARM, but I might be misremembering. > > > > > > > > I’m afraid I can’t be much help in debugging this, but maybe some > others > > > > on the list have more context. If all else fails, you can probably > narrow > > > > it down to the Java RocksDB library. If you create a standalone java > > > > program using the same rocks dependency that we do, then you can > pretty > > > > confidently raise it with the Rocks folks. > > > > > > > > If you want to side-step the issue while debugging this, plugging in > a > > > > different store implementation (like the in-memory one) would > probably > > > > work, although I’m sure memory is scarce on a Raspberry Pi. > > > > > > > > I hope this helps, > > > > -John > > > > > > > > On Mon, Aug 24, 2020, at 19:26, Steve Jones wrote: > > > > > I'm trying to install Kafka Streams on a Raspberry PI, it works > fine > > > as a > > > > > broker, works fine as both a producer and consumer, but when I try > and > > > > run > > > > > streams on the PI rather than on the Mac there is a linker issue: > > > > > > > > > > Exception in thread > > > > > "main-broker-f53264a1-0c70-445f-bf3f-bf634a9a1ed2-StreamThread-1" > > > > > java.lang.UnsatisfiedLinkError: > > > > /tmp/librocksdbjni15158764823832728522.so: > > > > > /tmp/librocksdbjni15158764823832728522.so: cannot open shared > object > > > > file: > > > > > No such file or directory (Possible cause: can't load IA 32 .so on > a > > > ARM > > > > > platform) > > > > > > > > > > at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native > Method) > > > > > > > > > > at > > > > > java.base/java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2452) > > > > > > > > > > at > > > > > > > > > > > > > java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2508) > > > > > > > > > > at > java.base/java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2704) > > > > > > > > > > at > java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2637) > > > > > > > > > > > > > > > I've recompiled rocksdb on the RaspberryPI and added that to the > > > loadpath > > > > > but the same error occurs. I've done the Google search but not > found > > > > > anything around streams on ARM (Raspberry PI) and what needs to be > > > > > recompiled/linked for this to work. > > > > > > > > > > > > > > > Help appreciated. > > > > > > > > > > > > > > > Steve Jones > > > > > > > > > > > > > > >
