Hi Yun, Yes, I will check how these classes are getting included.
I do see the below coming in from a Kafka-Streams dependency being used for another module. org.rocksdb:rocksdbjni:jar:5.18.4:compile Thanks, Sandip > On 10-Aug-2021, at 8:22 AM, Yun Tang <myas...@live.com> wrote: > > Hi Sandeep, > > I'm afraid that it's because the configuration of your maven plugins like > maven-assembly or maven-shaded to include the classes. > > Best > Yun Tang > From: Sandeep khanzode <sand...@shiftright.ai> > Sent: Saturday, August 7, 2021 10:20 > To: Yun Tang <myas...@live.com> > Cc: Stephan Ewen <se...@apache.org>; user <user@flink.apache.org> > Subject: Re: Bloom Filter - RocksDB - LinkageError Classloading > > Hi Yun, > > You are right. This is what I see in my application JAR … > > Does this mean that the Rocks DB dependency is still getting packaged in my > application JAR and this is what is causing the issue? > > I have made the scope as Provided for that dependency. Is it being > transitively pulled in? All my Flink dependencies are set to Provided in > Maven. > > <Screenshot 2021-08-07 at 7.47.58 AM.png> > > <Screenshot 2021-08-07 at 7.48.42 AM.png> > >> On 06-Aug-2021, at 7:58 AM, Yun Tang <myas...@live.com >> <mailto:myas...@live.com>> wrote: >> >> Hi Sandeep, >> >> If you set the flink-statebackend-rocksdb as provided scope, it should not >> include the org.rocksdb classes, have you ever checked your application jar >> package directly just as what I described? >> >> >> Best >> Yun Tang >> From: Sandeep khanzode <sand...@shiftright.ai <mailto:sand...@shiftright.ai>> >> Sent: Friday, August 6, 2021 2:04 >> To: Stephan Ewen <se...@apache.org <mailto:se...@apache.org>> >> Cc: user <user@flink.apache.org <mailto:user@flink.apache.org>>; Yun Tang >> <myas...@live.com <mailto:myas...@live.com>> >> Subject: Re: Bloom Filter - RocksDB - LinkageError Classloading >> >> Hello Stephan, Yun, >> >> Thanks for your insights. >> >> All I have added is this: >> <dependency> >> <groupId>org.apache.flink</groupId> >> <artifactId>flink-statebackend-rocksdb_2.12</artifactId> >> <version>${flink.version}</version> >> <scope>provided</scope> >> </dependency> >> >> No other library explicitly added. I am assuming, as mentioned, is that the >> flink-dist.jar already contains the relevant classes and the App or parent >> class loader loads the Rocks DB classes. All other Flink dependencies are >> packaged as Maven - Provided. >> >> Moving to parent-first gives the Spring Framework serialisation issues … I >> will take a look at that … >> >> I thought it would be simpler to simply specify Bloom Filters as an option … >> >> Maybe, I will have to remove Spring dependency … >> >> >> Thanks, >> Sandip >> >> >> >>> On 05-Aug-2021, at 5:55 PM, Stephan Ewen <se...@apache.org >>> <mailto:se...@apache.org>> wrote: >>> >>> @Yun Tang >>> >>> Our FRocksDB has the same java package names (org.rocksdb.). Adding >>> 'org.rocksdb' to parent-first patterns ensures it will be loaded only once, >>> and not accidentally multiple times (as Child-first classloading does). >>> >>> The RocksDB code here is a bit like Flink internal components, which we >>> always force parent-first (everything that starts with "org.apache.fink."). >>> >>> To use RocksDB from the application jar, I think you would need to remove >>> the RocksDB state backend from the classpath (lib folder), or you get >>> exactly the error reported above. >>> >>> I cannot think of a downside to add RocksDB to the parent first patterns. >>> >>> On Thu, Aug 5, 2021 at 10:04 AM Yun Tang <myas...@live.com >>> <mailto:myas...@live.com>> wrote: >>> Hi Stephan, >>> >>> Since we use our own FRocksDB instead of the original RocksDB as >>> dependency, I am not sure whether this problem has relationship with this. >>> From my knowledge, more customers would include Flink classes within the >>> application jar package, and it might cause problems if the client has >>> different flink version with servers. >>> >>> >>> Best, >>> Yun Tang >>> From: Stephan Ewen <se...@apache.org <mailto:se...@apache.org>> >>> Sent: Wednesday, August 4, 2021 19:10 >>> To: Yun Tang <myas...@live.com <mailto:myas...@live.com>> >>> Cc: Sandeep khanzode <sand...@shiftright.ai >>> <mailto:sand...@shiftright.ai>>; user <user@flink.apache.org >>> <mailto:user@flink.apache.org>> >>> Subject: Re: Bloom Filter - RocksDB - LinkageError Classloading >>> >>> @Yun Tang Does it make sense to add RocksDB to the "always parent-first >>> options" to avoid these kind of errors when users package apps incorrectly? >>> My feeling is that these packaging errors occur very frequently. >>> >>> >>> On Wed, Aug 4, 2021 at 10:41 AM Yun Tang <myas...@live.com >>> <mailto:myas...@live.com>> wrote: >>> Hi Sandeep, >>> >>> Did you include the RocksDB classes in the application jar package? You can >>> unpark your jar package to check whether them existed. >>> If so, since RocksDB classes are already included in the flink-dist >>> package, you don't need to include them in your jar package (maybe you >>> explicitly added the dependency of org.rocksdb:rocksdbjni in your pom). >>> >>> Best >>> Yun Tang >>> From: Sandeep khanzode <sand...@shiftright.ai >>> <mailto:sand...@shiftright.ai>> >>> Sent: Wednesday, August 4, 2021 11:54 >>> To: user <user@flink.apache.org <mailto:user@flink.apache.org>> >>> Subject: Bloom Filter - RocksDB - LinkageError Classloading >>> >>> Hello, >>> >>> I tried to add the bloom filter functionality as mentioned here: >>> https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/contrib/streaming/state/RocksDBOptionsFactory.html >>> >>> <https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/contrib/streaming/state/RocksDBOptionsFactory.html> >>> >>> rocksDbBackend.setRocksDBOptions(new RocksDBOptionsFactory() { >>> >>> public DBOptions createDBOptions(DBOptions currentOptions, >>> Collection<AutoCloseable> handlesToClose) { >>> return currentOptions.setMaxOpenFiles(1024); >>> } >>> >>> public ColumnFamilyOptions createColumnOptions(ColumnFamilyOptions >>> currentOptions, Collection<AutoCloseable> handlesToClose) { >>> BloomFilter bloomFilter = new BloomFilter(); >>> handlesToClose.add(bloomFilter); >>> >>> return currentOptions >>> .setTableFormatConfig( >>> new >>> BlockBasedTableConfig().setFilter(bloomFilter)); >>> } >>> }); >>> >>> This is in the main class where we setup in the StreamExecutionEnvironment … >>> >>> I get ClassLoading errors due to that ... >>> Caused by: java.lang.LinkageError: loader constraint violation: loader >>> org.apache.flink.util.ChildFirstClassLoader @1169afe1 wants to load class >>> org.rocksdb.ColumnFamilyOptions. A different class with the same name was >>> previously loaded by 'app'. (org.rocksdb.ColumnFamilyOptions is in unnamed >>> module of loader 'app') >>> >>> >>> What is documented is to change the order to parent-first in the >>> flink-conf.yaml … but then I get different issues for the basic/core Spring >>> Framework classes not being serializable … >>> >>> Any help will be appreciated. >>> >>> Thanks, >>> Sandip
