[
https://issues.apache.org/jira/browse/FLINK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977155#comment-15977155
]
Luke Hutchison commented on FLINK-3328:
---------------------------------------
Right, but shading requires a complete copy of the deps to be shipped inside
the jar. I understand the desire to reduce namespace clashes and to not allow
users to depend upon Flink's deps (or to not expose users to Flink's deps). But
I think it's actually worse to ship a complete copy of all deps (shaded) with a
library, due to increased overall size, if a consumer of the library also uses
the dep.
(Also, in the general case, bundling is almost always a bad idea for security
reasons -- if an urgent patch is needed to a shaded jar, users have to wait for
the producer of the library to re-build with the fix, rather than just
depending upon an ABI-compatible fixed version themselves -- although this is
mitigated here by the fact that Flink is open source, so users can build their
own Flink libraries if they need to. Bundling in general is generally frowned
upon for this reason, e.g. in the situation outside the JVM world where a large
number of programs statically link in OpenSSH and/or zlib, rather than using
the systemwide version of it.)
Apologies if I'm still misunderstanding the issues here.
> Incorrectly shaded dependencies in flink-runtime
> ------------------------------------------------
>
> Key: FLINK-3328
> URL: https://issues.apache.org/jira/browse/FLINK-3328
> Project: Flink
> Issue Type: Bug
> Components: Build System
> Affects Versions: 1.0.0
> Reporter: Stephan Ewen
> Assignee: Robert Metzger
> Priority: Blocker
> Fix For: 1.0.0
>
>
> There are apparently some dependencies shaded into {{flink-runtime}} fat jar
> that are not relocated. (the flink-runtime jar is now 70 MB)
> From the output of the shading in flink-dist, it looks as if this concerns at
> least
> - Zookeeper
> - slf4j
> - jline
> - netty (3.x)
> Possible more.
> {code}
> [WARNING] zookeeper-3.4.6.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 440
> overlapping classes:
> [WARNING] - org.apache.zookeeper.server.NettyServerCnxnFactory
> [WARNING] - org.apache.jute.compiler.JFile
> [WARNING] - org.apache.zookeeper.server.SessionTracker$Session
> [WARNING] - org.apache.zookeeper.server.quorum.AuthFastLeaderElection$1
> [WARNING] - org.apache.jute.compiler.JLong
> [WARNING] - org.apache.zookeeper.client.ZooKeeperSaslClient$SaslState
> [WARNING] - org.apache.zookeeper.server.auth.KerberosName$Rule
> [WARNING] - org.apache.jute.CsvOutputArchive
> [WARNING] - org.apache.zookeeper.server.quorum.QuorumPeer
> [WARNING] - org.apache.zookeeper.ZooKeeper$DataWatchRegistration
> [WARNING] - 430 more...
> [WARNING] slf4j-api-1.7.7.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 24
> overlapping classes:
> [WARNING] - org.slf4j.spi.MarkerFactoryBinder
> [WARNING] - org.slf4j.helpers.SubstituteLogger
> [WARNING] - org.slf4j.helpers.BasicMarker
> [WARNING] - org.slf4j.helpers.Util
> [WARNING] - org.slf4j.LoggerFactory
> [WARNING] - org.slf4j.Marker
> [WARNING] - org.slf4j.helpers.NamedLoggerBase
> [WARNING] - org.slf4j.Logger
> [WARNING] - org.slf4j.spi.LocationAwareLogger
> [WARNING] - org.slf4j.ILoggerFactory
> [WARNING] - 14 more...
> [WARNING] jansi-1.4.jar, jline-2.10.4.jar define 23 overlapping classes:
> [WARNING] - org.fusesource.jansi.Ansi$Erase
> [WARNING] - org.fusesource.jansi.Ansi
> [WARNING] - org.fusesource.jansi.AnsiOutputStream
> [WARNING] - org.fusesource.jansi.internal.CLibrary
> [WARNING] - org.fusesource.jansi.Ansi$2
> [WARNING] - org.fusesource.jansi.WindowsAnsiOutputStream
> [WARNING] - org.fusesource.jansi.AnsiRenderer$Code
> [WARNING] - org.fusesource.jansi.AnsiConsole
> [WARNING] - org.fusesource.jansi.Ansi$Attribute
> [WARNING] - org.fusesource.jansi.internal.Kernel32
> [WARNING] - 13 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-collections-3.2.2.jar,
> commons-beanutils-1.7.0.jar define 10 overlapping classes:
> [WARNING] - org.apache.commons.collections.FastHashMap$EntrySet
> [WARNING] - org.apache.commons.collections.ArrayStack
> [WARNING] - org.apache.commons.collections.FastHashMap$1
> [WARNING] - org.apache.commons.collections.FastHashMap$KeySet
> [WARNING] - org.apache.commons.collections.FastHashMap$CollectionView
> [WARNING] - org.apache.commons.collections.BufferUnderflowException
> [WARNING] - org.apache.commons.collections.Buffer
> [WARNING] -
> org.apache.commons.collections.FastHashMap$CollectionView$CollectionViewIterator
> [WARNING] - org.apache.commons.collections.FastHashMap$Values
> [WARNING] - org.apache.commons.collections.FastHashMap
> [WARNING] flink-streaming-scala_2.10-1.0-SNAPSHOT.jar,
> flink-core-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar,
> flink-java-1.0-SNAPSHOT.jar, flink-streaming-java_2.10-1.0-SNAPSHOT.jar,
> flink-scala_2.10-1.0-SNAPSHOT.jar, flink-clients_2.10-1.0-SNAPSHOT.jar,
> flink-optimizer_2.10-1.0-SNAPSHOT.jar,
> flink-runtime-web_2.10-1.0-SNAPSHOT.jar define 1690 overlapping classes:
> [WARNING] -
> org.apache.flink.shaded.com.google.common.collect.LinkedListMultimap
> [WARNING] -
> org.apache.flink.shaded.com.google.common.io.ByteSource$AsCharSource
> [WARNING] - org.apache.flink.shaded.com.google.common.escape.Platform
> [WARNING] -
> org.apache.flink.shaded.com.google.common.util.concurrent.Futures$ImmediateFailedCheckedFuture
> [WARNING] -
> org.apache.flink.shaded.com.google.common.primitives.SignedBytes$LexicographicalComparator
> [WARNING] -
> org.apache.flink.shaded.com.google.common.cache.LocalCache$WriteQueue$2
> [WARNING] - org.apache.flink.shaded.com.google.common.escape.Escaper$1
> [WARNING] -
> org.apache.flink.shaded.com.google.common.collect.MultimapBuilder$SetMultimapBuilder
> [WARNING] -
> org.apache.flink.shaded.com.google.common.collect.Ordering$ArbitraryOrdering
> [WARNING] -
> org.apache.flink.shaded.com.google.common.collect.Synchronized$SynchronizedAsMapEntries$1
> [WARNING] - 1680 more...
> [WARNING] flink-scala_2.10-1.0-SNAPSHOT.jar, flink-java-1.0-SNAPSHOT.jar,
> flink-streaming-scala_2.10-1.0-SNAPSHOT.jar,
> flink-runtime_2.10-1.0-SNAPSHOT.jar define 25 overlapping classes:
> [WARNING] - org.apache.flink.shaded.org.objectweb.asm.Context
> [WARNING] - org.apache.flink.shaded.org.objectweb.asm.FieldWriter
> [WARNING] - org.apache.flink.shaded.org.objectweb.asm.TypePath
> [WARNING] - org.apache.flink.shaded.org.objectweb.asm.Handler
> [WARNING] - org.apache.flink.shaded.org.objectweb.asm.TypeReference
> [WARNING] -
> org.apache.flink.shaded.org.objectweb.asm.signature.SignatureVisitor
> [WARNING] - org.apache.flink.shaded.org.objectweb.asm.Frame
> [WARNING] - org.apache.flink.shaded.org.objectweb.asm.FieldVisitor
> [WARNING] - org.apache.flink.shaded.org.objectweb.asm.ByteVector
> [WARNING] - org.apache.flink.shaded.org.objectweb.asm.ClassVisitor
> [WARNING] - 15 more...
> [WARNING] jline-0.9.94.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 35
> overlapping classes:
> [WARNING] - jline.ArgumentCompletor$ArgumentList
> [WARNING] - jline.UnsupportedTerminal
> [WARNING] - jline.Terminal
> [WARNING] - jline.WindowsTerminal$ReplayPrefixOneCharInputStream
> [WARNING] - jline.History
> [WARNING] - jline.WindowsTerminal$1
> [WARNING] - jline.ConsoleReader
> [WARNING] - jline.ClassNameCompletor
> [WARNING] - jline.SimpleCompletor$SimpleCompletorFilter
> [WARNING] - jline.CandidateCycleCompletionHandler
> [WARNING] - 25 more...
> [WARNING] netty-3.8.0.Final.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define
> 810 overlapping classes:
> [WARNING] -
> org.jboss.netty.handler.codec.http.websocketx.WebSocketClientHandshaker
> [WARNING] - org.jboss.netty.handler.codec.base64.Base64Decoder
> [WARNING] - org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink$1
> [WARNING] - org.jboss.netty.util.VirtualExecutorService
> [WARNING] - org.jboss.netty.util.DefaultObjectSizeEstimator
> [WARNING] -
> org.jboss.netty.util.internal.ConcurrentIdentityHashMap$HashEntry
> [WARNING] - org.jboss.netty.channel.socket.oio.OioDatagramChannel
> [WARNING] - org.jboss.netty.logging.InternalLoggerFactory
> [WARNING] - org.jboss.netty.handler.codec.spdy.DefaultSpdyDataFrame
> [WARNING] - org.jboss.netty.channel.LifeCycleAwareChannelHandler
> [WARNING] - 800 more...
> [WARNING] flink-java-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar
> define 141 overlapping classes:
> [WARNING] - org.apache.flink.shaded.org.objectweb.asm.tree.ClassNode
> [WARNING] -
> org.apache.flink.shaded.org.objectweb.asm.commons.JSRInlinerAdapter$Instantiation
> [WARNING] -
> org.apache.flink.shaded.org.objectweb.asm.tree.analysis.BasicValue
> [WARNING] -
> org.apache.flink.shaded.org.objectweb.asm.xml.Processor$SingleDocElement
> [WARNING] -
> org.apache.flink.shaded.org.objectweb.asm.tree.TypeAnnotationNode
> [WARNING] -
> org.apache.flink.shaded.org.objectweb.asm.commons.CodeSizeEvaluator
> [WARNING] -
> org.apache.flink.shaded.org.objectweb.asm.xml.Processor$SAXWriter
> [WARNING] - org.apache.flink.shaded.org.objectweb.asm.util.TraceFieldVisitor
> [WARNING] -
> org.apache.flink.shaded.org.objectweb.asm.xml.Processor$ContentHandlerFactory
> [WARNING] -
> org.apache.flink.shaded.org.objectweb.asm.xml.ASMContentHandler$FrameTypeRule
> [WARNING] - 131 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-beanutils-1.7.0.jar
> define 82 overlapping classes:
> [WARNING] - org.apache.commons.beanutils.Converter
> [WARNING] - org.apache.commons.beanutils.WrapDynaBean
> [WARNING] - org.apache.commons.beanutils.converters.IntegerConverter
> [WARNING] - org.apache.commons.beanutils.locale.LocaleBeanUtilsBean
> [WARNING] -
> org.apache.commons.beanutils.locale.converters.DoubleLocaleConverter
> [WARNING] -
> org.apache.commons.beanutils.locale.converters.DecimalLocaleConverter
> [WARNING] - org.apache.commons.beanutils.converters.ShortConverter
> [WARNING] - org.apache.commons.beanutils.converters.StringArrayConverter
> [WARNING] - org.apache.commons.beanutils.locale.LocaleConvertUtilsBean
> [WARNING] - org.apache.commons.beanutils.LazyDynaClass
> [WARNING] - 72 more...
> [WARNING] commons-lang3-3.3.2.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define
> 217 overlapping classes:
> [WARNING] - org.apache.commons.lang3.builder.DiffResult
> [WARNING] - org.apache.commons.lang3.CharRange
> [WARNING] -
> org.apache.commons.lang3.builder.ToStringStyle$ShortPrefixToStringStyle
> [WARNING] - org.apache.commons.lang3.concurrent.ConcurrentException
> [WARNING] - org.apache.commons.lang3.builder.DiffBuilder$1
> [WARNING] - org.apache.commons.lang3.builder.DiffBuilder
> [WARNING] - org.apache.commons.lang3.builder.Diff
> [WARNING] - org.apache.commons.lang3.time.FastDatePrinter$TwoDigitYearField
> [WARNING] - org.apache.commons.lang3.ObjectUtils$Null
> [WARNING] - org.apache.commons.lang3.reflect.MemberUtils
> [WARNING] - 207 more...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
