binary stream

Hello sorry for my newbie question. can apache spark be used for analyzing binary streaming? such as the streaming of live VOD. Thanks. - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

heap fragmentation in G1

Hi, We occasionally encounter OutOfMemoryError errors when running Spark 3.1 with Java 17, G1 garbage collector (region size = 32MB), and a 200GB heap. The OOM happens in the ShuffleExternalSorter when it attempts to allocate a 1GB array for the pointer array, despite having about 80GB of heap ava

Re: Help wanted on securing spark with Apache Knox / JWT

You need to use the spark.ui.filters setting on the history server https://spark.apache.org/docs/latest/configuration.html#spark-ui: spark.ui.filters=org.apache.hadoop.security.authentication.server.AuthenticationFilter spark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.pa