Re: [GraphX] Preserving Partitions when reading from HDFS

;33554432")` to tune the partition size when reading from HDFS. > > Thanks, > Manu Zhang > > On Mon, Apr 15, 2019 at 11:28 PM M Bilal wrote: > >> Hi, >> >> I have implemented a custom partitioning algorithm to partition graphs in >> GraphX. Saving the

[GraphX] Preserving Partitions when reading from HDFS

Hi, I have implemented a custom partitioning algorithm to partition graphs in GraphX. Saving the partitioning graph (the edges) to HDFS creates separate files in the output folder with the number of files equal to the number of Partitions. However, reading back the edges creates number of partiti

Re: Observing DAGScheduler Log Messages

i > > https://about.me/JacekLaskowski > Mastering Spark SQL https://bit.ly/mastering-spark-sql > Spark Structured Streaming https://bit.ly/spark-structured-streaming > Mastering Kafka Streams https://bit.ly/mastering-kafka-streams > Follow me at https://twitter.com/jaceklaskowski > &

Observing DAGScheduler Log Messages

Hi, I want to observe the log messages from DAGScheduler in Apache Spark. Which log files do I need to check. I have tried observing the driver logs and worker stderr logs but I can't find any messages that are from that class. I am using Spark 3.0.0 snapshot in standalone mode. Thanks. Regard