comphead commented on code in PR #7544: URL: https://github.com/apache/datafusion-comet/pull/7544#discussion_r4645586104
########## docs/source/contributor-guide/native_executors.md: ########## @@ -0,0 +1,133 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Comet Native Executors + +**Experimental.** Comet Native Executors run whole Spark queries without a JVM in the data plane. A +Spark driver plans each query with Catalyst and Comet, then submits the plan as a graph of stages +to a cluster of one `comet-scheduler` and any number of `comet-executor` processes, built on +[Apache DataFusion Ballista](https://github.com/apache/datafusion-ballista). Each stage is a +_block_: a run of Comet operators that Comet serializes as one Rust plan, with the exchanges between +blocks as the edges of the graph. The design is in +[issue #7288](https://github.com/apache/datafusion-comet/issues/7288), and +[`native/native-executors/README.md`](https://github.com/apache/datafusion-comet/blob/main/native/native-executors/README.md) +describes the cluster, how to build it and how to test it. + +This page is for contributors. It covers the Spark driver side: how to run a query on a cluster and +what does not work yet. The driver never links Ballista. It sends the stage graph to the +scheduler's Arrow Flight service through a small client in libcomet, and the Spark tasks that read +the results use the same client. The cluster features that some queries rely on are in +[#7289](https://github.com/apache/datafusion-comet/pull/7289) and are not part of the cluster in +this repository yet. [Current limitations](#current-limitations) says which. + +## Running it + +1. Build libcomet with the `native-executors` Cargo feature. It is off by default, so the standard + build links no gRPC client. A libcomet without it fails the first query with an error that names + the feature. For example, `make COMET_FEATURES=native-executors` builds libcomet and the Comet + jar that bundles it. The Spark executors that read the results need this libcomet too. +2. Build and start the cluster. The `comet-scheduler` and `comet-executor` binaries are their own + Cargo workspace, so the main Comet build does not depend on Ballista. The cluster and the driver + must come from the same Comet build, which the scheduler checks. See + [`native/native-executors/README.md`](https://github.com/apache/datafusion-comet/blob/main/native/native-executors/README.md) + for how to build the binaries and for the environment variables that configure them. +3. Point Spark at the scheduler's Flight service and turn the feature on: + + ```shell + --conf spark.comet.nativeExecutors.enabled=true \ + --conf spark.comet.nativeExecutors.scheduler.url=http://scheduler-host:50060 + ``` + + Both configs are internal while the feature is experimental. Comet itself must be enabled as + usual, with native shuffle (`spark.comet.shuffle.mode=native`, as in the + [installation guide](../user-guide/latest/installation.md)), because the driver turns only + native shuffle exchanges into the edges of the stage graph. + +4. Check that the query ran on the cluster: its physical plan has a `CometNativeExecutors` leaf + (the `CometNativeExecutorsExec` class) where the Comet plan would otherwise be. A query that + cannot run entirely in the Comet pipeline and on the cluster fails instead, with a message that + names the operator and Comet's reasons for it. + +`CometNativeExecutorsSuite` runs queries against a cluster it starts from the binaries in +`COMET_NATIVE_EXECUTORS_BIN_DIR`. Its header comment gives the commands. It is not part of CI. Its +tests that need a cluster are skipped when the variable is not set. The tests that check what the +driver refuses to plan run without a cluster, but they still need a libcomet built with the +`native-executors` feature. Its TPC-H tests need the cluster of Review Comment: This is out of date since the refusal tests moved to `CometNativeExecutorsPlanSuite`. Every test in `CometNativeExecutorsSuite` is now a `nativeExecutorsTest` and needs the cluster, and the refusals need neither the cluster nor the feature. Could this sentence go, so the paragraph sends readers to the plan suite for the refusals? ########## spark/src/main/scala/org/apache/spark/sql/comet/CometNativeExecutorsExec.scala: ########## @@ -0,0 +1,669 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.spark.sql.comet + +import java.util.IdentityHashMap + +import scala.collection.mutable.ArrayBuffer +import scala.jdk.CollectionConverters._ + +import org.apache.spark.{InterruptibleIterator, Partition, SparkContext, TaskContext} +import org.apache.spark.rdd.RDD +import org.apache.spark.sql.catalyst.InternalRow +import org.apache.spark.sql.catalyst.expressions.{Attribute, DynamicPruning, DynamicPruningExpression, Expression, Literal} +import org.apache.spark.sql.catalyst.plans.QueryPlan +import org.apache.spark.sql.catalyst.plans.physical.{HashPartitioning, RangePartitioning, SinglePartition} +import org.apache.spark.sql.comet.execution.shuffle.{CometNativeShuffle, CometShuffleExchangeExec} +import org.apache.spark.sql.execution.{ColumnarToRowExec, CommandResultExec, DeserializeToObjectExec, LeafExecNode, ScalarSubquery, SparkPlan} +import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanExec, QueryStageExec} +import org.apache.spark.sql.execution.command.{DataWritingCommandExec, ExecutedCommandExec} +import org.apache.spark.sql.execution.datasources.v2.V2CommandExec +import org.apache.spark.sql.execution.exchange.{Exchange, ReusedExchangeExec} +import org.apache.spark.sql.internal.SQLConf +import org.apache.spark.sql.types.StructField +import org.apache.spark.sql.vectorized.ColumnarBatch + +import com.google.protobuf.ByteString +import com.google.protobuf.Descriptors.FieldDescriptor +import com.google.protobuf.Message + +import org.apache.comet.{CometConf, CometExecIterator, CometExplainInfo, CometRuntimeException, NativeExecutorsClient} +import org.apache.comet.serde.{ExprOuterClass, PartitioningOuterClass, QueryPlanSerde} +import org.apache.comet.serde.Config.ConfigMap +import org.apache.comet.serde.NativeExecutors.{QueryRequest, ResultEndpoint, ResultEndpoints, Stage, StageInput} +import org.apache.comet.serde.OperatorOuterClass +import org.apache.comet.serde.OperatorOuterClass.Operator +import org.apache.comet.serde.operator.schema2Proto +import org.apache.comet.vector.NativeUtil + +/** + * Runs a whole query on a Comet Native Executor cluster instead of on Spark's executors. See + * https://github.com/apache/datafusion-comet/issues/7288. + * + * This leaf replaces the plan Comet would otherwise execute. Planning only checks that the plan + * can run on a cluster ([[CometNativeExecutors.validate]]). `doExecuteColumnar` builds the query, + * which lists the files the scans read, submits it to the cluster's scheduler and waits for it on + * the driver; each partition of the RDD it returns then reads one result partition from the + * executor that holds it. + * + * @param nativePlan + * the plan Comet would execute, which [[CometNativeExecutors.buildQuery]] turns into the + * `QueryRequest` the cluster runs + * @param subqueryResults + * the query's scalar subqueries. They are expressions of this node, so Spark runs them before + * it executes the node, and their results replace them in the query before it is submitted. + */ +case class CometNativeExecutorsExec( + override val output: Seq[Attribute], + nativePlan: SparkPlan, + schedulerUrl: String, + subqueryResults: Seq[Expression] = Nil) + extends LeafExecNode { + + override def supportsColumnar: Boolean = true + + override protected def doExecute(): RDD[InternalRow] = + throw new UnsupportedOperationException(s"$nodeName only produces columnar output") + + override protected def doExecuteColumnar(): RDD[ColumnarBatch] = { + NativeExecutorsClient.requireNativeSupport() + val request = CometNativeExecutors.withSubqueryResults( + CometNativeExecutors.buildQuery(nativePlan), + subqueryResults.collect { case sub: ScalarSubquery => sub }) + val endpoints = ResultEndpoints.parseFrom( + new NativeExecutorsClient().submit(schedulerUrl, request.toByteArray)) + new CometNativeExecutorsRDD( + sparkContext, + endpoints.getEndpointsList.asScala.toSeq, + output.length) + } + + override def stringArgs: Iterator[Any] = + Iterator(output, schedulerUrl) + + // Explain prints the plan the cluster runs under this leaf, as `InMemoryTableScanExec` prints + // its cached plan. + override def innerChildren: Seq[QueryPlan[_]] = Seq(nativePlan) ++ super.innerChildren +} + +private class CometNativeExecutorsPartition(override val index: Int, val endpoint: ResultEndpoint) + extends Partition + +/** + * The result partitions of a query that ran on a Comet Native Executor cluster, each read from + * the executor that holds it. + */ +private class CometNativeExecutorsRDD( + sc: SparkContext, + endpoints: Seq[ResultEndpoint], + numColumns: Int) + extends RDD[ColumnarBatch](sc, Nil) { + + override protected def getPartitions: Array[Partition] = + endpoints.zipWithIndex.map { case (endpoint, index) => + new CometNativeExecutorsPartition(index, endpoint): Partition + }.toArray + + override def compute(split: Partition, context: TaskContext): Iterator[ColumnarBatch] = { + val partition = split.asInstanceOf[CometNativeExecutorsPartition] + // A Spark executor needs a libcomet built with the feature too. + NativeExecutorsClient.requireNativeSupport() + val client = new NativeExecutorsClient() + val nativeUtil = new NativeUtil() + // The task's completion listener closes nativeUtil, but only once there is one to register. + val handle = + try { + client.openPartition( + partition.endpoint.getLocation, + partition.endpoint.getTicket.toByteArray, + partition.endpoint.getExpirationTimeMillis) + } catch { + case t: Throwable => + nativeUtil.close() + throw t + } + val batches = new Iterator[ColumnarBatch] with AutoCloseable { + private var current: Option[ColumnarBatch] = None + private var pending: Option[ColumnarBatch] = None + private var closed = false + + override def hasNext: Boolean = { + if (pending.isEmpty && !closed) { + // The consumer has moved past the batch it last got. + current.foreach(_.close()) + current = None + pending = nativeUtil.getNextBatch( + numColumns, + (arrays, schemas) => client.nextBatch(handle, arrays, schemas)) + if (pending.isEmpty) { + close() + } + } + pending.isDefined + } + + override def next(): ColumnarBatch = { + if (!hasNext) { + throw new NoSuchElementException("No more batches in this result partition") + } + current = pending + pending = None + current.get + } + + override def close(): Unit = if (!closed) { + closed = true + try { + current.foreach(_.close()) + current = None + pending.foreach(_.close()) + pending = None + } finally { + // Each step runs even if an earlier one throws. + try client.closePartition(handle) + finally nativeUtil.close() + } + } + } + context.addTaskCompletionListener[Unit](_ => batches.close()) + // A killed task stops at the next batch, as Spark's own source RDDs do. + new InterruptibleIterator(context, batches) + } +} + +/** + * Plans queries for Comet Native Executors: turns the plan Comet would execute into the stage + * graph a cluster runs, and refuses any query that cannot run entirely in the Comet pipeline. + */ +object CometNativeExecutors { + + /** + * Messages of native plans that call back into the JVM, which a cluster doesn't have. A scalar + * subquery's `Subquery` is not one of them: the driver runs the subquery and puts its result in + * the plan before submitting it. The native planner could refuse them all instead + * (https://github.com/apache/datafusion-comet/issues/7375). + */ + private val jvmCallbacks = Set("JvmScalarUdf", "ArrowPythonUdf") + + /** + * Whether Comet runs `plan` on a Comet Native Executor cluster. It must be a whole query: not a + * command, not a stage AQE has split off, and not a plan already replaced. The caller checks + * that Comet executes the plan at all. + */ + def applies(conf: SQLConf, plan: SparkPlan, queryStagePrep: Boolean): Boolean = + CometConf.COMET_NATIVE_EXECUTORS_ENABLED.get(conf) && + !(conf.adaptiveExecutionEnabled && !queryStagePrep && plan.isInstanceOf[Exchange]) && + !plan.exists { + case _: CometNativeExecutorsExec | _: AdaptiveSparkPlanExec | _: QueryStageExec | + _: CommandResultExec | _: ExecutedCommandExec | _: DataWritingCommandExec | + _: V2CommandExec => + true + case _ => false + } + + /** + * Replaces `converted`, the plan Comet would execute for `original`, with a leaf that runs it + * on a Comet Native Executor cluster. An operator on top that only turns the query's result + * into JVM objects, as `Dataset.rdd` adds, stays on the driver above the leaf. + */ + def offload(original: SparkPlan, converted: SparkPlan): SparkPlan = { + NativeExecutorsClient.requireNativeSupport() + offloadPlan(original, converted) + } + + private def offloadPlan(original: SparkPlan, converted: SparkPlan): SparkPlan = + converted match { + case toObjects: DeserializeToObjectExec => + toObjects.withNewChildren(Seq(offloadPlan(toObjects.child, toObjects.child))) + case native => + validate(native) + val exec = CometNativeExecutorsExec( + native.output, + native, + CometConf.COMET_NATIVE_EXECUTORS_SCHEDULER_URL.get(), + scalarSubqueries(native)) + native.logicalLink.orElse(original.logicalLink).foreach(exec.setLogicalLink) + exec + } + + /** + * Checks that a cluster can run `plan`, the plan Comet would execute, without running anything: + * it lists no files and resolves no subquery, so it is safe at planning time. + * + * @throws CometRuntimeException + * naming the first operator that cannot run on a cluster + */ + def validate(plan: SparkPlan): Unit = { + stageGraph(plan, withFiles = false) + () + } + + /** + * The query a Comet Native Executor cluster runs for `plan`, the plan Comet would execute: one + * stage per block of Comet operators that run as one Rust plan, with an input for each exchange + * that feeds the block. A stage that reads files has a plan for each partition, naming its + * files, so this lists them and resolves the subqueries in the scans' filters. It belongs at + * execution time. + * + * @throws CometRuntimeException + * naming the first operator that cannot run on a cluster + */ + def buildQuery(plan: SparkPlan): QueryRequest = { + val stages = stageGraph(plan, withFiles = true) + QueryRequest + .newBuilder() + .putAllConf(nativeConf()) + .addAllStages(stages.asJava) + .build() + } + + private def stageGraph(plan: SparkPlan, withFiles: Boolean): Seq[Stage] = { + val builder = new StageGraphBuilder(plan, withFiles) + val root = plan match { + case ColumnarToRowExec(child) => child + case CometColumnarToRowExec(child) => child + case other => other + } + builder.stageOf(root) + builder.stages.toSeq + } + + /** The scalar subqueries of `plan`'s expressions, through exchanges, each once. */ + private[comet] def scalarSubqueries(plan: SparkPlan): Seq[Expression] = { + val found = ArrayBuffer.empty[ScalarSubquery] + def walk(node: SparkPlan): Unit = { + node.expressions.foreach(_.foreach { + case sub: ScalarSubquery if !found.exists(_.exprId == sub.exprId) => found += sub + case _ => + }) + node match { + case reused: ReusedExchangeExec => walk(reused.child) + case _ => node.children.foreach(walk) + } + } + walk(plan) + found.toSeq + } + + /** + * `query` with each scalar subquery's `Subquery` expression replaced by a literal of its + * result. The subqueries must have run. Sending the results alongside the plan would spare + * rewriting it (https://github.com/apache/datafusion-comet/issues/7374). + */ + def withSubqueryResults(query: QueryRequest, subqueries: Seq[ScalarSubquery]): QueryRequest = { + if (subqueries.isEmpty) { + return query + } + val literals: Map[Long, ExprOuterClass.Expr] = subqueries.map { sub => + // The result is already in Catalyst's internal form. + val literal = Literal(sub.eval(null), sub.dataType) + val expr = QueryPlanSerde + .exprToProto(literal, Seq.empty) + .getOrElse(throw new CometRuntimeException( + s"Comet Native Executors cannot pass the result of $sub to the cluster")) + sub.exprId.id -> expr + }.toMap + val replace: PartialFunction[Message, Message] = { + case expr: ExprOuterClass.Expr if expr.hasSubquery => + literals.getOrElse( + expr.getSubquery.getId, + throw new CometRuntimeException( + s"Comet Native Executors found no result for subquery ${expr.getSubquery.getId}")) + } + val request = query.toBuilder + (0 until request.getStagesCount).foreach { i => + val stage = request.getStages(i) + val plan = transformMessages(stage.getPlan)(replace) + // A partition's plan is the stage's with its files added, so it holds a subquery only + // when the stage's plan does. + if (plan ne stage.getPlan) { + val replaced = stage.toBuilder.setPlan(plan.asInstanceOf[Operator]) + (0 until stage.getPartitionPlansCount).foreach { p => + val partitionPlan = Operator.parseFrom(stage.getPartitionPlans(p)) + replaced.setPartitionPlans(p, transformMessages(partitionPlan)(replace).toByteString) + } + request.setStages(i, replaced) + } + } + request.build() + } + + /** + * The messages directly inside `message`, each with how to put a replacement for it in a + * builder of `message`. + */ + private def childMessages( + message: Message): Iterator[(Message, (Message.Builder, Message) => Unit)] = + message.getAllFields.asScala.iterator + .filter { case (field, _) => field.getJavaType == FieldDescriptor.JavaType.MESSAGE } + .flatMap { case (field, value) => + if (field.isRepeated) { + value.asInstanceOf[java.util.List[Message]].asScala.iterator.zipWithIndex.map { + case (child, index) => + ( + child, + (builder: Message.Builder, m: Message) => { + builder.setRepeatedField(field, index, m) + () + }) + } + } else { + Iterator( + ( + value.asInstanceOf[Message], + (builder: Message.Builder, m: Message) => { + builder.setField(field, m) + () + })) + } + } + + /** + * `message` with every message inside it that `pf` matches replaced by what `pf` returns. A + * message nothing inside changed is returned as is. + */ + private def transformMessages(message: Message)( + pf: PartialFunction[Message, Message]): Message = + pf.applyOrElse( + message, + (m: Message) => { + var builder: Message.Builder = null + childMessages(m).foreach { case (child, set) => + val replaced = transformMessages(child)(pf) + if (replaced ne child) { + if (builder == null) { + builder = m.toBuilder + } + set(builder, replaced) + } + } + if (builder == null) m else builder.build() + }) + + /** The configs a Spark task would pass to native code with a plan. */ + private def nativeConf(): java.util.Map[String, String] = { + val conf = new java.util.HashMap[String, String]( + ConfigMap.parseFrom(CometExecIterator.serializeCometSQLConfs()).getEntriesMap) + // A Spark task passes the batch size apart from the other configs. + conf.put(CometConf.COMET_BATCH_SIZE.key, CometConf.COMET_BATCH_SIZE.get().toString) + conf + } + + private val singlePartition = PartitioningOuterClass.Partitioning + .newBuilder() + .setSinglePartition(PartitioningOuterClass.SinglePartition.newBuilder()) + .build() + + /** The planning data of a partition that reads no files. */ + private val noFiles = OperatorOuterClass.NativeScan + .newBuilder() + .setFilePartition(OperatorOuterClass.SparkFilePartition.getDefaultInstance) + .build() + .toByteArray + + private class StageGraphBuilder(query: SparkPlan, withFiles: Boolean) { + val stages: ArrayBuffer[Stage] = ArrayBuffer.empty + private val stageIndexes = new IdentityHashMap[SparkPlan, Integer]() + + /** + * The index of the stage that runs the block of Comet operators rooted at `plan`, adding it + * first. + */ + def stageOf(plan: SparkPlan): Int = { + val known = stageIndexes.get(plan) + if (known != null) { + return known + } + val block = plan match { + case take: CometTakeOrderedAndProjectExec => return takeOrderedStage(take) + case native: CometNativeExec if native.serializedPlanOpt.isDefined => native + case other => fail(other, "it is not a Comet operator that runs in Rust") + } + val scans = ArrayBuffer.empty[CometNativeScanExec] + val inputs = inputsOf(block, scans).map { + case exchange: CometShuffleExchangeExec => + stageInput(exchange.child)(_.setPartitioning(partitioningOf(exchange))) + case broadcast => stageInput(broadcast.children.head)(_.setBroadcast(true)) + } + val blockPlan = Operator.parseFrom(block.serializedPlanOpt.plan.get) + + // A block that reads files runs one copy per partition, each naming its own files. + val (common, perPartition) = + if (withFiles) filesOf(block, scans.toSeq) + else (Map.empty[String, Array[Byte]], Map.empty[String, Array[PartitionPlanData]]) + val partitionPlans = if (perPartition.isEmpty) { + Seq.empty + } else { + val base = new PlanDataInjector.CachedPlanData(blockPlan) + val counts = perPartition.map { case (key, data) => key -> data.length } + require( + counts.values.toSet.size == 1, + s"The scans of one block must have the same number of partitions, but found $counts") + (0 until counts.values.head).map { i => + val partitionData = perPartition.map { case (key, data) => key -> data(i).bytes } + PlanDataInjector.injectPlanData(base, common, partitionData).toByteString + } + } + // The cluster reads it only for a stage that reads no exchange. A stage that does has as + // many partitions as its exchanges deliver. A stage that reads neither and reports no + // partitions, such as an empty range, runs one, which produces no rows, since the cluster + // refuses a stage with none. + val numPartitions = + if (partitionPlans.nonEmpty) partitionPlans.length + else if (inputs.isEmpty) block.outputPartitioning.numPartitions.max(1) Review Comment: This branch reaches more than the empty range. A block whose only scan is an Iceberg scan that plans no tasks gets here too, because `findAllPlanData` returns nothing for that scan and `filesOf` only adds an empty partition for `CometNativeScanExec`. Before this commit the cluster refused that stage with `The stage has no partitions`. Now it runs one partition of the base plan, and that plan's `IcebergScan` has only the stub `common` that `CometIcebergNativeScan.convert` sets (metadata location and scan hash, no `required_schema`). I have not run it, so I don't know whether the cluster returns no rows or fails in `block_schema` or the native planner. Could the `max(1)` apply only to blocks without scans, for example when every leaf is a `CometRangeExec`? Or could the Iceberg scan get an empty partition the way `filesOf` gives one to Parquet? Either way, a test for the empty Iceberg case would pin it down. ########## spark/src/test/scala/org/apache/comet/exec/CometNativeExecutorsSuite.scala: ########## @@ -0,0 +1,402 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.comet.exec + +import java.io.File +import java.net.{ServerSocket, Socket} +import java.nio.file.Files +import java.util.concurrent.TimeUnit + +import scala.util.Try + +import org.apache.spark.SparkConf +import org.apache.spark.sql.{DataFrame, Row} +import org.apache.spark.sql.comet.{CometNativeExecutorsExec, CometNativeExecutorsTestBase} +import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper +import org.apache.spark.sql.functions.{col, count, max, sum} +import org.apache.spark.sql.internal.SQLConf + +import org.apache.comet.CometConf + +/** + * Runs queries on a Comet Native Executor cluster (see + * https://github.com/apache/datafusion-comet/issues/7288): a comet-scheduler and a comet-executor + * started from the binaries in the directory `COMET_NATIVE_EXECUTORS_BIN_DIR` names. The tests + * are skipped when it isn't set. To run them: + * + * {{{ + * cargo build --manifest-path native/Cargo.toml -p datafusion-comet --features native-executors + * CARGO_TARGET_DIR=native/target cargo build --manifest-path native/native-executors/Cargo.toml + * COMET_NATIVE_EXECUTORS_BIN_DIR=native/target/debug ./mvnw test -Dtest=none \ + * -Dsuites=org.apache.comet.exec.CometNativeExecutorsSuite + * }}} + * + * The TPC-H tests need the cluster of https://github.com/apache/datafusion-comet/pull/7289. On + * the cluster in this repository, q1, q6, q11 and q15 pass and the other queries, which join, + * fail while the scheduler plans them. + * + * `CometNativeExecutorsPlanSuite` checks the stage graphs and the refusals without a cluster, and + * runs in CI. + */ +class CometNativeExecutorsSuite + extends CometNativeExecutorsTestBase + with AdaptiveSparkPlanHelper { + + private val binDir: Option[File] = + sys.env.get("COMET_NATIVE_EXECUTORS_BIN_DIR").map(new File(_).getAbsoluteFile) + private var processes: Seq[Process] = Nil + private var schedulerUrl: String = _ + + override protected def sparkConf: SparkConf = { + val conf = super.sparkConf + conf.set(CometConf.COMET_SHUFFLE_MODE.key, "native") + conf.set(SQLConf.SHUFFLE_PARTITIONS.key, "4") + conf + } + + override def beforeAll(): Unit = { + super.beforeAll() + binDir.foreach(startCluster) + } + + override def afterAll(): Unit = { + try { + // The executor first, so that it doesn't report to a scheduler that has gone. + processes.reverse.foreach { process => + process.destroy() + if (!process.waitFor(10, TimeUnit.SECONDS)) { + process.destroyForcibly() + } + } + } finally { + super.afterAll() + } + } + + private def startCluster(dir: File): Unit = { + val schedulerPort = freePort() + val flightPort = freePort() + val executorPort = freePort() + val executorGrpcPort = freePort() + val logs = Files.createTempDirectory("comet-native-executors").toFile + logInfo(s"Comet Native Executor cluster logs: $logs") + start( + new File(dir, "comet-scheduler"), + Map( + "COMET_SCHEDULER_BIND_HOST" -> "127.0.0.1", + "COMET_SCHEDULER_PORT" -> schedulerPort.toString, + "COMET_SCHEDULER_FLIGHT_PORT" -> flightPort.toString), + new File(logs, "scheduler.log")) + waitForPort(schedulerPort) + waitForPort(flightPort) + start( + new File(dir, "comet-executor"), + Map( + "COMET_EXECUTOR_BIND_HOST" -> "127.0.0.1", + "COMET_EXECUTOR_PORT" -> executorPort.toString, + "COMET_EXECUTOR_GRPC_PORT" -> executorGrpcPort.toString, + "COMET_SCHEDULER_HOST" -> "127.0.0.1", + "COMET_SCHEDULER_PORT" -> schedulerPort.toString, + "COMET_EXECUTOR_TASKS" -> "4", + "COMET_EXECUTOR_WORK_DIR" -> new File(logs, "work").getPath), + new File(logs, "executor.log")) + waitForPort(executorPort) + schedulerUrl = s"http://127.0.0.1:$flightPort" + } + + private def start(binary: File, env: Map[String, String], log: File): Unit = { + val builder = new ProcessBuilder(binary.getPath) + env.foreach { case (key, value) => builder.environment().put(key, value) } + builder.redirectErrorStream(true).redirectOutput(log) + processes :+= builder.start() + } + + private def freePort(): Int = { + val socket = new ServerSocket(0) + try socket.getLocalPort + finally socket.close() + } + + private def waitForPort(port: Int): Unit = { + val deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(60) + while (Try(new Socket("127.0.0.1", port).close()).isFailure) { + assert(System.nanoTime() < deadline, s"Nothing is listening on port $port") + Thread.sleep(100) + } + } + + private def nativeExecutorsTest(name: String)(body: => Unit): Unit = test(name) { + assume(binDir.isDefined, "COMET_NATIVE_EXECUTORS_BIN_DIR is not set") + body + } + + private def onCluster(body: => Unit): Unit = withSQLConf( + CometConf.COMET_NATIVE_EXECUTORS_ENABLED.key -> "true", + // Without a cluster, only tests that the planner refuses run, and those never connect. + CometConf.COMET_NATIVE_EXECUTORS_SCHEDULER_URL.key -> + Option(schedulerUrl).getOrElse("http://localhost:50060"))(body) Review Comment: Now that every test here is a `nativeExecutorsTest`, `schedulerUrl` is always set when `onCluster` runs. That makes the `getOrElse("http://localhost:50060")` fallback and the comment about refusal tests dead. The move also left the `withTable` doc comment at line 188 attached to the first test. The one test that still needs no cluster is `a command's plan stays in Spark` (line 389). `CometNativeExecutors.applies` turns down a plan with `ExecutedCommandExec` or `CommandResultExec`, so `offload` and `requireNativeSupport` never run. Could it move to `CometNativeExecutorsPlanSuite` so the command guard runs in CI, as the refusals do now? ########## docs/source/contributor-guide/native_executors.md: ########## @@ -0,0 +1,133 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Comet Native Executors + +**Experimental.** Comet Native Executors run whole Spark queries without a JVM in the data plane. A +Spark driver plans each query with Catalyst and Comet, then submits the plan as a graph of stages +to a cluster of one `comet-scheduler` and any number of `comet-executor` processes, built on +[Apache DataFusion Ballista](https://github.com/apache/datafusion-ballista). Each stage is a +_block_: a run of Comet operators that Comet serializes as one Rust plan, with the exchanges between +blocks as the edges of the graph. The design is in +[issue #7288](https://github.com/apache/datafusion-comet/issues/7288), and +[`native/native-executors/README.md`](https://github.com/apache/datafusion-comet/blob/main/native/native-executors/README.md) +describes the cluster, how to build it and how to test it. + +This page is for contributors. It covers the Spark driver side: how to run a query on a cluster and +what does not work yet. The driver never links Ballista. It sends the stage graph to the +scheduler's Arrow Flight service through a small client in libcomet, and the Spark tasks that read +the results use the same client. The cluster features that some queries rely on are in +[#7289](https://github.com/apache/datafusion-comet/pull/7289) and are not part of the cluster in +this repository yet. [Current limitations](#current-limitations) says which. + +## Running it + +1. Build libcomet with the `native-executors` Cargo feature. It is off by default, so the standard + build links no gRPC client. A libcomet without it fails the first query with an error that names + the feature. For example, `make COMET_FEATURES=native-executors` builds libcomet and the Comet + jar that bundles it. The Spark executors that read the results need this libcomet too. +2. Build and start the cluster. The `comet-scheduler` and `comet-executor` binaries are their own + Cargo workspace, so the main Comet build does not depend on Ballista. The cluster and the driver + must come from the same Comet build, which the scheduler checks. See + [`native/native-executors/README.md`](https://github.com/apache/datafusion-comet/blob/main/native/native-executors/README.md) + for how to build the binaries and for the environment variables that configure them. +3. Point Spark at the scheduler's Flight service and turn the feature on: + + ```shell + --conf spark.comet.nativeExecutors.enabled=true \ + --conf spark.comet.nativeExecutors.scheduler.url=http://scheduler-host:50060 + ``` + + Both configs are internal while the feature is experimental. Comet itself must be enabled as + usual, with native shuffle (`spark.comet.shuffle.mode=native`, as in the + [installation guide](../user-guide/latest/installation.md)), because the driver turns only + native shuffle exchanges into the edges of the stage graph. + +4. Check that the query ran on the cluster: its physical plan has a `CometNativeExecutors` leaf + (the `CometNativeExecutorsExec` class) where the Comet plan would otherwise be. A query that + cannot run entirely in the Comet pipeline and on the cluster fails instead, with a message that + names the operator and Comet's reasons for it. + +`CometNativeExecutorsSuite` runs queries against a cluster it starts from the binaries in +`COMET_NATIVE_EXECUTORS_BIN_DIR`. Its header comment gives the commands. It is not part of CI. Its +tests that need a cluster are skipped when the variable is not set. The tests that check what the +driver refuses to plan run without a cluster, but they still need a libcomet built with the +`native-executors` feature. Its TPC-H tests need the cluster of +[#7289](https://github.com/apache/datafusion-comet/pull/7289): on the cluster in this repository, +only q1, q6, q11 and q15 pass. `CometNativeExecutorsPlanSuite` runs in CI. It takes the plan Comet +would execute and checks the stage graph, the queries the driver refuses, and how a scalar +subquery's result goes into the plan, with neither a cluster nor the feature. + +## Current limitations + +Comet Native Executors are a first version. The sub-issues of +[#7288](https://github.com/apache/datafusion-comet/issues/7288) track what is missing: + +- Whole queries only. See [below](#whole-queries-only). +- Only native shuffle exchanges. A query whose exchanges use JVM shuffle is refused. +- `LIMIT` and `show()`: [#7365](https://github.com/apache/datafusion-comet/issues/7365). +- Global sorts. The cluster in this repository reads a range exchange as one partition, so every + global sort runs in a single task. + [#7289](https://github.com/apache/datafusion-comet/pull/7289) adds a cluster that sorts the + partitions in parallel and merges them, except for sorts by floats or nested types, which still + run in a single task: [#7366](https://github.com/apache/datafusion-comet/issues/7366). +- Hash partitioning is DataFusion's, not Spark-compatible: + [#7368](https://github.com/apache/datafusion-comet/issues/7368). +- A small broadcast side runs in every task that reads it: + [#7369](https://github.com/apache/datafusion-comet/issues/7369). Moving a join whose build side + turns out small into the stage that produces its probe side is in + [#7289](https://github.com/apache/datafusion-comet/pull/7289). +- Dynamic partition pruning: [#7370](https://github.com/apache/datafusion-comet/issues/7370). A scan + with a dynamic pruning filter is refused. +- Writes: [#7371](https://github.com/apache/datafusion-comet/issues/7371). Commands and writes stay + in Spark. +- Cancellation: [#7372](https://github.com/apache/datafusion-comet/issues/7372). The driver thread + waits for the cluster in native code until the query finishes, so cancelling the Spark job or + interrupting the thread does not stop the wait, and a killed Spark job leaves its query running + on the cluster. A killed Spark task that reads a result partition stops at the next batch; a + read blocked waiting for a batch still waits for it. +- Errors: [#7621](https://github.com/apache/datafusion-comet/issues/7621). A failure on the + cluster reaches Spark as a `CometNativeException` with the cluster's message, not as the exception + Spark would raise. For example, a division by zero with ANSI mode on is not a + `SparkArithmeticException` with the `DIVIDE_BY_ZERO` error class. +- Spark UI metrics: [#7373](https://github.com/apache/datafusion-comet/issues/7373). +- Security. The scheduler and the executors have no authentication and no TLS, and a query names + the files and object stores the executors read. Run a cluster only on a trusted network. +- Result expiry. A query's results stay on the cluster for + `COMET_SCHEDULER_RESULT_RETENTION_SECONDS` after it ends (five minutes by default), whether or + not Spark has read them. The scheduler gives each result partition its expiration time, and a + Spark task that reads a partition after that fails with a message that names the setting. Review Comment: What happens when a cluster executor that holds result partitions dies before Spark reads them? `CometNativeExecutorsRDD` has no dependencies (`RDD[ColumnarBatch](sc, Nil)`), and each partition keeps the `ResultEndpoint` it got at submission. So every retry of a read task goes back to the same executor and fails, and after `spark.task.maxFailures` attempts the job fails. Nothing resubmits the query. In Spark, a lost shuffle output raises `FetchFailedException` and the scheduler reruns the map stage. Could Current limitations say this next to result expiry, with a sub-issue of https://github.com/apache/datafusion-comet/issues/7288 if one is planned? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
