comphead commented on code in PR #7544:
URL: https://github.com/apache/datafusion-comet/pull/7544#discussion_r4645586104


##########
docs/source/contributor-guide/native_executors.md:
##########
@@ -0,0 +1,133 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Comet Native Executors
+
+**Experimental.** Comet Native Executors run whole Spark queries without a JVM 
in the data plane. A
+Spark driver plans each query with Catalyst and Comet, then submits the plan 
as a graph of stages
+to a cluster of one `comet-scheduler` and any number of `comet-executor` 
processes, built on
+[Apache DataFusion Ballista](https://github.com/apache/datafusion-ballista). 
Each stage is a
+_block_: a run of Comet operators that Comet serializes as one Rust plan, with 
the exchanges between
+blocks as the edges of the graph. The design is in
+[issue #7288](https://github.com/apache/datafusion-comet/issues/7288), and
+[`native/native-executors/README.md`](https://github.com/apache/datafusion-comet/blob/main/native/native-executors/README.md)
+describes the cluster, how to build it and how to test it.
+
+This page is for contributors. It covers the Spark driver side: how to run a 
query on a cluster and
+what does not work yet. The driver never links Ballista. It sends the stage 
graph to the
+scheduler's Arrow Flight service through a small client in libcomet, and the 
Spark tasks that read
+the results use the same client. The cluster features that some queries rely 
on are in
+[#7289](https://github.com/apache/datafusion-comet/pull/7289) and are not part 
of the cluster in
+this repository yet. [Current limitations](#current-limitations) says which.
+
+## Running it
+
+1. Build libcomet with the `native-executors` Cargo feature. It is off by 
default, so the standard
+   build links no gRPC client. A libcomet without it fails the first query 
with an error that names
+   the feature. For example, `make COMET_FEATURES=native-executors` builds 
libcomet and the Comet
+   jar that bundles it. The Spark executors that read the results need this 
libcomet too.
+2. Build and start the cluster. The `comet-scheduler` and `comet-executor` 
binaries are their own
+   Cargo workspace, so the main Comet build does not depend on Ballista. The 
cluster and the driver
+   must come from the same Comet build, which the scheduler checks. See
+   
[`native/native-executors/README.md`](https://github.com/apache/datafusion-comet/blob/main/native/native-executors/README.md)
+   for how to build the binaries and for the environment variables that 
configure them.
+3. Point Spark at the scheduler's Flight service and turn the feature on:
+
+   ```shell
+   --conf spark.comet.nativeExecutors.enabled=true \
+   --conf spark.comet.nativeExecutors.scheduler.url=http://scheduler-host:50060
+   ```
+
+   Both configs are internal while the feature is experimental. Comet itself 
must be enabled as
+   usual, with native shuffle (`spark.comet.shuffle.mode=native`, as in the
+   [installation guide](../user-guide/latest/installation.md)), because the 
driver turns only
+   native shuffle exchanges into the edges of the stage graph.
+
+4. Check that the query ran on the cluster: its physical plan has a 
`CometNativeExecutors` leaf
+   (the `CometNativeExecutorsExec` class) where the Comet plan would otherwise 
be. A query that
+   cannot run entirely in the Comet pipeline and on the cluster fails instead, 
with a message that
+   names the operator and Comet's reasons for it.
+
+`CometNativeExecutorsSuite` runs queries against a cluster it starts from the 
binaries in
+`COMET_NATIVE_EXECUTORS_BIN_DIR`. Its header comment gives the commands. It is 
not part of CI. Its
+tests that need a cluster are skipped when the variable is not set. The tests 
that check what the
+driver refuses to plan run without a cluster, but they still need a libcomet 
built with the
+`native-executors` feature. Its TPC-H tests need the cluster of

Review Comment:
   This is out of date since the refusal tests moved to 
`CometNativeExecutorsPlanSuite`. Every test in `CometNativeExecutorsSuite` is 
now a `nativeExecutorsTest` and needs the cluster, and the refusals need 
neither the cluster nor the feature. Could this sentence go, so the paragraph 
sends readers to the plan suite for the refusals?



##########
spark/src/main/scala/org/apache/spark/sql/comet/CometNativeExecutorsExec.scala:
##########
@@ -0,0 +1,669 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import java.util.IdentityHashMap
+
+import scala.collection.mutable.ArrayBuffer
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.{InterruptibleIterator, Partition, SparkContext, 
TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, DynamicPruning, 
DynamicPruningExpression, Expression, Literal}
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.catalyst.plans.physical.{HashPartitioning, 
RangePartitioning, SinglePartition}
+import org.apache.spark.sql.comet.execution.shuffle.{CometNativeShuffle, 
CometShuffleExchangeExec}
+import org.apache.spark.sql.execution.{ColumnarToRowExec, CommandResultExec, 
DeserializeToObjectExec, LeafExecNode, ScalarSubquery, SparkPlan}
+import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanExec, 
QueryStageExec}
+import org.apache.spark.sql.execution.command.{DataWritingCommandExec, 
ExecutedCommandExec}
+import org.apache.spark.sql.execution.datasources.v2.V2CommandExec
+import org.apache.spark.sql.execution.exchange.{Exchange, ReusedExchangeExec}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructField
+import org.apache.spark.sql.vectorized.ColumnarBatch
+
+import com.google.protobuf.ByteString
+import com.google.protobuf.Descriptors.FieldDescriptor
+import com.google.protobuf.Message
+
+import org.apache.comet.{CometConf, CometExecIterator, CometExplainInfo, 
CometRuntimeException, NativeExecutorsClient}
+import org.apache.comet.serde.{ExprOuterClass, PartitioningOuterClass, 
QueryPlanSerde}
+import org.apache.comet.serde.Config.ConfigMap
+import org.apache.comet.serde.NativeExecutors.{QueryRequest, ResultEndpoint, 
ResultEndpoints, Stage, StageInput}
+import org.apache.comet.serde.OperatorOuterClass
+import org.apache.comet.serde.OperatorOuterClass.Operator
+import org.apache.comet.serde.operator.schema2Proto
+import org.apache.comet.vector.NativeUtil
+
+/**
+ * Runs a whole query on a Comet Native Executor cluster instead of on Spark's 
executors. See
+ * https://github.com/apache/datafusion-comet/issues/7288.
+ *
+ * This leaf replaces the plan Comet would otherwise execute. Planning only 
checks that the plan
+ * can run on a cluster ([[CometNativeExecutors.validate]]). 
`doExecuteColumnar` builds the query,
+ * which lists the files the scans read, submits it to the cluster's scheduler 
and waits for it on
+ * the driver; each partition of the RDD it returns then reads one result 
partition from the
+ * executor that holds it.
+ *
+ * @param nativePlan
+ *   the plan Comet would execute, which [[CometNativeExecutors.buildQuery]] 
turns into the
+ *   `QueryRequest` the cluster runs
+ * @param subqueryResults
+ *   the query's scalar subqueries. They are expressions of this node, so 
Spark runs them before
+ *   it executes the node, and their results replace them in the query before 
it is submitted.
+ */
+case class CometNativeExecutorsExec(
+    override val output: Seq[Attribute],
+    nativePlan: SparkPlan,
+    schedulerUrl: String,
+    subqueryResults: Seq[Expression] = Nil)
+    extends LeafExecNode {
+
+  override def supportsColumnar: Boolean = true
+
+  override protected def doExecute(): RDD[InternalRow] =
+    throw new UnsupportedOperationException(s"$nodeName only produces columnar 
output")
+
+  override protected def doExecuteColumnar(): RDD[ColumnarBatch] = {
+    NativeExecutorsClient.requireNativeSupport()
+    val request = CometNativeExecutors.withSubqueryResults(
+      CometNativeExecutors.buildQuery(nativePlan),
+      subqueryResults.collect { case sub: ScalarSubquery => sub })
+    val endpoints = ResultEndpoints.parseFrom(
+      new NativeExecutorsClient().submit(schedulerUrl, request.toByteArray))
+    new CometNativeExecutorsRDD(
+      sparkContext,
+      endpoints.getEndpointsList.asScala.toSeq,
+      output.length)
+  }
+
+  override def stringArgs: Iterator[Any] =
+    Iterator(output, schedulerUrl)
+
+  // Explain prints the plan the cluster runs under this leaf, as 
`InMemoryTableScanExec` prints
+  // its cached plan.
+  override def innerChildren: Seq[QueryPlan[_]] = Seq(nativePlan) ++ 
super.innerChildren
+}
+
+private class CometNativeExecutorsPartition(override val index: Int, val 
endpoint: ResultEndpoint)
+    extends Partition
+
+/**
+ * The result partitions of a query that ran on a Comet Native Executor 
cluster, each read from
+ * the executor that holds it.
+ */
+private class CometNativeExecutorsRDD(
+    sc: SparkContext,
+    endpoints: Seq[ResultEndpoint],
+    numColumns: Int)
+    extends RDD[ColumnarBatch](sc, Nil) {
+
+  override protected def getPartitions: Array[Partition] =
+    endpoints.zipWithIndex.map { case (endpoint, index) =>
+      new CometNativeExecutorsPartition(index, endpoint): Partition
+    }.toArray
+
+  override def compute(split: Partition, context: TaskContext): 
Iterator[ColumnarBatch] = {
+    val partition = split.asInstanceOf[CometNativeExecutorsPartition]
+    // A Spark executor needs a libcomet built with the feature too.
+    NativeExecutorsClient.requireNativeSupport()
+    val client = new NativeExecutorsClient()
+    val nativeUtil = new NativeUtil()
+    // The task's completion listener closes nativeUtil, but only once there 
is one to register.
+    val handle =
+      try {
+        client.openPartition(
+          partition.endpoint.getLocation,
+          partition.endpoint.getTicket.toByteArray,
+          partition.endpoint.getExpirationTimeMillis)
+      } catch {
+        case t: Throwable =>
+          nativeUtil.close()
+          throw t
+      }
+    val batches = new Iterator[ColumnarBatch] with AutoCloseable {
+      private var current: Option[ColumnarBatch] = None
+      private var pending: Option[ColumnarBatch] = None
+      private var closed = false
+
+      override def hasNext: Boolean = {
+        if (pending.isEmpty && !closed) {
+          // The consumer has moved past the batch it last got.
+          current.foreach(_.close())
+          current = None
+          pending = nativeUtil.getNextBatch(
+            numColumns,
+            (arrays, schemas) => client.nextBatch(handle, arrays, schemas))
+          if (pending.isEmpty) {
+            close()
+          }
+        }
+        pending.isDefined
+      }
+
+      override def next(): ColumnarBatch = {
+        if (!hasNext) {
+          throw new NoSuchElementException("No more batches in this result 
partition")
+        }
+        current = pending
+        pending = None
+        current.get
+      }
+
+      override def close(): Unit = if (!closed) {
+        closed = true
+        try {
+          current.foreach(_.close())
+          current = None
+          pending.foreach(_.close())
+          pending = None
+        } finally {
+          // Each step runs even if an earlier one throws.
+          try client.closePartition(handle)
+          finally nativeUtil.close()
+        }
+      }
+    }
+    context.addTaskCompletionListener[Unit](_ => batches.close())
+    // A killed task stops at the next batch, as Spark's own source RDDs do.
+    new InterruptibleIterator(context, batches)
+  }
+}
+
+/**
+ * Plans queries for Comet Native Executors: turns the plan Comet would 
execute into the stage
+ * graph a cluster runs, and refuses any query that cannot run entirely in the 
Comet pipeline.
+ */
+object CometNativeExecutors {
+
+  /**
+   * Messages of native plans that call back into the JVM, which a cluster 
doesn't have. A scalar
+   * subquery's `Subquery` is not one of them: the driver runs the subquery 
and puts its result in
+   * the plan before submitting it. The native planner could refuse them all 
instead
+   * (https://github.com/apache/datafusion-comet/issues/7375).
+   */
+  private val jvmCallbacks = Set("JvmScalarUdf", "ArrowPythonUdf")
+
+  /**
+   * Whether Comet runs `plan` on a Comet Native Executor cluster. It must be 
a whole query: not a
+   * command, not a stage AQE has split off, and not a plan already replaced. 
The caller checks
+   * that Comet executes the plan at all.
+   */
+  def applies(conf: SQLConf, plan: SparkPlan, queryStagePrep: Boolean): 
Boolean =
+    CometConf.COMET_NATIVE_EXECUTORS_ENABLED.get(conf) &&
+      !(conf.adaptiveExecutionEnabled && !queryStagePrep && 
plan.isInstanceOf[Exchange]) &&
+      !plan.exists {
+        case _: CometNativeExecutorsExec | _: AdaptiveSparkPlanExec | _: 
QueryStageExec |
+            _: CommandResultExec | _: ExecutedCommandExec | _: 
DataWritingCommandExec |
+            _: V2CommandExec =>
+          true
+        case _ => false
+      }
+
+  /**
+   * Replaces `converted`, the plan Comet would execute for `original`, with a 
leaf that runs it
+   * on a Comet Native Executor cluster. An operator on top that only turns 
the query's result
+   * into JVM objects, as `Dataset.rdd` adds, stays on the driver above the 
leaf.
+   */
+  def offload(original: SparkPlan, converted: SparkPlan): SparkPlan = {
+    NativeExecutorsClient.requireNativeSupport()
+    offloadPlan(original, converted)
+  }
+
+  private def offloadPlan(original: SparkPlan, converted: SparkPlan): 
SparkPlan =
+    converted match {
+      case toObjects: DeserializeToObjectExec =>
+        toObjects.withNewChildren(Seq(offloadPlan(toObjects.child, 
toObjects.child)))
+      case native =>
+        validate(native)
+        val exec = CometNativeExecutorsExec(
+          native.output,
+          native,
+          CometConf.COMET_NATIVE_EXECUTORS_SCHEDULER_URL.get(),
+          scalarSubqueries(native))
+        
native.logicalLink.orElse(original.logicalLink).foreach(exec.setLogicalLink)
+        exec
+    }
+
+  /**
+   * Checks that a cluster can run `plan`, the plan Comet would execute, 
without running anything:
+   * it lists no files and resolves no subquery, so it is safe at planning 
time.
+   *
+   * @throws CometRuntimeException
+   *   naming the first operator that cannot run on a cluster
+   */
+  def validate(plan: SparkPlan): Unit = {
+    stageGraph(plan, withFiles = false)
+    ()
+  }
+
+  /**
+   * The query a Comet Native Executor cluster runs for `plan`, the plan Comet 
would execute: one
+   * stage per block of Comet operators that run as one Rust plan, with an 
input for each exchange
+   * that feeds the block. A stage that reads files has a plan for each 
partition, naming its
+   * files, so this lists them and resolves the subqueries in the scans' 
filters. It belongs at
+   * execution time.
+   *
+   * @throws CometRuntimeException
+   *   naming the first operator that cannot run on a cluster
+   */
+  def buildQuery(plan: SparkPlan): QueryRequest = {
+    val stages = stageGraph(plan, withFiles = true)
+    QueryRequest
+      .newBuilder()
+      .putAllConf(nativeConf())
+      .addAllStages(stages.asJava)
+      .build()
+  }
+
+  private def stageGraph(plan: SparkPlan, withFiles: Boolean): Seq[Stage] = {
+    val builder = new StageGraphBuilder(plan, withFiles)
+    val root = plan match {
+      case ColumnarToRowExec(child) => child
+      case CometColumnarToRowExec(child) => child
+      case other => other
+    }
+    builder.stageOf(root)
+    builder.stages.toSeq
+  }
+
+  /** The scalar subqueries of `plan`'s expressions, through exchanges, each 
once. */
+  private[comet] def scalarSubqueries(plan: SparkPlan): Seq[Expression] = {
+    val found = ArrayBuffer.empty[ScalarSubquery]
+    def walk(node: SparkPlan): Unit = {
+      node.expressions.foreach(_.foreach {
+        case sub: ScalarSubquery if !found.exists(_.exprId == sub.exprId) => 
found += sub
+        case _ =>
+      })
+      node match {
+        case reused: ReusedExchangeExec => walk(reused.child)
+        case _ => node.children.foreach(walk)
+      }
+    }
+    walk(plan)
+    found.toSeq
+  }
+
+  /**
+   * `query` with each scalar subquery's `Subquery` expression replaced by a 
literal of its
+   * result. The subqueries must have run. Sending the results alongside the 
plan would spare
+   * rewriting it (https://github.com/apache/datafusion-comet/issues/7374).
+   */
+  def withSubqueryResults(query: QueryRequest, subqueries: 
Seq[ScalarSubquery]): QueryRequest = {
+    if (subqueries.isEmpty) {
+      return query
+    }
+    val literals: Map[Long, ExprOuterClass.Expr] = subqueries.map { sub =>
+      // The result is already in Catalyst's internal form.
+      val literal = Literal(sub.eval(null), sub.dataType)
+      val expr = QueryPlanSerde
+        .exprToProto(literal, Seq.empty)
+        .getOrElse(throw new CometRuntimeException(
+          s"Comet Native Executors cannot pass the result of $sub to the 
cluster"))
+      sub.exprId.id -> expr
+    }.toMap
+    val replace: PartialFunction[Message, Message] = {
+      case expr: ExprOuterClass.Expr if expr.hasSubquery =>
+        literals.getOrElse(
+          expr.getSubquery.getId,
+          throw new CometRuntimeException(
+            s"Comet Native Executors found no result for subquery 
${expr.getSubquery.getId}"))
+    }
+    val request = query.toBuilder
+    (0 until request.getStagesCount).foreach { i =>
+      val stage = request.getStages(i)
+      val plan = transformMessages(stage.getPlan)(replace)
+      // A partition's plan is the stage's with its files added, so it holds a 
subquery only
+      // when the stage's plan does.
+      if (plan ne stage.getPlan) {
+        val replaced = stage.toBuilder.setPlan(plan.asInstanceOf[Operator])
+        (0 until stage.getPartitionPlansCount).foreach { p =>
+          val partitionPlan = Operator.parseFrom(stage.getPartitionPlans(p))
+          replaced.setPartitionPlans(p, 
transformMessages(partitionPlan)(replace).toByteString)
+        }
+        request.setStages(i, replaced)
+      }
+    }
+    request.build()
+  }
+
+  /**
+   * The messages directly inside `message`, each with how to put a 
replacement for it in a
+   * builder of `message`.
+   */
+  private def childMessages(
+      message: Message): Iterator[(Message, (Message.Builder, Message) => 
Unit)] =
+    message.getAllFields.asScala.iterator
+      .filter { case (field, _) => field.getJavaType == 
FieldDescriptor.JavaType.MESSAGE }
+      .flatMap { case (field, value) =>
+        if (field.isRepeated) {
+          
value.asInstanceOf[java.util.List[Message]].asScala.iterator.zipWithIndex.map {
+            case (child, index) =>
+              (
+                child,
+                (builder: Message.Builder, m: Message) => {
+                  builder.setRepeatedField(field, index, m)
+                  ()
+                })
+          }
+        } else {
+          Iterator(
+            (
+              value.asInstanceOf[Message],
+              (builder: Message.Builder, m: Message) => {
+                builder.setField(field, m)
+                ()
+              }))
+        }
+      }
+
+  /**
+   * `message` with every message inside it that `pf` matches replaced by what 
`pf` returns. A
+   * message nothing inside changed is returned as is.
+   */
+  private def transformMessages(message: Message)(
+      pf: PartialFunction[Message, Message]): Message =
+    pf.applyOrElse(
+      message,
+      (m: Message) => {
+        var builder: Message.Builder = null
+        childMessages(m).foreach { case (child, set) =>
+          val replaced = transformMessages(child)(pf)
+          if (replaced ne child) {
+            if (builder == null) {
+              builder = m.toBuilder
+            }
+            set(builder, replaced)
+          }
+        }
+        if (builder == null) m else builder.build()
+      })
+
+  /** The configs a Spark task would pass to native code with a plan. */
+  private def nativeConf(): java.util.Map[String, String] = {
+    val conf = new java.util.HashMap[String, String](
+      
ConfigMap.parseFrom(CometExecIterator.serializeCometSQLConfs()).getEntriesMap)
+    // A Spark task passes the batch size apart from the other configs.
+    conf.put(CometConf.COMET_BATCH_SIZE.key, 
CometConf.COMET_BATCH_SIZE.get().toString)
+    conf
+  }
+
+  private val singlePartition = PartitioningOuterClass.Partitioning
+    .newBuilder()
+    .setSinglePartition(PartitioningOuterClass.SinglePartition.newBuilder())
+    .build()
+
+  /** The planning data of a partition that reads no files. */
+  private val noFiles = OperatorOuterClass.NativeScan
+    .newBuilder()
+    .setFilePartition(OperatorOuterClass.SparkFilePartition.getDefaultInstance)
+    .build()
+    .toByteArray
+
+  private class StageGraphBuilder(query: SparkPlan, withFiles: Boolean) {
+    val stages: ArrayBuffer[Stage] = ArrayBuffer.empty
+    private val stageIndexes = new IdentityHashMap[SparkPlan, Integer]()
+
+    /**
+     * The index of the stage that runs the block of Comet operators rooted at 
`plan`, adding it
+     * first.
+     */
+    def stageOf(plan: SparkPlan): Int = {
+      val known = stageIndexes.get(plan)
+      if (known != null) {
+        return known
+      }
+      val block = plan match {
+        case take: CometTakeOrderedAndProjectExec => return 
takeOrderedStage(take)
+        case native: CometNativeExec if native.serializedPlanOpt.isDefined => 
native
+        case other => fail(other, "it is not a Comet operator that runs in 
Rust")
+      }
+      val scans = ArrayBuffer.empty[CometNativeScanExec]
+      val inputs = inputsOf(block, scans).map {
+        case exchange: CometShuffleExchangeExec =>
+          
stageInput(exchange.child)(_.setPartitioning(partitioningOf(exchange)))
+        case broadcast => 
stageInput(broadcast.children.head)(_.setBroadcast(true))
+      }
+      val blockPlan = Operator.parseFrom(block.serializedPlanOpt.plan.get)
+
+      // A block that reads files runs one copy per partition, each naming its 
own files.
+      val (common, perPartition) =
+        if (withFiles) filesOf(block, scans.toSeq)
+        else (Map.empty[String, Array[Byte]], Map.empty[String, 
Array[PartitionPlanData]])
+      val partitionPlans = if (perPartition.isEmpty) {
+        Seq.empty
+      } else {
+        val base = new PlanDataInjector.CachedPlanData(blockPlan)
+        val counts = perPartition.map { case (key, data) => key -> data.length 
}
+        require(
+          counts.values.toSet.size == 1,
+          s"The scans of one block must have the same number of partitions, 
but found $counts")
+        (0 until counts.values.head).map { i =>
+          val partitionData = perPartition.map { case (key, data) => key -> 
data(i).bytes }
+          PlanDataInjector.injectPlanData(base, common, 
partitionData).toByteString
+        }
+      }
+      // The cluster reads it only for a stage that reads no exchange. A stage 
that does has as
+      // many partitions as its exchanges deliver. A stage that reads neither 
and reports no
+      // partitions, such as an empty range, runs one, which produces no rows, 
since the cluster
+      // refuses a stage with none.
+      val numPartitions =
+        if (partitionPlans.nonEmpty) partitionPlans.length
+        else if (inputs.isEmpty) block.outputPartitioning.numPartitions.max(1)

Review Comment:
   This branch reaches more than the empty range. A block whose only scan is an 
Iceberg scan that plans no tasks gets here too, because `findAllPlanData` 
returns nothing for that scan and `filesOf` only adds an empty partition for 
`CometNativeScanExec`. Before this commit the cluster refused that stage with 
`The stage has no partitions`. Now it runs one partition of the base plan, and 
that plan's `IcebergScan` has only the stub `common` that 
`CometIcebergNativeScan.convert` sets (metadata location and scan hash, no 
`required_schema`). I have not run it, so I don't know whether the cluster 
returns no rows or fails in `block_schema` or the native planner. Could the 
`max(1)` apply only to blocks without scans, for example when every leaf is a 
`CometRangeExec`? Or could the Iceberg scan get an empty partition the way 
`filesOf` gives one to Parquet? Either way, a test for the empty Iceberg case 
would pin it down.



##########
spark/src/test/scala/org/apache/comet/exec/CometNativeExecutorsSuite.scala:
##########
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.exec
+
+import java.io.File
+import java.net.{ServerSocket, Socket}
+import java.nio.file.Files
+import java.util.concurrent.TimeUnit
+
+import scala.util.Try
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.{DataFrame, Row}
+import org.apache.spark.sql.comet.{CometNativeExecutorsExec, 
CometNativeExecutorsTestBase}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.functions.{col, count, max, sum}
+import org.apache.spark.sql.internal.SQLConf
+
+import org.apache.comet.CometConf
+
+/**
+ * Runs queries on a Comet Native Executor cluster (see
+ * https://github.com/apache/datafusion-comet/issues/7288): a comet-scheduler 
and a comet-executor
+ * started from the binaries in the directory `COMET_NATIVE_EXECUTORS_BIN_DIR` 
names. The tests
+ * are skipped when it isn't set. To run them:
+ *
+ * {{{
+ * cargo build --manifest-path native/Cargo.toml -p datafusion-comet 
--features native-executors
+ * CARGO_TARGET_DIR=native/target cargo build --manifest-path 
native/native-executors/Cargo.toml
+ * COMET_NATIVE_EXECUTORS_BIN_DIR=native/target/debug ./mvnw test -Dtest=none \
+ *   -Dsuites=org.apache.comet.exec.CometNativeExecutorsSuite
+ * }}}
+ *
+ * The TPC-H tests need the cluster of 
https://github.com/apache/datafusion-comet/pull/7289. On
+ * the cluster in this repository, q1, q6, q11 and q15 pass and the other 
queries, which join,
+ * fail while the scheduler plans them.
+ *
+ * `CometNativeExecutorsPlanSuite` checks the stage graphs and the refusals 
without a cluster, and
+ * runs in CI.
+ */
+class CometNativeExecutorsSuite
+    extends CometNativeExecutorsTestBase
+    with AdaptiveSparkPlanHelper {
+
+  private val binDir: Option[File] =
+    sys.env.get("COMET_NATIVE_EXECUTORS_BIN_DIR").map(new 
File(_).getAbsoluteFile)
+  private var processes: Seq[Process] = Nil
+  private var schedulerUrl: String = _
+
+  override protected def sparkConf: SparkConf = {
+    val conf = super.sparkConf
+    conf.set(CometConf.COMET_SHUFFLE_MODE.key, "native")
+    conf.set(SQLConf.SHUFFLE_PARTITIONS.key, "4")
+    conf
+  }
+
+  override def beforeAll(): Unit = {
+    super.beforeAll()
+    binDir.foreach(startCluster)
+  }
+
+  override def afterAll(): Unit = {
+    try {
+      // The executor first, so that it doesn't report to a scheduler that has 
gone.
+      processes.reverse.foreach { process =>
+        process.destroy()
+        if (!process.waitFor(10, TimeUnit.SECONDS)) {
+          process.destroyForcibly()
+        }
+      }
+    } finally {
+      super.afterAll()
+    }
+  }
+
+  private def startCluster(dir: File): Unit = {
+    val schedulerPort = freePort()
+    val flightPort = freePort()
+    val executorPort = freePort()
+    val executorGrpcPort = freePort()
+    val logs = Files.createTempDirectory("comet-native-executors").toFile
+    logInfo(s"Comet Native Executor cluster logs: $logs")
+    start(
+      new File(dir, "comet-scheduler"),
+      Map(
+        "COMET_SCHEDULER_BIND_HOST" -> "127.0.0.1",
+        "COMET_SCHEDULER_PORT" -> schedulerPort.toString,
+        "COMET_SCHEDULER_FLIGHT_PORT" -> flightPort.toString),
+      new File(logs, "scheduler.log"))
+    waitForPort(schedulerPort)
+    waitForPort(flightPort)
+    start(
+      new File(dir, "comet-executor"),
+      Map(
+        "COMET_EXECUTOR_BIND_HOST" -> "127.0.0.1",
+        "COMET_EXECUTOR_PORT" -> executorPort.toString,
+        "COMET_EXECUTOR_GRPC_PORT" -> executorGrpcPort.toString,
+        "COMET_SCHEDULER_HOST" -> "127.0.0.1",
+        "COMET_SCHEDULER_PORT" -> schedulerPort.toString,
+        "COMET_EXECUTOR_TASKS" -> "4",
+        "COMET_EXECUTOR_WORK_DIR" -> new File(logs, "work").getPath),
+      new File(logs, "executor.log"))
+    waitForPort(executorPort)
+    schedulerUrl = s"http://127.0.0.1:$flightPort";
+  }
+
+  private def start(binary: File, env: Map[String, String], log: File): Unit = 
{
+    val builder = new ProcessBuilder(binary.getPath)
+    env.foreach { case (key, value) => builder.environment().put(key, value) }
+    builder.redirectErrorStream(true).redirectOutput(log)
+    processes :+= builder.start()
+  }
+
+  private def freePort(): Int = {
+    val socket = new ServerSocket(0)
+    try socket.getLocalPort
+    finally socket.close()
+  }
+
+  private def waitForPort(port: Int): Unit = {
+    val deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(60)
+    while (Try(new Socket("127.0.0.1", port).close()).isFailure) {
+      assert(System.nanoTime() < deadline, s"Nothing is listening on port 
$port")
+      Thread.sleep(100)
+    }
+  }
+
+  private def nativeExecutorsTest(name: String)(body: => Unit): Unit = 
test(name) {
+    assume(binDir.isDefined, "COMET_NATIVE_EXECUTORS_BIN_DIR is not set")
+    body
+  }
+
+  private def onCluster(body: => Unit): Unit = withSQLConf(
+    CometConf.COMET_NATIVE_EXECUTORS_ENABLED.key -> "true",
+    // Without a cluster, only tests that the planner refuses run, and those 
never connect.
+    CometConf.COMET_NATIVE_EXECUTORS_SCHEDULER_URL.key ->
+      Option(schedulerUrl).getOrElse("http://localhost:50060";))(body)

Review Comment:
   Now that every test here is a `nativeExecutorsTest`, `schedulerUrl` is 
always set when `onCluster` runs. That makes the 
`getOrElse("http://localhost:50060";)` fallback and the comment about refusal 
tests dead. The move also left the `withTable` doc comment at line 188 attached 
to the first test. The one test that still needs no cluster is `a command's 
plan stays in Spark` (line 389). `CometNativeExecutors.applies` turns down a 
plan with `ExecutedCommandExec` or `CommandResultExec`, so `offload` and 
`requireNativeSupport` never run. Could it move to 
`CometNativeExecutorsPlanSuite` so the command guard runs in CI, as the 
refusals do now?



##########
docs/source/contributor-guide/native_executors.md:
##########
@@ -0,0 +1,133 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Comet Native Executors
+
+**Experimental.** Comet Native Executors run whole Spark queries without a JVM 
in the data plane. A
+Spark driver plans each query with Catalyst and Comet, then submits the plan 
as a graph of stages
+to a cluster of one `comet-scheduler` and any number of `comet-executor` 
processes, built on
+[Apache DataFusion Ballista](https://github.com/apache/datafusion-ballista). 
Each stage is a
+_block_: a run of Comet operators that Comet serializes as one Rust plan, with 
the exchanges between
+blocks as the edges of the graph. The design is in
+[issue #7288](https://github.com/apache/datafusion-comet/issues/7288), and
+[`native/native-executors/README.md`](https://github.com/apache/datafusion-comet/blob/main/native/native-executors/README.md)
+describes the cluster, how to build it and how to test it.
+
+This page is for contributors. It covers the Spark driver side: how to run a 
query on a cluster and
+what does not work yet. The driver never links Ballista. It sends the stage 
graph to the
+scheduler's Arrow Flight service through a small client in libcomet, and the 
Spark tasks that read
+the results use the same client. The cluster features that some queries rely 
on are in
+[#7289](https://github.com/apache/datafusion-comet/pull/7289) and are not part 
of the cluster in
+this repository yet. [Current limitations](#current-limitations) says which.
+
+## Running it
+
+1. Build libcomet with the `native-executors` Cargo feature. It is off by 
default, so the standard
+   build links no gRPC client. A libcomet without it fails the first query 
with an error that names
+   the feature. For example, `make COMET_FEATURES=native-executors` builds 
libcomet and the Comet
+   jar that bundles it. The Spark executors that read the results need this 
libcomet too.
+2. Build and start the cluster. The `comet-scheduler` and `comet-executor` 
binaries are their own
+   Cargo workspace, so the main Comet build does not depend on Ballista. The 
cluster and the driver
+   must come from the same Comet build, which the scheduler checks. See
+   
[`native/native-executors/README.md`](https://github.com/apache/datafusion-comet/blob/main/native/native-executors/README.md)
+   for how to build the binaries and for the environment variables that 
configure them.
+3. Point Spark at the scheduler's Flight service and turn the feature on:
+
+   ```shell
+   --conf spark.comet.nativeExecutors.enabled=true \
+   --conf spark.comet.nativeExecutors.scheduler.url=http://scheduler-host:50060
+   ```
+
+   Both configs are internal while the feature is experimental. Comet itself 
must be enabled as
+   usual, with native shuffle (`spark.comet.shuffle.mode=native`, as in the
+   [installation guide](../user-guide/latest/installation.md)), because the 
driver turns only
+   native shuffle exchanges into the edges of the stage graph.
+
+4. Check that the query ran on the cluster: its physical plan has a 
`CometNativeExecutors` leaf
+   (the `CometNativeExecutorsExec` class) where the Comet plan would otherwise 
be. A query that
+   cannot run entirely in the Comet pipeline and on the cluster fails instead, 
with a message that
+   names the operator and Comet's reasons for it.
+
+`CometNativeExecutorsSuite` runs queries against a cluster it starts from the 
binaries in
+`COMET_NATIVE_EXECUTORS_BIN_DIR`. Its header comment gives the commands. It is 
not part of CI. Its
+tests that need a cluster are skipped when the variable is not set. The tests 
that check what the
+driver refuses to plan run without a cluster, but they still need a libcomet 
built with the
+`native-executors` feature. Its TPC-H tests need the cluster of
+[#7289](https://github.com/apache/datafusion-comet/pull/7289): on the cluster 
in this repository,
+only q1, q6, q11 and q15 pass. `CometNativeExecutorsPlanSuite` runs in CI. It 
takes the plan Comet
+would execute and checks the stage graph, the queries the driver refuses, and 
how a scalar
+subquery's result goes into the plan, with neither a cluster nor the feature.
+
+## Current limitations
+
+Comet Native Executors are a first version. The sub-issues of
+[#7288](https://github.com/apache/datafusion-comet/issues/7288) track what is 
missing:
+
+- Whole queries only. See [below](#whole-queries-only).
+- Only native shuffle exchanges. A query whose exchanges use JVM shuffle is 
refused.
+- `LIMIT` and `show()`: 
[#7365](https://github.com/apache/datafusion-comet/issues/7365).
+- Global sorts. The cluster in this repository reads a range exchange as one 
partition, so every
+  global sort runs in a single task.
+  [#7289](https://github.com/apache/datafusion-comet/pull/7289) adds a cluster 
that sorts the
+  partitions in parallel and merges them, except for sorts by floats or nested 
types, which still
+  run in a single task: 
[#7366](https://github.com/apache/datafusion-comet/issues/7366).
+- Hash partitioning is DataFusion's, not Spark-compatible:
+  [#7368](https://github.com/apache/datafusion-comet/issues/7368).
+- A small broadcast side runs in every task that reads it:
+  [#7369](https://github.com/apache/datafusion-comet/issues/7369). Moving a 
join whose build side
+  turns out small into the stage that produces its probe side is in
+  [#7289](https://github.com/apache/datafusion-comet/pull/7289).
+- Dynamic partition pruning: 
[#7370](https://github.com/apache/datafusion-comet/issues/7370). A scan
+  with a dynamic pruning filter is refused.
+- Writes: [#7371](https://github.com/apache/datafusion-comet/issues/7371). 
Commands and writes stay
+  in Spark.
+- Cancellation: 
[#7372](https://github.com/apache/datafusion-comet/issues/7372). The driver 
thread
+  waits for the cluster in native code until the query finishes, so cancelling 
the Spark job or
+  interrupting the thread does not stop the wait, and a killed Spark job 
leaves its query running
+  on the cluster. A killed Spark task that reads a result partition stops at 
the next batch; a
+  read blocked waiting for a batch still waits for it.
+- Errors: [#7621](https://github.com/apache/datafusion-comet/issues/7621). A 
failure on the
+  cluster reaches Spark as a `CometNativeException` with the cluster's 
message, not as the exception
+  Spark would raise. For example, a division by zero with ANSI mode on is not a
+  `SparkArithmeticException` with the `DIVIDE_BY_ZERO` error class.
+- Spark UI metrics: 
[#7373](https://github.com/apache/datafusion-comet/issues/7373).
+- Security. The scheduler and the executors have no authentication and no TLS, 
and a query names
+  the files and object stores the executors read. Run a cluster only on a 
trusted network.
+- Result expiry. A query's results stay on the cluster for
+  `COMET_SCHEDULER_RESULT_RETENTION_SECONDS` after it ends (five minutes by 
default), whether or
+  not Spark has read them. The scheduler gives each result partition its 
expiration time, and a
+  Spark task that reads a partition after that fails with a message that names 
the setting.

Review Comment:
   What happens when a cluster executor that holds result partitions dies 
before Spark reads them? `CometNativeExecutorsRDD` has no dependencies 
(`RDD[ColumnarBatch](sc, Nil)`), and each partition keeps the `ResultEndpoint` 
it got at submission. So every retry of a read task goes back to the same 
executor and fails, and after `spark.task.maxFailures` attempts the job fails. 
Nothing resubmits the query. In Spark, a lost shuffle output raises 
`FetchFailedException` and the scheduler reruns the map stage. Could Current 
limitations say this next to result expiry, with a sub-issue of 
https://github.com/apache/datafusion-comet/issues/7288 if one is planned?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to