issues  

Messages by Date

• 2026/11/07 [jira] [Assigned] (SPARK-60974) Accept a DDL string schema in mapInPandas, applyInPandas and the other map functions in Spark Connect Rust as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61110) Add DataFrame._repr_html_, the HTML table of eager evaluation in notebooks, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61136) Collect the decimals of a negative scale in the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61122) Re-export the exceptions of pyspark.errors from pyspark.sql.utils, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61115) Add the module-level names and the connect modules that the Python client lacks, as PySpark 4.3.0 has them Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-60945) from_protobuf/to_protobuf can corrupt a shared options map when the plan is canonicalized concurrently Lihang Shen (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61124) Check the arguments of nearestByJoin, and take them, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61135) Parse the JSON of GeometryType and GeographyType, such as bare geometry and geography(SPHERICAL), and add their _from_crs, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61140) Send the renames of DataFrame.withColumnsRenamed in their order, in the Rust core and the Python client, as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61140) Send the renames of DataFrame.withColumnsRenamed in their order, in the Rust core and the Python client, as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61126) Make DataFrame.pandas_api take index_col, and DataFrame.plot the PySparkPlotAccessor, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61126) Make DataFrame.pandas_api take index_col, and DataFrame.plot the PySparkPlotAccessor, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61034) Send the output schema of transformWithState in the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-60987) Fix `NoClassDefFoundError` when running Kafka benchmarks in the benchmark workflow Dongjoon Hyun (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61121) Take a float frame bound past its threshold as unbounded in rowsBetween and rangeBetween, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-60971) Infer the iterator forms of GroupedData.applyInPandas and applyInArrow from type hints in Spark Connect Rust as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61129) Make SparkSession and DataFrame picklable, of the active session, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-60890) Upgrade `snowflake-jdbc` to 4.4.1 Dongjoon Hyun (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61138) Make str of a VariantVal its JSON, and createDataFrame reject rows that are VariantVals, in the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61135) Parse the JSON of GeometryType and GeographyType, such as bare geometry and geography(SPHERICAL), in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61113) Add DataFrame.__arrow_c_stream__, the Arrow C stream of a DataFrame, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61133) Add the private helpers of pyspark.sql.types, such as _infer_schema, _merge_type and _make_type_verifier, in the Python client as PySpark has them ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61129) Make SparkSession and DataFrame picklable, of the active session, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61101) Validate the save mode of DataFrameWriter, and the arguments of DataFrameReader, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61138) Make str of a VariantVal its JSON, and createDataFrame reject rows that are VariantVals, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61120) Add DataFrame._session, SparkSession._client, and the methods and results of the client that vendored modules use, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61124) Check the arguments of nearestByJoin, and take them, in the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61119) Set the element type of the array literals of the Rust core, as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61138) Make str of a VariantVal its JSON in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61139) Fix wrong results when `ConstantPropagation` or `InferFiltersFromConstraints` replaces a `FLOAT` or `DOUBLE` zero with the other signed zero Kousuke Saruta (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61138) Make str of a VariantVal its JSON in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61134) Make DayTimeIntervalType and YearMonthIntervalType check their fields, and the types without parameters their arguments, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61137) Make StructType give back the same StructField objects in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61137) Make StructType give back the same StructField objects in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61136) Collect the decimals of a negative scale in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61125) Pickle the handler of a UDTF when it is used, and raise UDTF_SERIALIZATION_ERROR, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61125) Pickle the handler of a UDTF when it is used, and raise UDTF_SERIALIZATION_ERROR, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61120) Add DataFrame._session, SparkSession._client, and the methods and results of the client that vendored modules use, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61128) Make ArrayType, MapType, StructField and StructType give back and pickle the user-defined types they are made of, and StructField picklable, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61118) Add the CLI modules of pyspark.pipelines, in the Python client as PySpark has them Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Comment Edited] (SPARK-55211) Create an optimizer rule to choose when to use transpiled Monica G (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61117) Call the hooks of SparkSession.Hook before each execution, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61123) Take a Python Enum as the literal of its value in lit, Column operators and functions, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61118) Add the CLI modules of pyspark.pipelines, and the user context extensions of the client, in the Python client as PySpark has them ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61127) Make the rows of DataFrame.collect and toLocalIterator honor spark.sql.execution.pyspark.binaryAsBytes in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61128) Make ArrayType, MapType, StructField and StructType give back and pickle the user-defined types they are made of, and StructField picklable, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61127) Make the rows of DataFrame.collect and toLocalIterator honor spark.sql.execution.pyspark.binaryAsBytes in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61123) Take a Python Enum as the literal of its value in lit, Column operators and functions, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61124) Check the arguments of nearestByJoin, and take them, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-60704) Avoid repeated Decimal type lookup for explicit decimal local input in Spark Connect Rust Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61127) Make the rows of DataFrame.collect and toLocalIterator honor spark.sql.execution.pyspark.binaryAsBytes in the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61078) Skip the throwaway allocations in 's deserialization constructor Dongjoon Hyun (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61129) Make SparkSession and DataFrame picklable, of the active session, in the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61087) to_time, try_to_time and time_from_seconds return TIME(6) values with more than 6 fractional digits ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Commented] (SPARK-60806) CREATE FUNCTION should reject only the temporary functions a permanent function's body actually calls Anshul Baliga (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61132) Raise a clear error for TIMESTAMP_NTZ columns in the Hive ORC data source ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61126) Make DataFrame.pandas_api take index_col, and DataFrame.plot the PySparkPlotAccessor, in the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61133) Add the private helpers of pyspark.sql.types, such as _infer_schema, _merge_type and _make_type_verifier, in the Python client as PySpark has them Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61133) Add the private helpers of pyspark.sql.types, such as _infer_schema, _merge_type and _make_type_verifier, in the Python client as PySpark has them Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61120) Add DataFrame._session, SparkSession._client, and the methods and results of the client that vendored modules use, in the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-60915) Send the call sites of DataFrame API calls in Spark Connect Rust, so that errors carry DataFrame query contexts as in PySpark Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-60958) Accept url in SparkSession.builder.remote() in Spark Connect Rust as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61103) Create a DataFrame of a numpy array in createDataFrame in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61102) Reject unsupported NumPy longdouble scalar classes as PySpark does DB Tsai (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61121) Take a float frame bound past its threshold as unbounded in rowsBetween and rangeBetween, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61125) Pickle the handler of a UDTF when it is used, and raise UDTF_SERIALIZATION_ERROR, in the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61128) Make ArrayType, MapType, StructField and StructType give back and pickle the user-defined types they are made of, and StructField picklable, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61132) Raise a clear error for TIMESTAMP_NTZ columns in the Hive ORC data source Anshul Baliga (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61128) Make ArrayType, MapType, StructField and StructType give back and pickle the user-defined types they are made of, and StructField picklable, in the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61088) Fix wrong results by stale partitioning and constraints when a checkpointed Dataset is self-joined Dongjoon Hyun (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61088) Fix wrong results by stale partitioning and constraints when a checkpointed Dataset is self-joined Dongjoon Hyun (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-60987) Fix `NoClassDefFoundError` when running Kafka benchmarks in the benchmark workflow Dongjoon Hyun (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61082) Fix UnsafeHashedRelation build failure when many small relations are alive in one task Dongjoon Hyun (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61130)  Add activeExecutors, readyExecutors, maxExecutorId and ExecutorStates to SparkApplication status Pradeep Reddy (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-60702) Don't skip a runtime bloom filter for a DPP filter that physical planning drops Peter Toth (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61066) Raise the errors of join types and dict aggregations, and return the status of a streaming query, in the Rust client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-60995) Check the arguments of when, bucket, pandas_udf, arrow_udf and arrow_udtf of the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61066) Raise the errors of join types and dict aggregations, and return the status of a streaming query, in the Rust client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-60992) Fix marking a succeeded task as failed when a post-status-update listener fails ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61084) Register progress handlers, and call them with the progress of each execution, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-60981) Alias a Column with several names, and with metadata of any JSON values, in Spark Connect Rust as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61118) Add the CLI modules of pyspark.pipelines, and the user context extensions of the client, in the Python client as PySpark has them Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61131) Fix wrong results of decimal in ANSI mode when the quotient overflows Dongjoon Hyun (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-60808) Runtime bloom filter built from a non-deterministic creation side can drop join rows Dongjoon Hyun (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61130)  Add activeExecutors, readyExecutors, maxExecutorId and ExecutorStates to SparkApplication status Pradeep Reddy (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61130)  Add activeExecutors, readyExecutors, maxExecutorId and ExecutorStates to SparkApplication status Pradeep Reddy (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61105) Upgrade docker images to Apache Spark 4.3.0, 4.1.4, and 4.0.5 Dongjoon Hyun (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61105) Upgrade docker images to Apache Spark 4.3.0, 4.1.4, and 4.0.5 Dongjoon Hyun (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61050) Fix `ArithmeticException` of decimal remainder, `pmod` and `div` when the integral quotient has more than 39 digits Dongjoon Hyun (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61108) Name the nested fields of collected rows by the schema, and raise for calendar intervals in collect, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61120) Add DataFrame._session, SparkSession._client and the results of execute_command that vendored modules read, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61120) Add DataFrame._session, SparkSession._client, and the methods and results of the client that vendored modules use, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61118) Add the CLI modules of pyspark.pipelines, and the user context extensions of the client, in the Python client as PySpark has them Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61035) Send the resource profile of mapInPandas and mapInArrow in the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61104) Convert pandas input without a schema as PySpark's Arrow conversion does in the Python client ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61051) Fix LongHashedRelation build failure when the dense key array does not fit in execution memory Dongjoon Hyun (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61113) Add DataFrame.__arrow_c_stream__, the Arrow C stream of a DataFrame, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-60924) Use GraphQL to count commits for post-merge CI sampling Ruifeng Zheng (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61114) Run the full matrix in sampled post-merge CI Ruifeng Zheng (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61129) Make SparkSession and DataFrame picklable, of the active session, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61076) Accept intermediate storage level keyword arguments in PySpark KMeans Ruifeng Zheng (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61128) Make ArrayType, MapType, StructField and StructType give back and pickle the user-defined types they are made of, and StructField picklable, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61116) Port the Read, Range and SQL plans, Column.to_plan, and the type and conversion helpers of the connect import path, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-60702) Don't skip a runtime bloom filter for a DPP filter that physical planning drops Peter Toth (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61129) Make SparkSession and DataFrame picklable, of the active session, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61122) Re-export the exceptions of pyspark.errors from pyspark.sql.utils, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61123) Take a Python Enum as the literal of its value in lit, Column operators and functions, in the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61116) Port the Read, Range and SQL plans, Column.to_plan, and the type and conversion helpers of the connect import path, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61119) Set the element type of the array literals of the Rust core, as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61127) Make the rows of DataFrame.collect and toLocalIterator honor spark.sql.execution.pyspark.binaryAsBytes in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61121) Take a float frame bound past its threshold as unbounded in rowsBetween and rangeBetween, in the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61126) Make DataFrame.pandas_api take index_col, and DataFrame.plot the PySparkPlotAccessor, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61126) Make DataFrame.pandas_api take index_col, and DataFrame.plot the PySparkPlotAccessor, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61125) Pickle the handler of a UDTF when it is used, and raise UDTF_SERIALIZATION_ERROR, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61124) Check the arguments of nearestByJoin, and take them, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61124) Check the arguments of nearestByJoin, and take them, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61123) Take a Python Enum as the literal of its value in lit, Column operators and functions, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61116) Port the Read, Range and SQL plans, Column.to_plan, and the type and conversion helpers of the connect import path, in the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61121) Take a float frame bound past its threshold as unbounded in rowsBetween and rangeBetween, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61120) Add DataFrame._session, SparkSession._client and the results of execute_command that vendored modules read, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61115) Add the module-level names and the connect modules that the Python client lacks, as PySpark 4.3.0 has them Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61115) Add the module-level names and the connect modules that the Python client lacks, as PySpark 4.3.0 has them Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61106) Convert the values of local rows of a schema, and raise their errors, as PyArrow does in createDataFrame in the Python client Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61119) Set the element type of the array literals of the Rust core, as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61119) Set the element type of the array literals of the Rust core, as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61117) Call the hooks of SparkSession.Hook before each execution, in the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61101) Validate the save mode of DataFrameWriter, and the arguments of DataFrameReader, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-60123) Reading ORC tables with collated STRING columns fails ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61111) Support the ML API of pyspark.ml, of estimators, models, pipelines and evaluators on the server, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61111) Support the ML API of pyspark.ml, of estimators, models, pipelines and evaluators on the server, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61117) Call the hooks of SparkSession.Hook before each execution, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61118) Add the CLI modules of pyspark.pipelines, in the Python client as PySpark has them Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61114) Run the full matrix in sampled post-merge CI Ruifeng Zheng (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61113) Add DataFrame.__arrow_c_stream__, the Arrow C stream of a DataFrame, in the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61116) Port the Read, Range and SQL plans, Column.to_plan, and the type and conversion helpers of the connect import path, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61114) Run the full matrix in sampled post-merge CI ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61107) Cast the table of toArrow to the Arrow schema of the DataFrame in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61076) Accept intermediate storage level keyword arguments in PySpark KMeans Ruifeng Zheng (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61017) Support configurable intermediate storage levels in iterative classifiers Ruifeng Zheng (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61017) Support configurable intermediate storage levels in iterative classifiers Ruifeng Zheng (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61017) Support configurable intermediate storage levels in iterative classifiers Ruifeng Zheng (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61076) Accept intermediate storage level keyword arguments in PySpark KMeans Ruifeng Zheng (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61091) Return the NamedTuples of pyspark.sql.catalog from Catalog, and register functions with registerFunction, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61112) Run the doctests of `dev/` scripts in the linter job Kousuke Saruta (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61107) Cast the table of toArrow to the Arrow schema of the DataFrame in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-60871) Keep the collation of a collated variant_get target type in comparisons Stevo Mitric (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61031) Take the arguments of the readers, the writer and the catalog, and create the tables of createTable on the call, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61111) Support the ML API of pyspark.ml, of estimators, models, pipelines and evaluators on the server, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61107) Cast the table of toArrow to the Arrow schema of the DataFrame in the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61106) Convert the values of local rows of a schema, and raise their errors, as PyArrow does in createDataFrame in the Python client Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61108) Name the nested fields of collected rows by the schema, and raise for calendar intervals in collect, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61106) Convert the values of local rows of a schema, and raise their errors, as PyArrow does in createDataFrame in the Python client Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61083) Raise the errors of PySpark for the tags, the table name, the paths of copyFromLocalToFs and the JVM attributes of a session in the Python client Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61111) Support the ML API of pyspark.ml, of estimators, models, pipelines and evaluators on the server, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61110) Add DataFrame._repr_html_, the HTML table of eager evaluation in notebooks, in the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61078) Skip the throwaway allocations in 's deserialization constructor Dongjoon Hyun (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61090) Harden reattach cleanup regression tests in Spark Connect Rust Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61090) Harden reattach cleanup regression tests in Spark Connect Rust ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61090) Harden reattach cleanup regression tests in Spark Connect Rust Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61109) Merged LIMIT and OFFSET values can overflow Int in the optimizer, so valid queries fail or return wrong rows ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61110) Add DataFrame._repr_html_, the HTML table of eager evaluation in notebooks, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61109) Merged LIMIT and OFFSET values can overflow Int in the optimizer, so valid queries fail or return wrong rows Norio Akagi (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61109) Merged LIMIT and OFFSET values can overflow Int in the optimizer, so valid queries fail or return wrong rows Norio Akagi (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61106) Convert the values of local rows of a schema, and raise their errors, as PyArrow does in createDataFrame in the Python client ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61103) Create a DataFrame of a numpy array in createDataFrame in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61103) Create a DataFrame of a numpy array in createDataFrame in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61106) Convert the values of local rows of a schema, and raise their errors, as PyArrow does in createDataFrame in the Python client Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61103) Create a DataFrame of a numpy array in createDataFrame in the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61101) Validate the save mode of DataFrameWriter, and the arguments of DataFrameReader, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61093) Raise the errors of PySpark of Column.try_cast, and of Column.isin of a tuple, in the Python client Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61107) Cast the table of toArrow to the Arrow schema of the DataFrame in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61033) Fix query errors by ConstantPropagation when a rewritten operand of AND, OR, COALESCE or IN guards an expression that can fail Dongjoon Hyun (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61107) Cast the table of toArrow to the Arrow schema of the DataFrame in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61108) Name the nested fields of collected rows by the schema, and raise for calendar intervals in collect, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61108) Name the nested fields of collected rows by the schema, and raise for calendar intervals in collect, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61037) Group by the grouping columns of groupingSets in the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Resolved] (SPARK-61010) Fit native Rust LogisticRegression and predict with its cached model Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61106) Convert the values of local rows of a schema, and raise their errors, as PyArrow does in createDataFrame in the Python client Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61103) Create a DataFrame of a numpy array in createDataFrame in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61089) Allow UDTF AnalyzeResult subclasses in the Rust Python client DB Tsai (Jira) via issues
• 2026/11/07 [jira] [Updated] (SPARK-61046) Pivot only a groupBy, and raise the errors of the arguments of pivot, in the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61095) Add the modules geo_utils, sql_formatter, table_arg and tvf_argument of pyspark.sql in the Python client as the PySpark client has them Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Created] (SPARK-61104) Convert pandas input without a schema as PySpark's Arrow conversion does in the Python client Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61096) Represent GroupedData by its grouping expressions, the types of its DataFrame and its type in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/07 [jira] [Assigned] (SPARK-61095) Add the modules geo_utils, sql_formatter, table_arg and tvf_argument of pyspark.sql in the Python client as the PySpark client has them Hyukjin Kwon (Jira) via issues
• 2026/11/06 [jira] [Resolved] (SPARK-61094) Add UnparsedDataType of pyspark.sql.connect.types, the returnType of a UDTF of a DDL string, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/06 [jira] [Resolved] (SPARK-61096) Represent GroupedData by its grouping expressions, the types of its DataFrame and its type in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/06 [jira] [Assigned] (SPARK-61094) Add UnparsedDataType of pyspark.sql.connect.types, the returnType of a UDTF of a DDL string, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/06 [jira] [Resolved] (SPARK-60911) Raise server errors in Spark Connect Rust with the exception class and details PySpark gives Hyukjin Kwon (Jira) via issues
• 2026/11/06 [jira] [Updated] (SPARK-61105) Upgrade docker images to Apache Spark 4.3.0, 4.1.4, and 4.0.5 ASF GitHub Bot (Jira) via issues
• 2026/11/06 [jira] [Resolved] (SPARK-61071) Type pandas columns of Decimals and other values as PySpark does in createDataFrame of the Rust client Hyukjin Kwon (Jira) via issues
• 2026/11/06 [jira] [Assigned] (SPARK-61058) Take the arguments of queryName, partitionBy and clusterBy of DataStreamWriter in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/06 [jira] [Updated] (SPARK-60694) Push predicates through external lookup joins ASF GitHub Bot (Jira) via issues
• 2026/11/06 [jira] [Updated] (SPARK-61101) Validate the save mode of DataFrameWriter, and the arguments of DataFrameReader, in the Python client as PySpark does ASF GitHub Bot (Jira) via issues
• 2026/11/06 [jira] [Assigned] (SPARK-61092) Take a client seed in sample and randomSplit, and the arguments of withMetadata, withWatermark and randomSplit and the attributes of a DataFrame, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/06 [jira] [Updated] (SPARK-61095) Add the modules geo_utils, sql_formatter, table_arg and tvf_argument of pyspark.sql in the Python client as the PySpark client has them Hyukjin Kwon (Jira) via issues
• 2026/11/06 [jira] [Updated] (SPARK-54474) Disable the XML reports on tests that are supposed to fail Dongjoon Hyun (Jira) via issues
• 2026/11/06 [jira] [Created] (SPARK-61101) Validate the save mode of DataFrameWriter, and the arguments of DataFrameReader, in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/06 [jira] [Assigned] (SPARK-61075) Convert values in toInternal and fromInternal of data types in the Python client as PySpark does Hyukjin Kwon (Jira) via issues
• 2026/11/06 [jira] [Updated] (SPARK-60856) Fix DSv2 CHECK constraint failing every write when its condition subscripts a lambda variable Dongjoon Hyun (Jira) via issues

• Earlier messages
• Later messages