[jira] [Created] (SPARK-51074) Set timeout for communicating init configs in streaming foreachBatch

2025-02-03 Thread Wei Liu (Jira)
Wei Liu created SPARK-51074: --- Summary: Set timeout for communicating init configs in streaming foreachBatch Key: SPARK-51074 URL: https://issues.apache.org/jira/browse/SPARK-51074 Project: Spark

[jira] [Created] (SPARK-51055) Streaming foreachBatch should put initialization logic inside try

2025-01-31 Thread Wei Liu (Jira)
Wei Liu created SPARK-51055: --- Summary: Streaming foreachBatch should put initialization logic inside try Key: SPARK-51055 URL: https://issues.apache.org/jira/browse/SPARK-51055 Project: Spark Issu

[jira] [Created] (SPARK-51043) Spark Connect Streaming ForeachBatch log improvement

2025-01-30 Thread Wei Liu (Jira)
Wei Liu created SPARK-51043: --- Summary: Spark Connect Streaming ForeachBatch log improvement Key: SPARK-51043 URL: https://issues.apache.org/jira/browse/SPARK-51043 Project: Spark Issue Type: Improv

[jira] [Created] (SPARK-50817) Fix rocksdb error reporting

2025-01-14 Thread Wei Liu (Jira)
Wei Liu created SPARK-50817: --- Summary: Fix rocksdb error reporting Key: SPARK-50817 URL: https://issues.apache.org/jira/browse/SPARK-50817 Project: Spark Issue Type: Improvement Componen

[jira] [Created] (SPARK-50654) CommitMetadata should set stateUniqueIds to None in V1

2024-12-23 Thread Wei Liu (Jira)
Wei Liu created SPARK-50654: --- Summary: CommitMetadata should set stateUniqueIds to None in V1 Key: SPARK-50654 URL: https://issues.apache.org/jira/browse/SPARK-50654 Project: Spark Issue Type: Impr

[jira] [Updated] (SPARK-50652) RocksDB V2 backward compatibility currently not suuported

2024-12-23 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-50652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-50652: Epic Link: SPARK-49374 > RocksDB V2 backward compatibility currently not suuported > -

[jira] [Created] (SPARK-50653) Support RocksDB V2 -> V1 backward compatibility

2024-12-23 Thread Wei Liu (Jira)
Wei Liu created SPARK-50653: --- Summary: Support RocksDB V2 -> V1 backward compatibility Key: SPARK-50653 URL: https://issues.apache.org/jira/browse/SPARK-50653 Project: Spark Issue Type: Improvemen

[jira] [Created] (SPARK-50652) RocksDB V2 backward compatibility currently not suuported

2024-12-23 Thread Wei Liu (Jira)
Wei Liu created SPARK-50652: --- Summary: RocksDB V2 backward compatibility currently not suuported Key: SPARK-50652 URL: https://issues.apache.org/jira/browse/SPARK-50652 Project: Spark Issue Type: I

[jira] [Created] (SPARK-50622) RocksDB refactor

2024-12-19 Thread Wei Liu (Jira)
Wei Liu created SPARK-50622: --- Summary: RocksDB refactor Key: SPARK-50622 URL: https://issues.apache.org/jira/browse/SPARK-50622 Project: Spark Issue Type: Improvement Components: SS A

[jira] [Created] (SPARK-50565) Add transformWithState correctness test

2024-12-12 Thread Wei Liu (Jira)
Wei Liu created SPARK-50565: --- Summary: Add transformWithState correctness test Key: SPARK-50565 URL: https://issues.apache.org/jira/browse/SPARK-50565 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-50360) StateStoreChangelogReader should read state store version from the changelog

2024-11-19 Thread Wei Liu (Jira)
Wei Liu created SPARK-50360: --- Summary: StateStoreChangelogReader should read state store version from the changelog Key: SPARK-50360 URL: https://issues.apache.org/jira/browse/SPARK-50360 Project: Spark

[jira] [Created] (SPARK-50339) Enable changelog to store lineage information

2024-11-18 Thread Wei Liu (Jira)
Wei Liu created SPARK-50339: --- Summary: Enable changelog to store lineage information Key: SPARK-50339 URL: https://issues.apache.org/jira/browse/SPARK-50339 Project: Spark Issue Type: Task

[jira] [Created] (SPARK-49886) Query Level Correctness verification

2024-10-04 Thread Wei Liu (Jira)
Wei Liu created SPARK-49886: --- Summary: Query Level Correctness verification Key: SPARK-49886 URL: https://issues.apache.org/jira/browse/SPARK-49886 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-49885) Avoid listing when V2 lineage is provided in RocksDB

2024-10-04 Thread Wei Liu (Jira)
Wei Liu created SPARK-49885: --- Summary: Avoid listing when V2 lineage is provided in RocksDB Key: SPARK-49885 URL: https://issues.apache.org/jira/browse/SPARK-49885 Project: Spark Issue Type: Improv

[jira] [Updated] (SPARK-49885) Avoid listing when V2 lineage is provided in RocksDB

2024-10-04 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-49885: Epic Link: SPARK-49374 > Avoid listing when V2 lineage is provided in RocksDB > --

[jira] [Created] (SPARK-49884) Backward Compatibility and Test

2024-10-04 Thread Wei Liu (Jira)
Wei Liu created SPARK-49884: --- Summary: Backward Compatibility and Test Key: SPARK-49884 URL: https://issues.apache.org/jira/browse/SPARK-49884 Project: Spark Issue Type: Improvement Compo

[jira] [Created] (SPARK-49883) Integrate with RocksDB and RocksDBFileManager

2024-10-04 Thread Wei Liu (Jira)
Wei Liu created SPARK-49883: --- Summary: Integrate with RocksDB and RocksDBFileManager Key: SPARK-49883 URL: https://issues.apache.org/jira/browse/SPARK-49883 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-49824) SparkConnectStreamingQueryCache Log improvement

2024-09-27 Thread Wei Liu (Jira)
Wei Liu created SPARK-49824: --- Summary: SparkConnectStreamingQueryCache Log improvement Key: SPARK-49824 URL: https://issues.apache.org/jira/browse/SPARK-49824 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-49722) Fix Column ordering of dropDuplicates in spark connect

2024-09-19 Thread Wei Liu (Jira)
Wei Liu created SPARK-49722: --- Summary: Fix Column ordering of dropDuplicates in spark connect Key: SPARK-49722 URL: https://issues.apache.org/jira/browse/SPARK-49722 Project: Spark Issue Type: Task

[jira] [Commented] (SPARK-49127) How to restart failed spark stream job from the failure point

2024-09-05 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17879657#comment-17879657 ] Wei Liu commented on SPARK-49127: - A checkpoint define the streaming query progress. If

[jira] [Updated] (SPARK-49462) A query running under ckpt version v1 should be able to read a STATELESS query running under ckptv2

2024-08-29 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-49462: Epic Link: SPARK-49374 > A query running under ckpt version v1 should be able to read a STATELESS > query

[jira] [Created] (SPARK-49462) A query running under ckpt version v1 should be able to read a STATELESS query running under ckptv2

2024-08-29 Thread Wei Liu (Jira)
Wei Liu created SPARK-49462: --- Summary: A query running under ckpt version v1 should be able to read a STATELESS query running under ckptv2 Key: SPARK-49462 URL: https://issues.apache.org/jira/browse/SPARK-49462

[jira] [Created] (SPARK-49461) Persistent Checkpoint ID to commit logs and read it back

2024-08-29 Thread Wei Liu (Jira)
Wei Liu created SPARK-49461: --- Summary: Persistent Checkpoint ID to commit logs and read it back Key: SPARK-49461 URL: https://issues.apache.org/jira/browse/SPARK-49461 Project: Spark Issue Type: Ta

[jira] [Updated] (SPARK-49374) RocksDB State Store Checkpoint Structure V2

2024-08-23 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-49374: Description: h2. Motivation We expect the new checkpoint structure would benefit micro-batch mode by est

[jira] [Updated] (SPARK-49374) RocksDB State Store Checkpoint Structure V2

2024-08-23 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-49374: Description: h2. Motivation We expect the new checkpoint structure would benefit micro-batch mode by est

[jira] [Updated] (SPARK-49374) RocksDB State Store Checkpoint Structure V2

2024-08-23 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-49374: Issue Type: Epic (was: Task) > RocksDB State Store Checkpoint Structure V2 >

[jira] [Commented] (SPARK-48993) Maximum number of maxRecursiveFieldDepth should be a spark conf

2024-07-24 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868457#comment-17868457 ] Wei Liu commented on SPARK-48993: - ill followup on this > Maximum number of maxRecursiv

[jira] [Created] (SPARK-48993) Maximum number of maxRecursiveFieldDepth should be a spark conf

2024-07-24 Thread Wei Liu (Jira)
Wei Liu created SPARK-48993: --- Summary: Maximum number of maxRecursiveFieldDepth should be a spark conf Key: SPARK-48993 URL: https://issues.apache.org/jira/browse/SPARK-48993 Project: Spark Issue

[jira] [Created] (SPARK-48890) Add Streaming related fields to log4j ThreadContext

2024-07-12 Thread Wei Liu (Jira)
Wei Liu created SPARK-48890: --- Summary: Add Streaming related fields to log4j ThreadContext Key: SPARK-48890 URL: https://issues.apache.org/jira/browse/SPARK-48890 Project: Spark Issue Type: Sub-tas

[jira] [Created] (SPARK-48800) Deflake ClientStreamingQuerySuite

2024-07-03 Thread Wei Liu (Jira)
Wei Liu created SPARK-48800: --- Summary: Deflake ClientStreamingQuerySuite Key: SPARK-48800 URL: https://issues.apache.org/jira/browse/SPARK-48800 Project: Spark Issue Type: New Feature Com

[jira] [Created] (SPARK-48743) MergingSessionIterator should catch when getStruct returns null

2024-06-27 Thread Wei Liu (Jira)
Wei Liu created SPARK-48743: --- Summary: MergingSessionIterator should catch when getStruct returns null Key: SPARK-48743 URL: https://issues.apache.org/jira/browse/SPARK-48743 Project: Spark Issue

[jira] [Created] (SPARK-48717) Python foreachBatch streaming query cannot be stopped gracefully after pin thread mode is enabled and is running spark queries

2024-06-25 Thread Wei Liu (Jira)
Wei Liu created SPARK-48717: --- Summary: Python foreachBatch streaming query cannot be stopped gracefully after pin thread mode is enabled and is running spark queries Key: SPARK-48717 URL: https://issues.apache.org/jira

[jira] [Created] (SPARK-48567) Pyspark StreamingQuery lastProgress and friend should return actual StreamingQueryProgress

2024-06-07 Thread Wei Liu (Jira)
Wei Liu created SPARK-48567: --- Summary: Pyspark StreamingQuery lastProgress and friend should return actual StreamingQueryProgress Key: SPARK-48567 URL: https://issues.apache.org/jira/browse/SPARK-48567 Proj

[jira] [Created] (SPARK-48482) dropDuplicates and dropDuplicatesWithinWatermark should accept varargs

2024-05-30 Thread Wei Liu (Jira)
Wei Liu created SPARK-48482: --- Summary: dropDuplicates and dropDuplicatesWithinWatermark should accept varargs Key: SPARK-48482 URL: https://issues.apache.org/jira/browse/SPARK-48482 Project: Spark

[jira] [Created] (SPARK-48480) StreamingQueryListener thread should not be interruptable

2024-05-30 Thread Wei Liu (Jira)
Wei Liu created SPARK-48480: --- Summary: StreamingQueryListener thread should not be interruptable Key: SPARK-48480 URL: https://issues.apache.org/jira/browse/SPARK-48480 Project: Spark Issue Type: N

[jira] [Commented] (SPARK-48411) Add E2E test for DropDuplicateWithinWatermark

2024-05-24 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17849348#comment-17849348 ] Wei Liu commented on SPARK-48411: - Sorry I tagged the wrong Yuchen > Add E2E test f

[jira] [Created] (SPARK-48411) Add E2E test for DropDuplicateWithinWatermark

2024-05-24 Thread Wei Liu (Jira)
Wei Liu created SPARK-48411: --- Summary: Add E2E test for DropDuplicateWithinWatermark Key: SPARK-48411 URL: https://issues.apache.org/jira/browse/SPARK-48411 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-48411) Add E2E test for DropDuplicateWithinWatermark

2024-05-24 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17849202#comment-17849202 ] Wei Liu commented on SPARK-48411: - [~liuyuchen777] is going to work on this > Add E2E t

[jira] [Created] (SPARK-48181) Unify StreamingPythonRunner and PythonPlannerRunner

2024-05-07 Thread Wei Liu (Jira)
Wei Liu created SPARK-48181: --- Summary: Unify StreamingPythonRunner and PythonPlannerRunner Key: SPARK-48181 URL: https://issues.apache.org/jira/browse/SPARK-48181 Project: Spark Issue Type: New Fea

[jira] [Created] (SPARK-48147) Remove all client listeners when local spark session is deleted

2024-05-06 Thread Wei Liu (Jira)
Wei Liu created SPARK-48147: --- Summary: Remove all client listeners when local spark session is deleted Key: SPARK-48147 URL: https://issues.apache.org/jira/browse/SPARK-48147 Project: Spark Issue

[jira] [Created] (SPARK-48093) Add config to switch between client side listener and server side listener

2024-05-02 Thread Wei Liu (Jira)
Wei Liu created SPARK-48093: --- Summary: Add config to switch between client side listener and server side listener Key: SPARK-48093 URL: https://issues.apache.org/jira/browse/SPARK-48093 Project: Spark

[jira] [Created] (SPARK-48002) Add Observed metrics test in PySpark StreamingQueryListeners

2024-04-25 Thread Wei Liu (Jira)
Wei Liu created SPARK-48002: --- Summary: Add Observed metrics test in PySpark StreamingQueryListeners Key: SPARK-48002 URL: https://issues.apache.org/jira/browse/SPARK-48002 Project: Spark Issue Typ

[jira] [Created] (SPARK-47877) Speed up test_parity_listener

2024-04-16 Thread Wei Liu (Jira)
Wei Liu created SPARK-47877: --- Summary: Speed up test_parity_listener Key: SPARK-47877 URL: https://issues.apache.org/jira/browse/SPARK-47877 Project: Spark Issue Type: New Feature Compone

[jira] [Updated] (SPARK-47718) .sql() does not recognize watermark defined upstream

2024-04-09 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-47718: Labels: (was: pull-request-available) > .sql() does not recognize watermark defined upstream > -

[jira] [Created] (SPARK-47722) Wait until RocksDB background work finish before closing

2024-04-03 Thread Wei Liu (Jira)
Wei Liu created SPARK-47722: --- Summary: Wait until RocksDB background work finish before closing Key: SPARK-47722 URL: https://issues.apache.org/jira/browse/SPARK-47722 Project: Spark Issue Type: N

[jira] [Created] (SPARK-47332) StreamingPythonRunner don't need redundant logic for starting python process

2024-03-08 Thread Wei Liu (Jira)
Wei Liu created SPARK-47332: --- Summary: StreamingPythonRunner don't need redundant logic for starting python process Key: SPARK-47332 URL: https://issues.apache.org/jira/browse/SPARK-47332 Project: Spark

[jira] [Created] (SPARK-47292) safeMapToJValue should consider when map is null

2024-03-05 Thread Wei Liu (Jira)
Wei Liu created SPARK-47292: --- Summary: safeMapToJValue should consider when map is null Key: SPARK-47292 URL: https://issues.apache.org/jira/browse/SPARK-47292 Project: Spark Issue Type: New Featur

[jira] [Created] (SPARK-47277) PySpark util function assertDataFrameEqual should not support streaming DF

2024-03-04 Thread Wei Liu (Jira)
Wei Liu created SPARK-47277: --- Summary: PySpark util function assertDataFrameEqual should not support streaming DF Key: SPARK-47277 URL: https://issues.apache.org/jira/browse/SPARK-47277 Project: Spark

[jira] [Created] (SPARK-47174) Client Side Listener - Server side implementation

2024-02-26 Thread Wei Liu (Jira)
Wei Liu created SPARK-47174: --- Summary: Client Side Listener - Server side implementation Key: SPARK-47174 URL: https://issues.apache.org/jira/browse/SPARK-47174 Project: Spark Issue Type: Improveme

[jira] [Commented] (SPARK-47174) Client Side Listener - Server side implementation

2024-02-26 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820828#comment-17820828 ] Wei Liu commented on SPARK-47174: - im working on this > Client Side Listener - Server s

[jira] [Created] (SPARK-47173) fix typo in new streaming query listener explanation

2024-02-26 Thread Wei Liu (Jira)
Wei Liu created SPARK-47173: --- Summary: fix typo in new streaming query listener explanation Key: SPARK-47173 URL: https://issues.apache.org/jira/browse/SPARK-47173 Project: Spark Issue Type: Improv

[jira] [Created] (SPARK-46873) PySpark spark.streams should not recreate new StreamingQueryManager

2024-01-25 Thread Wei Liu (Jira)
Wei Liu created SPARK-46873: --- Summary: PySpark spark.streams should not recreate new StreamingQueryManager Key: SPARK-46873 URL: https://issues.apache.org/jira/browse/SPARK-46873 Project: Spark Is

[jira] [Closed] (SPARK-44460) Pass user auth credential to Python workers for foreachBatch and listener

2024-01-23 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu closed SPARK-44460. --- > Pass user auth credential to Python workers for foreachBatch and listener > --

[jira] [Updated] (SPARK-46627) Streaming UI hover-over shows incorrect value

2024-01-08 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-46627: Attachment: Screenshot 2024-01-08 at 15.06.24.png > Streaming UI hover-over shows incorrect value > --

[jira] [Commented] (SPARK-46627) Streaming UI hover-over shows incorrect value

2024-01-08 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804513#comment-17804513 ] Wei Liu commented on SPARK-46627: - Also batch percent doesn't add to 100% now: !Screens

[jira] [Updated] (SPARK-46627) Streaming UI hover-over shows incorrect value

2024-01-08 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-46627: Attachment: Screenshot 2024-01-08 at 1.55.57 PM.png > Streaming UI hover-over shows incorrect value >

[jira] [Commented] (SPARK-46627) Streaming UI hover-over shows incorrect value

2024-01-08 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804481#comment-17804481 ] Wei Liu commented on SPARK-46627: - Hi Kent : ) [~yao]  I was wondering if you have cont

[jira] [Updated] (SPARK-46627) Streaming UI hover-over shows incorrect value

2024-01-08 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-46627: Description: Running a simple streaming query: val df = spark.readStream.format("rate").option("rowsPerSe

[jira] [Created] (SPARK-46627) Streaming UI hover-over shows incorrect value

2024-01-08 Thread Wei Liu (Jira)
Wei Liu created SPARK-46627: --- Summary: Streaming UI hover-over shows incorrect value Key: SPARK-46627 URL: https://issues.apache.org/jira/browse/SPARK-46627 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-46384) Streaming UI doesn't display graph correctly

2023-12-12 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-46384: Summary: Streaming UI doesn't display graph correctly (was: Streaming UI doesn't show graph) > Streaming

[jira] [Updated] (SPARK-46384) Structured Streaming UI doesn't display graph correctly

2023-12-12 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-46384: Summary: Structured Streaming UI doesn't display graph correctly (was: Streaming UI doesn't display graph

[jira] [Created] (SPARK-46384) Streaming UI doesn't show graph

2023-12-12 Thread Wei Liu (Jira)
Wei Liu created SPARK-46384: --- Summary: Streaming UI doesn't show graph Key: SPARK-46384 URL: https://issues.apache.org/jira/browse/SPARK-46384 Project: Spark Issue Type: Task Components:

[jira] [Created] (SPARK-46250) Deflake test_parity_listener

2023-12-04 Thread Wei Liu (Jira)
Wei Liu created SPARK-46250: --- Summary: Deflake test_parity_listener Key: SPARK-46250 URL: https://issues.apache.org/jira/browse/SPARK-46250 Project: Spark Issue Type: Task Components: Con

[jira] [Created] (SPARK-45845) Streaming UI add number of evicted state rows

2023-11-08 Thread Wei Liu (Jira)
Wei Liu created SPARK-45845: --- Summary: Streaming UI add number of evicted state rows Key: SPARK-45845 URL: https://issues.apache.org/jira/browse/SPARK-45845 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-45637) Time window aggregation in separate streams followed by stream-stream join not returning results

2023-10-26 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-45637: Description: According to documentation update (SPARK-42591) resulting from SPARK-42376, Spark 3.5.0 shou

[jira] [Created] (SPARK-45677) Observe API error logging

2023-10-26 Thread Wei Liu (Jira)
Wei Liu created SPARK-45677: --- Summary: Observe API error logging Key: SPARK-45677 URL: https://issues.apache.org/jira/browse/SPARK-45677 Project: Spark Issue Type: Task Components: Struct

[jira] [Updated] (SPARK-45053) Improve python version mismatch logging

2023-09-01 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-45053: Description: Currently the syntax of the python version mismatching is a little bit confusing, it uses (3,

[jira] [Created] (SPARK-45056) Add process termination tests for Python foreachBatch and StreamingQueryListener

2023-09-01 Thread Wei Liu (Jira)
Wei Liu created SPARK-45056: --- Summary: Add process termination tests for Python foreachBatch and StreamingQueryListener Key: SPARK-45056 URL: https://issues.apache.org/jira/browse/SPARK-45056 Project: Spark

[jira] [Created] (SPARK-45053) Improve python version mismatch logging

2023-09-01 Thread Wei Liu (Jira)
Wei Liu created SPARK-45053: --- Summary: Improve python version mismatch logging Key: SPARK-45053 URL: https://issues.apache.org/jira/browse/SPARK-45053 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-44971) [BUG Fix] PySpark StreamingQuerProgress fromJson

2023-08-25 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44971: Issue Type: Bug (was: New Feature) > [BUG Fix] PySpark StreamingQuerProgress fromJson >

[jira] [Created] (SPARK-44971) [BUG Fix] PySpark StreamingQuerProgress fromJson

2023-08-25 Thread Wei Liu (Jira)
Wei Liu created SPARK-44971: --- Summary: [BUG Fix] PySpark StreamingQuerProgress fromJson Key: SPARK-44971 URL: https://issues.apache.org/jira/browse/SPARK-44971 Project: Spark Issue Type: New Featu

[jira] [Resolved] (SPARK-44917) PySpark Streaming DataStreamWriter table API

2023-08-22 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu resolved SPARK-44917. - Resolution: Not A Problem > PySpark Streaming DataStreamWriter table API > -

[jira] [Created] (SPARK-44917) PySpark Streaming DataStreamWriter table API

2023-08-22 Thread Wei Liu (Jira)
Wei Liu created SPARK-44917: --- Summary: PySpark Streaming DataStreamWriter table API Key: SPARK-44917 URL: https://issues.apache.org/jira/browse/SPARK-44917 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-44460) Pass user auth credential to Python workers for foreachBatch and listener

2023-08-21 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17757087#comment-17757087 ] Wei Liu commented on SPARK-44460: - [~rangadi] This seems to be a Databricks internal iss

[jira] [Created] (SPARK-44839) Better error logging when user accesses spark session in foreachBatch and Listener

2023-08-16 Thread Wei Liu (Jira)
Wei Liu created SPARK-44839: --- Summary: Better error logging when user accesses spark session in foreachBatch and Listener Key: SPARK-44839 URL: https://issues.apache.org/jira/browse/SPARK-44839 Project: Spa

[jira] [Commented] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17754280#comment-17754280 ] Wei Liu commented on SPARK-44808: - This seems to be against the design principle of spar

[jira] [Resolved] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu resolved SPARK-44808. - Resolution: Won't Do > refreshListener() API on StreamingQueryManager for spark connect > --

[jira] [Updated] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44808: Description: I’m thinking of an improvement for connect python listener and foreachBatch. Currently if yo

[jira] [Updated] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44808: Description: I’m thinking of an improvement for connect python listener and foreachBatch. Currently if yo

[jira] [Updated] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44808: Description: I’m thinking of an improvement for connect python listener and foreachBatch. Currently if yo

[jira] [Updated] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44808: Description: I’m thinking of an improvement for connect python listener and foreachBatch. Currently if yo

[jira] [Created] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
Wei Liu created SPARK-44808: --- Summary: refreshListener() API on StreamingQueryManager for spark connect Key: SPARK-44808 URL: https://issues.apache.org/jira/browse/SPARK-44808 Project: Spark Issue

[jira] [Updated] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44808: Description: I’m thinking of an improvement for python listener and foreachBatch. Currently if you define

[jira] [Created] (SPARK-44764) Streaming process improvement

2023-08-10 Thread Wei Liu (Jira)
Wei Liu created SPARK-44764: --- Summary: Streaming process improvement Key: SPARK-44764 URL: https://issues.apache.org/jira/browse/SPARK-44764 Project: Spark Issue Type: New Feature Compone

[jira] [Updated] (SPARK-44460) Pass user auth credential to Python workers for foreachBatch and listener

2023-07-28 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44460: Description: No user specific credentials are sent to Python worker that runs user functions like foreach

[jira] [Updated] (SPARK-44516) Spark Connect Python StreamingQueryListener removeListener method actually shut down the listener process

2023-07-24 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44516: Summary: Spark Connect Python StreamingQueryListener removeListener method actually shut down the listener

[jira] [Created] (SPARK-44516) Spark Connect Python StreamingQueryListener removeListener method

2023-07-23 Thread Wei Liu (Jira)
Wei Liu created SPARK-44516: --- Summary: Spark Connect Python StreamingQueryListener removeListener method Key: SPARK-44516 URL: https://issues.apache.org/jira/browse/SPARK-44516 Project: Spark Issu

[jira] [Created] (SPARK-44515) Code Improvement: PySpark add util function to set python version

2023-07-23 Thread Wei Liu (Jira)
Wei Liu created SPARK-44515: --- Summary: Code Improvement: PySpark add util function to set python version Key: SPARK-44515 URL: https://issues.apache.org/jira/browse/SPARK-44515 Project: Spark Issu

[jira] [Created] (SPARK-44502) Add mission versionchanged field to docs

2023-07-20 Thread Wei Liu (Jira)
Wei Liu created SPARK-44502: --- Summary: Add mission versionchanged field to docs Key: SPARK-44502 URL: https://issues.apache.org/jira/browse/SPARK-44502 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-44484) Add missing json field batchDuration to StreamingQueryProgress

2023-07-19 Thread Wei Liu (Jira)
Wei Liu created SPARK-44484: --- Summary: Add missing json field batchDuration to StreamingQueryProgress Key: SPARK-44484 URL: https://issues.apache.org/jira/browse/SPARK-44484 Project: Spark Issue T

[jira] [Commented] (SPARK-42962) Allow access to stopped streaming queries

2023-06-21 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735867#comment-17735867 ] Wei Liu commented on SPARK-42962: - Closing this as it's duplicate with SPARK-42940. Plea

[jira] [Resolved] (SPARK-42962) Allow access to stopped streaming queries

2023-06-21 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu resolved SPARK-42962. - Resolution: Duplicate > Allow access to stopped streaming queries >

[jira] [Commented] (SPARK-42941) Add support for streaming listener in Python

2023-06-21 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735866#comment-17735866 ] Wei Liu commented on SPARK-42941: - Hmmm this is actually still in progress. I was thinki

[jira] [Reopened] (SPARK-42941) Add support for streaming listener in Python

2023-06-21 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu reopened SPARK-42941: - > Add support for streaming listener in Python > > >

[jira] [Created] (SPARK-44046) Pyspark StreamingQueryListener listListener

2023-06-13 Thread Wei Liu (Jira)
Wei Liu created SPARK-44046: --- Summary: Pyspark StreamingQueryListener listListener Key: SPARK-44046 URL: https://issues.apache.org/jira/browse/SPARK-44046 Project: Spark Issue Type: New Feature

[jira] [Resolved] (SPARK-44029) Observation.py supports streaming dataframes

2023-06-12 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu resolved SPARK-44029. - Resolution: Not A Problem > Observation.py supports streaming dataframes > -

[jira] [Created] (SPARK-44029) Observation.py supports streaming dataframes

2023-06-12 Thread Wei Liu (Jira)
Wei Liu created SPARK-44029: --- Summary: Observation.py supports streaming dataframes Key: SPARK-44029 URL: https://issues.apache.org/jira/browse/SPARK-44029 Project: Spark Issue Type: Documentation

[jira] [Created] (SPARK-44010) Python StreamingQueryProgress rowsPerSecond type fix

2023-06-08 Thread Wei Liu (Jira)
Wei Liu created SPARK-44010: --- Summary: Python StreamingQueryProgress rowsPerSecond type fix Key: SPARK-44010 URL: https://issues.apache.org/jira/browse/SPARK-44010 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-43796) Streaming ForeachWriter can't accept custom user defined class

2023-05-25 Thread Wei Liu (Jira)
Wei Liu created SPARK-43796: --- Summary: Streaming ForeachWriter can't accept custom user defined class Key: SPARK-43796 URL: https://issues.apache.org/jira/browse/SPARK-43796 Project: Spark Issue T

[jira] [Created] (SPARK-43761) Streaming ForeachWriter with UnboundRowEncoder

2023-05-23 Thread Wei Liu (Jira)
Wei Liu created SPARK-43761: --- Summary: Streaming ForeachWriter with UnboundRowEncoder Key: SPARK-43761 URL: https://issues.apache.org/jira/browse/SPARK-43761 Project: Spark Issue Type: Task

  1   2   >