[jira] [Commented] (SPARK-27017) Creating orc table with special symbols in column name via spark.sql

2019-04-01 Thread Henryk Cesnolovic (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807438#comment-16807438 ] Henryk Cesnolovic commented on SPARK-27017: --- Hello, I have checked this one. S

[jira] [Comment Edited] (SPARK-27017) Creating orc table with special symbols in column name via spark.sql

2019-04-01 Thread Henryk Cesnolovic (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807438#comment-16807438 ] Henryk Cesnolovic edited comment on SPARK-27017 at 4/2/19 6:43 AM: ---

[jira] [Created] (SPARK-27345) Expose approx_percentile in python api

2019-04-01 Thread Roi Zaig (JIRA)
Roi Zaig created SPARK-27345: Summary: Expose approx_percentile in python api Key: SPARK-27345 URL: https://issues.apache.org/jira/browse/SPARK-27345 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-27220) Remove Yarn specific leftover from CoarseGrainedSchedulerBackend

2019-04-01 Thread Jacek Lewandowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807434#comment-16807434 ] Jacek Lewandowski commented on SPARK-27220: --- I know it looks very minor, but i

[jira] [Created] (SPARK-27344) Support the LocalDate and Instant classes in Java Bean encoders

2019-04-01 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-27344: -- Summary: Support the LocalDate and Instant classes in Java Bean encoders Key: SPARK-27344 URL: https://issues.apache.org/jira/browse/SPARK-27344 Project: Spark

[jira] [Created] (SPARK-27343) Use ConfigEntry for hardcoded configs for spark-sql-kafka

2019-04-01 Thread hehuiyuan (JIRA)
hehuiyuan created SPARK-27343: - Summary: Use ConfigEntry for hardcoded configs for spark-sql-kafka Key: SPARK-27343 URL: https://issues.apache.org/jira/browse/SPARK-27343 Project: Spark Issue Typ

[jira] [Created] (SPARK-27342) Optimize limit 0 queries

2019-04-01 Thread Aayushmaan Jain (JIRA)
Aayushmaan Jain created SPARK-27342: --- Summary: Optimize limit 0 queries Key: SPARK-27342 URL: https://issues.apache.org/jira/browse/SPARK-27342 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-27341) dead lock between TaskMemoryManager and UnsafeExternalSorter

2019-04-01 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-27341: --- Summary: dead lock between TaskMemoryManager and UnsafeExternalSorter Key: SPARK-27341 URL: https://issues.apache.org/jira/browse/SPARK-27341 Project: Spark I

[jira] [Resolved] (SPARK-26224) Results in stackOverFlowError when trying to add 3000 new columns using withColumn function of dataframe.

2019-04-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26224. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23285 [https://gi

[jira] [Assigned] (SPARK-26224) Results in stackOverFlowError when trying to add 3000 new columns using withColumn function of dataframe.

2019-04-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-26224: Assignee: Marco Gaido > Results in stackOverFlowError when trying to add 3000 new columns

[jira] [Commented] (SPARK-27337) QueryExecutionListener never cleans up listeners from the bus after SparkSession is closed

2019-04-01 Thread Sandeep Katta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807406#comment-16807406 ] Sandeep Katta commented on SPARK-27337: ---  I will test this scenario and if require

[jira] [Commented] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue

2019-04-01 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807400#comment-16807400 ] Lantao Jin commented on SPARK-27216: [~irashid] Not yet. It's petty challenging to f

[jira] [Updated] (SPARK-27340) Alias on TimeWIndow expression may cause watermark metadata lost

2019-04-01 Thread Kevin Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Zhang updated SPARK-27340: Description: When we use data api to write a structured streaming query job we usually specify a

[jira] [Updated] (SPARK-27340) Alias on TimeWIndow expression may cause watermark metadata lost

2019-04-01 Thread Kevin Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Zhang updated SPARK-27340: Description: When we use data api to write a structured streaming query job we usually specify a

[jira] [Created] (SPARK-27340) Alias on TimeWIndow expression may cause watermark metadata lost

2019-04-01 Thread Kevin Zhang (JIRA)
Kevin Zhang created SPARK-27340: --- Summary: Alias on TimeWIndow expression may cause watermark metadata lost Key: SPARK-27340 URL: https://issues.apache.org/jira/browse/SPARK-27340 Project: Spark

[jira] [Assigned] (SPARK-25496) Deprecate from_utc_timestamp and to_utc_timestamp

2019-04-01 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-25496: --- Assignee: Maxim Gekk > Deprecate from_utc_timestamp and to_utc_timestamp >

[jira] [Resolved] (SPARK-25496) Deprecate from_utc_timestamp and to_utc_timestamp

2019-04-01 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-25496. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24195 [https://gith

[jira] [Resolved] (SPARK-27325) Support implicit encoders for LocalDate and Instant

2019-04-01 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27325. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24249 [https://gith

[jira] [Assigned] (SPARK-27325) Support implicit encoders for LocalDate and Instant

2019-04-01 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27325: --- Assignee: Maxim Gekk > Support implicit encoders for LocalDate and Instant > --

[jira] [Commented] (SPARK-27330) ForeachWriter is not being closed once a batch is aborted

2019-04-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807316#comment-16807316 ] Hyukjin Kwon commented on SPARK-27330: -- (I think {{close()}} should be idempotent a

[jira] [Updated] (SPARK-27330) ForeachWriter is not being closed once a batch is aborted

2019-04-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27330: - Description: in cases where a micro batch is being killed (interrupted), not during actual proc

[jira] [Comment Edited] (SPARK-27330) ForeachWriter is not being closed once a batch is aborted

2019-04-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806654#comment-16806654 ] Hyukjin Kwon edited comment on SPARK-27330 at 4/2/19 12:27 AM: ---

[jira] [Commented] (SPARK-27335) cannot collect() from Correlation.corr

2019-04-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807312#comment-16807312 ] Hyukjin Kwon commented on SPARK-27335: -- Can you show how to reproduce from scratch?

[jira] [Commented] (SPARK-27339) Decimal up cast to higher scale fails while reading parquet to Dataset

2019-04-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807307#comment-16807307 ] Hyukjin Kwon commented on SPARK-27339: -- [~wrschneider99], you don't have to clone i

[jira] [Commented] (SPARK-25079) [PYTHON] upgrade python 3.4 -> 3.6

2019-04-01 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807245#comment-16807245 ] shane knapp commented on SPARK-25079: - UPDATE! i successfully created and extensive

[jira] [Updated] (SPARK-25079) [PYTHON] upgrade python 3.4 -> 3.6

2019-04-01 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp updated SPARK-25079: Summary: [PYTHON] upgrade python 3.4 -> 3.6 (was: [PYTHON] upgrade python 3.4 -> 3.5) > [PYTHON]

[jira] [Updated] (SPARK-27339) Decimal up cast to higher scale fails while reading parquet to Dataset

2019-04-01 Thread Bill Schneider (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Schneider updated SPARK-27339: --- Affects Version/s: 2.4.0 I am cloning/reopening this issue - the argument for why this shoul

[jira] [Created] (SPARK-27339) CLONE - Decimal up cast to higher scale fails while reading parquet to Dataset

2019-04-01 Thread Bill Schneider (JIRA)
Bill Schneider created SPARK-27339: -- Summary: CLONE - Decimal up cast to higher scale fails while reading parquet to Dataset Key: SPARK-27339 URL: https://issues.apache.org/jira/browse/SPARK-27339 Pr

[jira] [Updated] (SPARK-27339) Decimal up cast to higher scale fails while reading parquet to Dataset

2019-04-01 Thread Bill Schneider (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Schneider updated SPARK-27339: --- Summary: Decimal up cast to higher scale fails while reading parquet to Dataset (was: CLONE

[jira] [Resolved] (SPARK-27329) Pruning nested field in map of map key and value from object serializers

2019-04-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27329. --- Resolution: Fixed Assignee: Liang-Chi Hsieh Fix Version/s: 3.0.0 This is res

[jira] [Commented] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue

2019-04-01 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807181#comment-16807181 ] Imran Rashid commented on SPARK-27216: -- Any chance you know what the issue in roari

[jira] [Comment Edited] (SPARK-25003) Pyspark Does not use Spark Sql Extensions

2019-04-01 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807121#comment-16807121 ] Russell Spitzer edited comment on SPARK-25003 at 4/1/19 7:48 PM: -

[jira] [Commented] (SPARK-25003) Pyspark Does not use Spark Sql Extensions

2019-04-01 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807121#comment-16807121 ] Russell Spitzer commented on SPARK-25003: - There was no interest in putting in O

[jira] [Updated] (SPARK-12611) test_infer_schema_to_local depended on old handling of missing value in row

2019-04-01 Thread John Andrunas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Andrunas updated SPARK-12611: -- Attachment: (was: Network Management Downloads.zip) > test_infer_schema_to_local depended

[jira] [Commented] (SPARK-25003) Pyspark Does not use Spark Sql Extensions

2019-04-01 Thread Sunitha Kambhampati (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807073#comment-16807073 ] Sunitha Kambhampati commented on SPARK-25003: - Hi [~hyukjin.kwon], [~rspitze

[jira] [Resolved] (SPARK-9792) PySpark DenseMatrix, SparseMatrix should override __eq__

2019-04-01 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-9792. Resolution: Fixed Fix Version/s: 3.0.0 > PySpark DenseMatrix, SparseMatrix should override __eq__ >

[jira] [Updated] (SPARK-27337) QueryExecutionListener never cleans up listeners from the bus after SparkSession is closed

2019-04-01 Thread Vinoo Ganesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoo Ganesh updated SPARK-27337: - Affects Version/s: (was: 2.4.0) > QueryExecutionListener never cleans up listeners from the

[jira] [Created] (SPARK-27338) Deadlock between TaskMemoryManager and UnsafeExternalSorter$SpillableIterator

2019-04-01 Thread Venkata krishnan Sowrirajan (JIRA)
Venkata krishnan Sowrirajan created SPARK-27338: --- Summary: Deadlock between TaskMemoryManager and UnsafeExternalSorter$SpillableIterator Key: SPARK-27338 URL: https://issues.apache.org/jira/browse/SP

[jira] [Commented] (SPARK-27338) Deadlock between TaskMemoryManager and UnsafeExternalSorter$SpillableIterator

2019-04-01 Thread Venkata krishnan Sowrirajan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806946#comment-16806946 ] Venkata krishnan Sowrirajan commented on SPARK-27338: - We found this

[jira] [Commented] (SPARK-27336) Incorrect DataSet.summary() result

2019-04-01 Thread Chakravarthi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806947#comment-16806947 ] Chakravarthi commented on SPARK-27336: -- I'm checking this issue. > Incorrect DataS

[jira] [Created] (SPARK-27337) QueryExecutionListener never cleans up listeners from the bus after SparkSession is closed

2019-04-01 Thread Vinoo Ganesh (JIRA)
Vinoo Ganesh created SPARK-27337: Summary: QueryExecutionListener never cleans up listeners from the bus after SparkSession is closed Key: SPARK-27337 URL: https://issues.apache.org/jira/browse/SPARK-27337

[jira] [Updated] (SPARK-27337) QueryExecutionListener never cleans up listeners from the bus after SparkSession is closed

2019-04-01 Thread Vinoo Ganesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoo Ganesh updated SPARK-27337: - Attachment: image001-1.png > QueryExecutionListener never cleans up listeners from the bus after

[jira] [Resolved] (SPARK-27278) Optimize GetMapValue when the map is a foldable and the key is not

2019-04-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27278. --- Resolution: Fixed Assignee: Marco Gaido Fix Version/s: 3.0.0 This is resolve

[jira] [Updated] (SPARK-27278) Optimize GetMapValue when the map is a foldable and the key is not

2019-04-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27278: -- Summary: Optimize GetMapValue when the map is a foldable and the key is not (was: SimplifyExt

[jira] [Updated] (SPARK-27278) SimplifyExtractValueOps doesn't optimise Literal map access to CASE WHEN ... END

2019-04-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27278: -- Labels: (was: Optimization) > SimplifyExtractValueOps doesn't optimise Literal map access to

[jira] [Updated] (SPARK-27278) SimplifyExtractValueOps doesn't optimise Literal map access to CASE WHEN ... END

2019-04-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27278: -- Issue Type: Improvement (was: Bug) > SimplifyExtractValueOps doesn't optimise Literal map acc

[jira] [Updated] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue

2019-04-01 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-27216: --- Summary: Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue (was: Upgrade RoaringBit

[jira] [Assigned] (SPARK-26812) PushProjectionThroughUnion nullability issue

2019-04-01 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-26812: --- Assignee: Marco Gaido > PushProjectionThroughUnion nullability issue >

[jira] [Resolved] (SPARK-26812) PushProjectionThroughUnion nullability issue

2019-04-01 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-26812. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23726 [https://gith

[jira] [Commented] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45

2019-04-01 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806841#comment-16806841 ] Lantao Jin commented on SPARK-27216: Thanks for reporting this [~aallen]. It's a bug

[jira] [Commented] (SPARK-27298) Dataset except operation gives different results(dataset count) on Spark 2.3.0 Windows and Spark 2.3.0 Linux environment

2019-04-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806766#comment-16806766 ] Sean Owen commented on SPARK-27298: --- I take it back, that's not the issue. There are

[jira] [Updated] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45

2019-04-01 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-27216: --- Description: HighlyCompressedMapStatus uses RoaringBitmap to record the empty blocks. But RoaringBi

[jira] [Updated] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45

2019-04-01 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-27216: --- Summary: Upgrade RoaringBitmap to 0.7.45 (was: Kryo serialization with RoaringBitmap) > Upgrade Ro

[jira] [Updated] (SPARK-27336) Incorrect DataSet.summary() result

2019-04-01 Thread Gengliang Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-27336: --- Attachment: test.csv > Incorrect DataSet.summary() result >

[jira] [Created] (SPARK-27336) Incorrect DataSet.summary() result

2019-04-01 Thread Gengliang Wang (JIRA)
Gengliang Wang created SPARK-27336: -- Summary: Incorrect DataSet.summary() result Key: SPARK-27336 URL: https://issues.apache.org/jira/browse/SPARK-27336 Project: Spark Issue Type: Improvemen

[jira] [Updated] (SPARK-27336) Incorrect DataSet.summary() result

2019-04-01 Thread Gengliang Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-27336: --- Issue Type: Bug (was: Improvement) > Incorrect DataSet.summary() result > -

[jira] [Commented] (SPARK-27330) ForeachWriter is not being closed once a batch is aborted

2019-04-01 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806659#comment-16806659 ] Gabor Somogyi commented on SPARK-27330: --- Now I see the point and makes sense from

[jira] [Commented] (SPARK-27330) ForeachWriter is not being closed once a batch is aborted

2019-04-01 Thread Eyal Zituny (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806654#comment-16806654 ] Eyal Zituny commented on SPARK-27330: - I have updated the description, eventually i

[jira] [Updated] (SPARK-27330) ForeachWriter is not being closed once a batch is aborted

2019-04-01 Thread Eyal Zituny (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eyal Zituny updated SPARK-27330: Description: in cases where a micro batch is being killed (interrupted), not during actual proces

[jira] [Commented] (SPARK-27330) ForeachWriter is not being closed once a batch is aborted

2019-04-01 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806638#comment-16806638 ] Gabor Somogyi commented on SPARK-27330: --- If processing throws any kind of exceptio

[jira] [Comment Edited] (SPARK-2647) DAGScheduler plugs others when processing one JobSubmitted event

2019-04-01 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806554#comment-16806554 ] Sujith Chacko edited comment on SPARK-2647 at 4/1/19 9:32 AM: -

[jira] [Comment Edited] (SPARK-2647) DAGScheduler plugs others when processing one JobSubmitted event

2019-04-01 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806554#comment-16806554 ] Sujith Chacko edited comment on SPARK-2647 at 4/1/19 9:30 AM: -

[jira] [Commented] (SPARK-2647) DAGScheduler plugs others when processing one JobSubmitted event

2019-04-01 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806554#comment-16806554 ] Sujith Chacko commented on SPARK-2647: -- [~joshrosen] - May i know the Jira is where

[jira] [Comment Edited] (SPARK-2647) DAGScheduler plugs others when processing one JobSubmitted event

2019-04-01 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806554#comment-16806554 ] Sujith Chacko edited comment on SPARK-2647 at 4/1/19 9:30 AM: -

[jira] [Created] (SPARK-27335) cannot collect() from Correlation.corr

2019-04-01 Thread Natalino Busa (JIRA)
Natalino Busa created SPARK-27335: - Summary: cannot collect() from Correlation.corr Key: SPARK-27335 URL: https://issues.apache.org/jira/browse/SPARK-27335 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-27208) RestSubmissionClient only supports http

2019-04-01 Thread Jorge Machado (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806495#comment-16806495 ] Jorge Machado commented on SPARK-27208: --- Any help here please ? I really think thi

[jira] [Updated] (SPARK-27334) Support specify scheduler name for executor pods when submit

2019-04-01 Thread TommyLike (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] TommyLike updated SPARK-27334: -- Description: Currently, there are some external schedulers which bring a lot a great value into kubern

[jira] [Created] (SPARK-27334) Support specify scheduler name for executor pods when submit

2019-04-01 Thread TommyLike (JIRA)
TommyLike created SPARK-27334: - Summary: Support specify scheduler name for executor pods when submit Key: SPARK-27334 URL: https://issues.apache.org/jira/browse/SPARK-27334 Project: Spark Issue