[jira] [Commented] (SPARK-21074) Parquet files are read fully even though only count() is requested

[ https://issues.apache.org/jira/browse/SPARK-21074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16053532#comment-16053532 ] Kazuaki Ishizaki commented on SPARK-21074: -- I think that [~spektom] expected the

[jira] [Created] (SPARK-21217) Support ColumnVector.Array.toArray()

Kazuaki Ishizaki created SPARK-21217: Summary: Support ColumnVector.Array.toArray() Key: SPARK-21217 URL: https://issues.apache.org/jira/browse/SPARK-21217 Project: Spark Issue Type: Impr

[jira] [Updated] (SPARK-20783) Enhance ColumnVector to support compressed representation

[ https://issues.apache.org/jira/browse/SPARK-20783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-20783: - Summary: Enhance ColumnVector to support compressed representation (was: Enhance ColumnV

[jira] [Updated] (SPARK-20783) Enhance ColumnVector to support compressed representation

[ https://issues.apache.org/jira/browse/SPARK-20783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-20783: - Description: Current {{ColumnVector}} handles uncompressed data for parquet. For handlin

[jira] [Commented] (SPARK-21271) UnsafeRow.hashCode assertion when sizeInBytes not multiple of 8

[ https://issues.apache.org/jira/browse/SPARK-21271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070933#comment-16070933 ] Kazuaki Ishizaki commented on SPARK-21271: -- In {{UnsafeRow}}, the length of fixe

[jira] [Commented] (SPARK-21271) UnsafeRow.hashCode assertion when sizeInBytes not multiple of 8

[ https://issues.apache.org/jira/browse/SPARK-21271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070961#comment-16070961 ] Kazuaki Ishizaki commented on SPARK-21271: -- I see. For var-length part, its regu

[jira] [Commented] (SPARK-21271) UnsafeRow.hashCode assertion when sizeInBytes not multiple of 8

[ https://issues.apache.org/jira/browse/SPARK-21271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070966#comment-16070966 ] Kazuaki Ishizaki commented on SPARK-21271: -- I see. This issue comes from the vio

[jira] [Commented] (SPARK-21271) UnsafeRow.hashCode assertion when sizeInBytes not multiple of 8

[ https://issues.apache.org/jira/browse/SPARK-21271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070973#comment-16070973 ] Kazuaki Ishizaki commented on SPARK-21271: -- I see. I will work for this. Thank y

[jira] [Comment Edited] (SPARK-21271) UnsafeRow.hashCode assertion when sizeInBytes not multiple of 8

[ https://issues.apache.org/jira/browse/SPARK-21271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070973#comment-16070973 ] Kazuaki Ishizaki edited comment on SPARK-21271 at 7/1/17 3:18 AM: -

[jira] [Commented] (SPARK-21337) SQL which has large ‘case when’ expressions may cause code generation beyond 64KB

[ https://issues.apache.org/jira/browse/SPARK-21337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077952#comment-16077952 ] Kazuaki Ishizaki commented on SPARK-21337: -- In the master branch, I cannot see a

[jira] [Commented] (SPARK-21344) BinaryType comparison does signed byte array comparison

[ https://issues.apache.org/jira/browse/SPARK-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079010#comment-16079010 ] Kazuaki Ishizaki commented on SPARK-21344: -- I will work for this if anyone has f

[jira] [Commented] (SPARK-21337) SQL which has large ‘case when’ expressions may cause code generation beyond 64KB

[ https://issues.apache.org/jira/browse/SPARK-21337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079032#comment-16079032 ] Kazuaki Ishizaki commented on SPARK-21337: -- I cannot reproduce this by using the

[jira] [Commented] (SPARK-21364) IndexOutOfBoundsException on equality check of two complex array elements

[ https://issues.apache.org/jira/browse/SPARK-21364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16080963#comment-16080963 ] Kazuaki Ishizaki commented on SPARK-21364: -- When I ran the following test case t

[jira] [Updated] (SPARK-21373) Update Jetty to 9.3.20.v20170531 to fix CVE-2017-9735

[ https://issues.apache.org/jira/browse/SPARK-21373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-21373: - Description: This is derived from https://issues.apache.org/jira/browse/FELIX-5664. [~ar

[jira] [Created] (SPARK-21373) Update Jetty to 9.3.20.v20170531 to fix CVE-2017-9735

Kazuaki Ishizaki created SPARK-21373: Summary: Update Jetty to 9.3.20.v20170531 to fix CVE-2017-9735 Key: SPARK-21373 URL: https://issues.apache.org/jira/browse/SPARK-21373 Project: Spark

[jira] [Commented] (SPARK-21373) Update Jetty to 9.3.20.v20170531

[ https://issues.apache.org/jira/browse/SPARK-21373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082426#comment-16082426 ] Kazuaki Ishizaki commented on SPARK-21373: -- Since I have not clarified, I change

[jira] [Updated] (SPARK-21373) Update Jetty to 9.3.20.v20170531

[ https://issues.apache.org/jira/browse/SPARK-21373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-21373: - Summary: Update Jetty to 9.3.20.v20170531 (was: Update Jetty to 9.3.20.v20170531 to fix

[jira] [Comment Edited] (SPARK-21373) Update Jetty to 9.3.20.v20170531

[ https://issues.apache.org/jira/browse/SPARK-21373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082426#comment-16082426 ] Kazuaki Ishizaki edited comment on SPARK-21373 at 7/11/17 3:56 PM:

[jira] [Created] (SPARK-21387) org.apache.spark.memory.TaskMemoryManager.allocatePage causes OOM

Kazuaki Ishizaki created SPARK-21387: Summary: org.apache.spark.memory.TaskMemoryManager.allocatePage causes OOM Key: SPARK-21387 URL: https://issues.apache.org/jira/browse/SPARK-21387 Project: Sp

[jira] [Commented] (SPARK-21390) Dataset filter api inconsistency

[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084266#comment-16084266 ] Kazuaki Ishizaki commented on SPARK-21390: -- Thank you for reporting this. I can

[jira] [Comment Edited] (SPARK-21390) Dataset filter api inconsistency

[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084306#comment-16084306 ] Kazuaki Ishizaki edited comment on SPARK-21390 at 7/12/17 5:09 PM:

[jira] [Commented] (SPARK-21390) Dataset filter api inconsistency

[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084306#comment-16084306 ] Kazuaki Ishizaki commented on SPARK-21390: -- Another interesting results with Spa

[jira] [Commented] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset

[ https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084333#comment-16084333 ] Kazuaki Ishizaki commented on SPARK-21391: -- This program works with the master.

[jira] [Comment Edited] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset

[ https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084333#comment-16084333 ] Kazuaki Ishizaki edited comment on SPARK-21391 at 7/12/17 5:19 PM:

[jira] [Comment Edited] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset

[ https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084333#comment-16084333 ] Kazuaki Ishizaki edited comment on SPARK-21391 at 7/12/17 5:19 PM:

[jira] [Commented] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset

[ https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085099#comment-16085099 ] Kazuaki Ishizaki commented on SPARK-21391: -- [~hyukjin.kwon] I think that [SPARK

[jira] [Comment Edited] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset

[ https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085099#comment-16085099 ] Kazuaki Ishizaki edited comment on SPARK-21391 at 7/13/17 3:42 AM:

[jira] [Commented] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset

[ https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085342#comment-16085342 ] Kazuaki Ishizaki commented on SPARK-21391: -- [~neelrr] Do you want to have fix in

[jira] [Commented] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

[ https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085726#comment-16085726 ] Kazuaki Ishizaki commented on SPARK-21393: -- This program seems to require 7 csv

[jira] [Updated] (SPARK-21390) Dataset filter api inconsistency

[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-21390: - Affects Version/s: 2.2.0 > Dataset filter api inconsistency > ---

[jira] [Updated] (SPARK-21390) Dataset filter api inconsistency

[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-21390: - Affects Version/s: 2.1.0 > Dataset filter api inconsistency > ---

[jira] [Commented] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset

[ https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085939#comment-16085939 ] Kazuaki Ishizaki commented on SPARK-21391: -- I created [a PR|https://github.com/a

[jira] [Updated] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

[ https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-21393: - Affects Version/s: 2.2.0 > spark (pyspark) crashes unpredictably when using show() or toP

[jira] [Commented] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

[ https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16086189#comment-16086189 ] Kazuaki Ishizaki commented on SPARK-21393: -- Thank you for uploading files. When

[jira] [Comment Edited] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

[ https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16086189#comment-16086189 ] Kazuaki Ishizaki edited comment on SPARK-21393 at 7/13/17 6:39 PM:

[jira] [Commented] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

[ https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16086254#comment-16086254 ] Kazuaki Ishizaki commented on SPARK-21393: -- This program can cause the same exce

[jira] [Commented] (SPARK-21413) Multiple projections with CASE WHEN fails to run generated codes

[ https://issues.apache.org/jira/browse/SPARK-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16086927#comment-16086927 ] Kazuaki Ishizaki commented on SPARK-21413: -- Thank you for preparing a good repro

[jira] [Commented] (SPARK-21415) Triage scapegoat warnings, part 1

[ https://issues.apache.org/jira/browse/SPARK-21415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087097#comment-16087097 ] Kazuaki Ishizaki commented on SPARK-21415: -- Thank you. Is it better to create an

[jira] [Commented] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

[ https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087391#comment-16087391 ] Kazuaki Ishizaki commented on SPARK-21393: -- Not yet, however I created a patch n

[jira] [Commented] (SPARK-21418) NoSuchElementException: None.get on DataFrame.rdd

[ https://issues.apache.org/jira/browse/SPARK-21418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088659#comment-16088659 ] Kazuaki Ishizaki commented on SPARK-21418: -- I am curious why {java.io.ObjectOutp

[jira] [Comment Edited] (SPARK-21418) NoSuchElementException: None.get on DataFrame.rdd

[ https://issues.apache.org/jira/browse/SPARK-21418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088659#comment-16088659 ] Kazuaki Ishizaki edited comment on SPARK-21418 at 7/15/17 4:43 PM:

[jira] [Commented] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

[ https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088660#comment-16088660 ] Kazuaki Ishizaki commented on SPARK-21393: -- I confirmed that this python program

[jira] [Comment Edited] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

[ https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088660#comment-16088660 ] Kazuaki Ishizaki edited comment on SPARK-21393 at 7/15/17 4:53 PM:

[jira] [Comment Edited] (SPARK-21418) NoSuchElementException: None.get on DataFrame.rdd

[ https://issues.apache.org/jira/browse/SPARK-21418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088659#comment-16088659 ] Kazuaki Ishizaki edited comment on SPARK-21418 at 7/16/17 8:25 AM:

[jira] [Commented] (SPARK-21390) Dataset filter api inconsistency

[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088977#comment-16088977 ] Kazuaki Ishizaki commented on SPARK-21390: -- When I ran the following test suite

[jira] [Commented] (SPARK-21390) Dataset filter api inconsistency

[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089012#comment-16089012 ] Kazuaki Ishizaki commented on SPARK-21390: -- cc: [~ueshin] Is there any thought o

[jira] [Commented] (SPARK-21415) Triage scapegoat warnings, part 1

[ https://issues.apache.org/jira/browse/SPARK-21415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089915#comment-16089915 ] Kazuaki Ishizaki commented on SPARK-21415: -- I see. When another JIRA will happen

[jira] [Commented] (SPARK-21443) Very long planning duration for queries with lots of operations

[ https://issues.apache.org/jira/browse/SPARK-21443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090242#comment-16090242 ] Kazuaki Ishizaki commented on SPARK-21443: -- These two optimizations {{InferFilte

[jira] [Updated] (SPARK-20822) Generate code to get value from CachedBatchColumnVector in ColumnarBatch

[ https://issues.apache.org/jira/browse/SPARK-20822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-20822: - Summary: Generate code to get value from CachedBatchColumnVector in ColumnarBatch (was:

[jira] [Updated] (SPARK-20822) Generate code to get value from ColumnVector in ColumnarBatch

[ https://issues.apache.org/jira/browse/SPARK-20822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-20822: - Summary: Generate code to get value from ColumnVector in ColumnarBatch (was: Generate co

[jira] [Created] (SPARK-21512) DatasetCacheSuite need to execute unpersistent after executing peristent

Kazuaki Ishizaki created SPARK-21512: Summary: DatasetCacheSuite need to execute unpersistent after executing peristent Key: SPARK-21512 URL: https://issues.apache.org/jira/browse/SPARK-21512 Proj

[jira] [Updated] (SPARK-21512) DatasetCacheSuites need to execute unpersistent after executing peristent

[ https://issues.apache.org/jira/browse/SPARK-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-21512: - Summary: DatasetCacheSuites need to execute unpersistent after executing peristent (was:

[jira] [Updated] (SPARK-21512) DatasetCacheSuite needs to execute unpersistent after executing peristent

[ https://issues.apache.org/jira/browse/SPARK-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-21512: - Summary: DatasetCacheSuite needs to execute unpersistent after executing peristent (was:

[jira] [Commented] (SPARK-21512) DatasetCacheSuite needs to execute unpersistent after executing peristent

[ https://issues.apache.org/jira/browse/SPARK-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16097565#comment-16097565 ] Kazuaki Ishizaki commented on SPARK-21512: -- When {DatasetCacheSuite} is executed

[jira] [Comment Edited] (SPARK-21512) DatasetCacheSuite needs to execute unpersistent after executing peristent

[ https://issues.apache.org/jira/browse/SPARK-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16097565#comment-16097565 ] Kazuaki Ishizaki edited comment on SPARK-21512 at 7/24/17 4:53 AM:

[jira] [Created] (SPARK-21516) overriding afterEach() in DatasetCacheSuite must call super.afterEach()

Kazuaki Ishizaki created SPARK-21516: Summary: overriding afterEach() in DatasetCacheSuite must call super.afterEach() Key: SPARK-21516 URL: https://issues.apache.org/jira/browse/SPARK-21516 Proje

[jira] [Commented] (SPARK-21501) Spark shuffle index cache size should be memory based

[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098531#comment-16098531 ] Kazuaki Ishizaki commented on SPARK-21501: -- I guess that to use Spark 2.1 or lat

[jira] [Reopened] (SPARK-21387) org.apache.spark.memory.TaskMemoryManager.allocatePage causes OOM

[ https://issues.apache.org/jira/browse/SPARK-21387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki reopened SPARK-21387: -- > org.apache.spark.memory.TaskMemoryManager.allocatePage causes OOM > -

[jira] [Commented] (SPARK-21387) org.apache.spark.memory.TaskMemoryManager.allocatePage causes OOM

[ https://issues.apache.org/jira/browse/SPARK-21387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098545#comment-16098545 ] Kazuaki Ishizaki commented on SPARK-21387: -- While I got OOM in my unit test, I h

[jira] [Closed] (SPARK-21387) org.apache.spark.memory.TaskMemoryManager.allocatePage causes OOM

[ https://issues.apache.org/jira/browse/SPARK-21387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki closed SPARK-21387. Resolution: Fixed > org.apache.spark.memory.TaskMemoryManager.allocatePage causes OOM > ---

[jira] [Closed] (SPARK-21387) org.apache.spark.memory.TaskMemoryManager.allocatePage causes OOM

[ https://issues.apache.org/jira/browse/SPARK-21387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki closed SPARK-21387. Resolution: Cannot Reproduce > org.apache.spark.memory.TaskMemoryManager.allocatePage cause

[jira] [Commented] (SPARK-21501) Spark shuffle index cache size should be memory based

[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099351#comment-16099351 ] Kazuaki Ishizaki commented on SPARK-21501: -- I see. I misunderstood the descripti

[jira] [Commented] (SPARK-21517) Fetch local data via block manager cause oom

[ https://issues.apache.org/jira/browse/SPARK-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099654#comment-16099654 ] Kazuaki Ishizaki commented on SPARK-21517: -- Does it occur in Spark 2.2? > Fetch

[jira] [Commented] (SPARK-21496) Support codegen for TakeOrderedAndProjectExec

[ https://issues.apache.org/jira/browse/SPARK-21496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099840#comment-16099840 ] Kazuaki Ishizaki commented on SPARK-21496: -- Is there any good benchmark program

[jira] [Commented] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16104284#comment-16104284 ] Kazuaki Ishizaki commented on SPARK-18016: -- [~jamcon] Thank you reporting the pr

[jira] [Commented] (SPARK-21591) Implement treeAggregate on Dataset API

[ https://issues.apache.org/jira/browse/SPARK-21591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108644#comment-16108644 ] Kazuaki Ishizaki commented on SPARK-21591: -- I like this idea > Implement treeAg

[jira] [Commented] (SPARK-21390) Dataset filter api inconsistency

[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110576#comment-16110576 ] Kazuaki Ishizaki commented on SPARK-21390: -- Thank you very much for pointing out

[jira] [Created] (SPARK-16213) Reduce runtime overhead of a program that creates an primitive array in DataFrame

Kazuaki Ishizaki created SPARK-16213: Summary: Reduce runtime overhead of a program that creates an primitive array in DataFrame Key: SPARK-16213 URL: https://issues.apache.org/jira/browse/SPARK-16213

[jira] [Created] (SPARK-16215) Reduce runtime overhead of a program that writes an primitive array in Dataframe/Dataset

Kazuaki Ishizaki created SPARK-16215: Summary: Reduce runtime overhead of a program that writes an primitive array in Dataframe/Dataset Key: SPARK-16215 URL: https://issues.apache.org/jira/browse/SPARK-16215

[jira] [Commented] (SPARK-16070) DataFrame/Parquet issues with primitive arrays

[ https://issues.apache.org/jira/browse/SPARK-16070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350005#comment-15350005 ] Kazuaki Ishizaki commented on SPARK-16070: -- I added two JIRA entries, which addr

[jira] [Created] (SPARK-16223) Codegen failure with a Dataframe program using an array

Kazuaki Ishizaki created SPARK-16223: Summary: Codegen failure with a Dataframe program using an array Key: SPARK-16223 URL: https://issues.apache.org/jira/browse/SPARK-16223 Project: Spark

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15360752#comment-15360752 ] Kazuaki Ishizaki commented on SPARK-15467: -- My [PR|https://github.com/janino-com

[jira] [Created] (SPARK-16412) Generate Java code that gets an array in each column of CachedBatch when DataFrame.cache() is called

Kazuaki Ishizaki created SPARK-16412: Summary: Generate Java code that gets an array in each column of CachedBatch when DataFrame.cache() is called Key: SPARK-16412 URL: https://issues.apache.org/jira/browse/S

[jira] [Updated] (SPARK-16412) Generate Java code that gets an array in each column of CachedBatch when DataFrame.cache() is called

[ https://issues.apache.org/jira/browse/SPARK-16412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-16412: - Issue Type: Improvement (was: Bug) > Generate Java code that gets an array in each colum

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369703#comment-15369703 ] Kazuaki Ishizaki commented on SPARK-15467: -- [Janino 3.0.0|https://mvnrepository

[jira] [Commented] (SPARK-15687) Columnar execution engine

[ https://issues.apache.org/jira/browse/SPARK-15687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376999#comment-15376999 ] Kazuaki Ishizaki commented on SPARK-15687: -- It would be good to introduce trait

[jira] [Comment Edited] (SPARK-15687) Columnar execution engine

[ https://issues.apache.org/jira/browse/SPARK-15687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376999#comment-15376999 ] Kazuaki Ishizaki edited comment on SPARK-15687 at 7/14/16 2:25 PM:

[jira] [Updated] (SPARK-16223) Codegen failure with a Dataframe program using an array

[ https://issues.apache.org/jira/browse/SPARK-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-16223: - Description: When we compile a Dataframe program with an operation to large array, compi

[jira] [Commented] (SPARK-16807) Optimize some ABS() statements

[ https://issues.apache.org/jira/browse/SPARK-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400532#comment-15400532 ] Kazuaki Ishizaki commented on SPARK-16807: -- Interesting if we can ensure {{x - y

[jira] [Created] (SPARK-27397) Take care of OpenJ9 in JVM dependant parts

Kazuaki Ishizaki created SPARK-27397: Summary: Take care of OpenJ9 in JVM dependant parts Key: SPARK-27397 URL: https://issues.apache.org/jira/browse/SPARK-27397 Project: Spark Issue Type

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815812#comment-16815812 ] Kazuaki Ishizaki commented on SPARK-27396: -- Sorry for being late to comment sin

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16817152#comment-16817152 ] Kazuaki Ishizaki commented on SPARK-27396: -- Thank you for sharing low-level API

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819181#comment-16819181 ] Kazuaki Ishizaki commented on SPARK-27396: -- I have one question regarding low-l

[jira] [Commented] (SPARK-27684) Reduce ScalaUDF conversion overheads for primitives

[ https://issues.apache.org/jira/browse/SPARK-27684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16838229#comment-16838229 ] Kazuaki Ishizaki commented on SPARK-27684: -- Interesting idea > Reduce ScalaUDF

[jira] [Commented] (SPARK-18147) Broken Spark SQL Codegen

[ https://issues.apache.org/jira/browse/SPARK-18147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15614959#comment-15614959 ] Kazuaki Ishizaki commented on SPARK-18147: -- This also cause the same exception.

[jira] [Commented] (SPARK-18125) Spark generated code causes CompileException when groupByKey, reduceGroups and map(_._2) are used

[ https://issues.apache.org/jira/browse/SPARK-18125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15615034#comment-15615034 ] Kazuaki Ishizaki commented on SPARK-18125: -- I confirmed this code can reproduce

[jira] [Commented] (SPARK-18207) class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection" grows beyond 64 KB

[ https://issues.apache.org/jira/browse/SPARK-18207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15629522#comment-15629522 ] Kazuaki Ishizaki commented on SPARK-18207: -- I created a smaller program to repro

[jira] [Created] (SPARK-18284) Scheme of DataFrame generated from RDD is diffrent between master and 2.0

Kazuaki Ishizaki created SPARK-18284: Summary: Scheme of DataFrame generated from RDD is diffrent between master and 2.0 Key: SPARK-18284 URL: https://issues.apache.org/jira/browse/SPARK-18284 Pro

[jira] [Updated] (SPARK-18284) Scheme of DataFrame generated from RDD is diffrent between master and 2.0

[ https://issues.apache.org/jira/browse/SPARK-18284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-18284: - Affects Version/s: 2.1.0 > Scheme of DataFrame generated from RDD is diffrent between mas

[jira] [Updated] (SPARK-18284) Scheme of DataFrame generated from RDD is diffrent between master and 2.0

[ https://issues.apache.org/jira/browse/SPARK-18284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-18284: - Description: When the following program is executed, a schema of dataframe is different

[jira] [Commented] (SPARK-18458) core dumped running Spark SQL on large data volume (100TB)

[ https://issues.apache.org/jira/browse/SPARK-18458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15668948#comment-15668948 ] Kazuaki Ishizaki commented on SPARK-18458: -- I work for this. > core dumped runn

[jira] [Commented] (SPARK-18458) core dumped running Spark SQL on large data volume (100TB)

[ https://issues.apache.org/jira/browse/SPARK-18458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670828#comment-15670828 ] Kazuaki Ishizaki commented on SPARK-18458: -- I see. I will do that. > core dumpe

[jira] [Commented] (SPARK-18458) core dumped running Spark SQL on large data volume (100TB)

[ https://issues.apache.org/jira/browse/SPARK-18458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671229#comment-15671229 ] Kazuaki Ishizaki commented on SPARK-18458: -- I worked with [~jfc...@us.ibm.com].

[jira] [Issue Comment Deleted] (SPARK-18458) core dumped running Spark SQL on large data volume (100TB)

[ https://issues.apache.org/jira/browse/SPARK-18458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-18458: - Comment: was deleted (was: I worked with [~jfc...@us.ibm.com]. Then, I identified that a

[jira] [Commented] (SPARK-18458) core dumped running Spark SQL on large data volume (100TB)

[ https://issues.apache.org/jira/browse/SPARK-18458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671237#comment-15671237 ] Kazuaki Ishizaki commented on SPARK-18458: -- I worked with [~jfc...@us.ibm.com].

[jira] [Commented] (SPARK-18492) GeneratedIterator grows beyond 64 KB

[ https://issues.apache.org/jira/browse/SPARK-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15674599#comment-15674599 ] Kazuaki Ishizaki commented on SPARK-18492: -- Can you post a small program that ca

[jira] [Comment Edited] (SPARK-18492) GeneratedIterator grows beyond 64 KB

[ https://issues.apache.org/jira/browse/SPARK-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15674599#comment-15674599 ] Kazuaki Ishizaki edited comment on SPARK-18492 at 11/17/16 7:31 PM: ---

[jira] [Updated] (SPARK-14098) Generate code that get a value in each column from CachedBatch when DataFrame.cache() is called

[ https://issues.apache.org/jira/browse/SPARK-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-14098: - Target Version/s: 2.0.0 > Generate code that get a value in each column from CachedBatch

[jira] [Updated] (SPARK-14098) Generate code that get a float/double value in each column from CachedBatch when DataFrame.cache() is called

[ https://issues.apache.org/jira/browse/SPARK-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-14098: - Summary: Generate code that get a float/double value in each column from CachedBatch when

[jira] [Updated] (SPARK-14098) Generate code that get a float/double value in each column from CachedBatch when DataFrame.cache() is called

[ https://issues.apache.org/jira/browse/SPARK-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-14098: - Description: When DataFrame.cache() is called, data is stored as column-oriented storage

  1   2   3   4   5   6   7   8   >