>
> I think the key issue is the format. The proposed 10-byte format doesn't
> seem like a standard and the one in Iceberg/Parquet does not support the
> required range by ANSI SQL: year 0001 to year . We should address this
> issue first. Note that Parquet has an INT96 timestamp that supports
Maybe we should discuss the key issues on the dev list as it's easy to lose
track of Google Doc comments.
I think all the proposals for adding new data types need to prove that the
new data type is common/standard in the ecosystem. This means 3 things:
- it has common/standard semantic. TIMESTAMP
https://github.com/apache/spark/pull/50437
IMO, it will be better to keep 2 separate commits, one undo revert and one fix,
so fix for guava is properly documented.
Also, while testing, I see that if I exit the shell and start it again, it
fails.
Thank you,
Vlad
On Mar 27, 2025, at 2:33 PM, H
Hi,
I'm trying to build the project, but I'm encountering multiple errors due
to long lines. Is this expected? I built the project a few weeks ago and
don’t recall seeing these errors.
Is anyone else experiencing the same issue?
[image: image.png]
Thanks in advance.
Thanks!!!
DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1
> On Mar 27, 2025, at 3:56 PM, Qi Tan wrote:
>
> Thanks DB,
>
> I just noticed a few more comments came in after I initiated the vote. I'm
> going to postpone the voting process and address those outstanding comments.
>
>
Thanks DB,
I just noticed a few more comments came in after I initiated the vote. I'm
going to postpone the voting process and address those outstanding comments.
Qi Tan
DB Tsai 于2025年3月27日周四 15:12写道：
> Hello Qi,
>
> I'm supportive of the NanoSecond Timestamps proposal; however, before we
> in
Hello Qi,
I'm supportive of the NanoSecond Timestamps proposal; however, before we
initiate the vote, there are a few outstanding comments in the SPIP document
that haven't been addressed yet. Since the vote is on the document itself,
could we resolve these items beforehand?
For example:
The d
Vlad, let's open a PR and discuss it there. We have many other committees
to review / help with as well.
On Fri, Mar 28, 2025 at 6:28 AM Rozov, Vlad
wrote:
> Hi Hyukjin,
>
> I open https://issues.apache.org/jira/browse/SPARK-51643 and
> https://issues.apache.org/jira/browse/SPARK-51644. Please a
Hi Hyukjin,
I open https://issues.apache.org/jira/browse/SPARK-51643 and
https://issues.apache.org/jira/browse/SPARK-51644. Please add more details to
the first JIRA. As far as I can see
https://github.com/vrozov/spark/tree/spark-shell should fix both JIRAs and if
not I’d like to understand wh
+1
On Thu, Mar 27, 2025 at 1:22 PM Qi Tan wrote:
> Hi all,
>
> I would like to start a vote on adding support for nanoseconds timestamps.
>
> *Discussion thread: *
> https://lists.apache.org/thread/y2vzrjl1499j5dvbpg3m81jxdhf4b6of
> *SPIP:*
> https://docs.google.com/document/d/1wjFsBdlV2YK75x7UO
Hi all,
I would like to start a vote on adding support for nanoseconds timestamps.
*Discussion thread: *
https://lists.apache.org/thread/y2vzrjl1499j5dvbpg3m81jxdhf4b6of
*SPIP:*
https://docs.google.com/document/d/1wjFsBdlV2YK75x7UOk2HhDOqWVA0yC7iEiqOMnNnxlA/edit?usp=sharing
*JIRA:* https://issue
Casting my own +1 (non-binding).
Angel, I echo what Wenchen said. Connectors and Spark interact via DSv2,
therefore it requires changes in that layer. It is going to be optional but
will make a ton of sense for many connectors, especially in modern open
table formats that decouple table metadata f
Thanks for the reply. That helps..
On Thu, Mar 27, 2025, 7:29 AM Wenchen Fan wrote:
> The file source in Spark has not been migrated to DS v2 yet and uses
> dedicated catalyst rules to do runtime filtering, e.g. PartitionPruning
> and PlanDynamicPruningFilters
>
> On Thu, Mar 27, 2025 at 6:53 PM
The file source in Spark has not been migrated to DS v2 yet and uses
dedicated catalyst rules to do runtime filtering, e.g. PartitionPruning
and PlanDynamicPruningFilters
On Thu, Mar 27, 2025 at 6:53 PM Asif Shahid wrote:
> Hi Experts,
> Could you please allow me to pick your brain on the follo
Back in the very early days of Spark (before it was even an Apache
Incubator project), Maven was clearly a more mature, capable and
stable tool suite for building, testing and publishing JVM code, even
Scala code, so some of the earliest commercial adopters of Spark
relied upon Maven. It made sense
Hi Experts,
Could you please allow me to pick your brain on the following:
For Hive Tables ( managed), the scan operator is FileSourceScanExec.
Is there any particular reason why its underlying HadoopFSRelations'
field, FileFormat does not implement an interface like
SupportsRuntimeFiltering ?
Li
+1
在 2025-03-26 14:45:09，"Chao Sun" 写道：
+1
On Tue, Mar 25, 2025 at 10:22 PM Ángel Álvarez Pascua
wrote:
I meant ... a data validation API would be great, but why in the DSv2? isn't
data validation something more general? do we have to use DSv2 to have our data
validated?
El mié, 26
18 matches
Mail list logo
