Re: [PR] [experiment] Deterministic Hash Function for Upsert Type-4 UUID Primary Keys [pinot]

2024-03-01 Thread via GitHub
ankitsultana commented on code in PR #12538: URL: https://github.com/apache/pinot/pull/12538#discussion_r1509906862 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/HashUtils.java: ## @@ -44,6 +77,8 @@ public static Object hashPrimaryKey(PrimaryKey prima

Re: [PR] [experiment] Deterministic Hash Function for Upsert Type-4 UUID Primary Keys [pinot]

2024-03-01 Thread via GitHub
ankitsultana commented on code in PR #12538: URL: https://github.com/apache/pinot/pull/12538#discussion_r1509906166 ## pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/HashUtils.java: ## @@ -36,6 +40,35 @@ public static byte[] hashMD5(byte[] bytes) { r

[PR] [experiment] Add Deterministic Hash Function for Upserts for Encoding Type-4 UUIDs [pinot]

2024-03-01 Thread via GitHub
ankitsultana opened a new pull request, #12538: URL: https://github.com/apache/pinot/pull/12538 This can help reduce memory usage for scenarios where all columns of the Primary Key are type-4 UUIDs. This can also handle scenarios where some values are not valid type-4 UUIDs, by simply switc

[PR] DRAFT: Adding SHOW TABLES/DESCRIBE table support [pinot]

2024-03-01 Thread via GitHub
aadilkhalifa opened a new pull request, #12537: URL: https://github.com/apache/pinot/pull/12537 PR to add support for "SHOW TABLES" and "DESCRIBE table" queries. Current status: - [x] SHOW TABLES - [ ] DESCRIBE table -- This is an automated message from the Apache Git Service.

Re: [PR] mockito 5.11.0 [pinot]

2024-03-01 Thread via GitHub
codecov-commenter commented on PR #12536: URL: https://github.com/apache/pinot/pull/12536#issuecomment-1974202097 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12536?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

[PR] mockito 5.11.0 [pinot]

2024-03-01 Thread via GitHub
sullis opened a new pull request, #12536: URL: https://github.com/apache/pinot/pull/12536 mockito 5.11.0 is the latest version of mockito. ## mockito-inline mockito-inline no longer exists see: https://github.com/mockito/mockito/issues/2877 -- This is an automated

(pinot) branch master updated: Allow passing custom record reader to be inited/closed in SegmentProcessorFramework (#12529)

2024-03-01 Thread snlee
This is an automated email from the ASF dual-hosted git repository. snlee pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 9a8fa79199 Allow passing custom record reader to b

Re: [PR] Allow passing custom record reader to be inited/closed in SegmentProcessorFramework [pinot]

2024-03-01 Thread via GitHub
snleee merged PR #12529: URL: https://github.com/apache/pinot/pull/12529 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apach

Re: [PR] adding support of mega_bytes when configuring broker response size [pinot]

2024-03-01 Thread via GitHub
Jackie-Jiang commented on code in PR #12510: URL: https://github.com/apache/pinot/pull/12510#discussion_r1509748576 ## pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java: ## @@ -1773,15 +1774,17 @@ private void setMaxServerResponseSiz

[I] Segment Compaction tasks fail when snapshot file doesn't exist on lagging server [pinot]

2024-03-01 Thread via GitHub
robertzych opened a new issue, #12535: URL: https://github.com/apache/pinot/issues/12535 When the compaction is configured with a `bufferTimePeriod` value `< (2 * segmentCommitInterval)` the tasks fail due to the following `NotFoundException`: ![image](https://github.com/apache/pinot

(pinot) branch master updated: Prepare for next development iteration (#12530)

2024-03-01 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 60d23ca97b Prepare for next development iteration

Re: [PR] Prepare for next development iteration [pinot]

2024-03-01 Thread via GitHub
Jackie-Jiang merged PR #12530: URL: https://github.com/apache/pinot/pull/12530 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

(pinot) branch master updated: reformat ClusterTest.java (#12531)

2024-03-01 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 7b71babb73 reformat ClusterTest.java (#12531) 7b7

Re: [PR] reformat ClusterTest.java [pinot]

2024-03-01 Thread via GitHub
Jackie-Jiang merged PR #12531: URL: https://github.com/apache/pinot/pull/12531 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [I] [Flaky test] CaseTransformFunctionTest. testCaseTransformFunctionWithIntResults() [pinot]

2024-03-01 Thread via GitHub
Jackie-Jiang commented on issue #12367: URL: https://github.com/apache/pinot/issues/12367#issuecomment-1974115082 Yes. When we use float as literal (e.g. `WHERE col = 0.1`), it is actually parsed as double. If the `col` is of type `FLOAT`, it might not match this filter because `(double) 0.

(pinot) branch master updated: use Temurin instead of AdoptOpenJdk (#12533)

2024-03-01 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 954f5d7e51 use Temurin instead of AdoptOpenJdk (#

Re: [PR] use Temurin instead of AdoptOpenJdk [pinot]

2024-03-01 Thread via GitHub
Jackie-Jiang merged PR #12533: URL: https://github.com/apache/pinot/pull/12533 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [I] Consuming segment flush rows significantly reduced after force commit [pinot]

2024-03-01 Thread via GitHub
Jackie-Jiang closed issue #12509: Consuming segment flush rows significantly reduced after force commit URL: https://github.com/apache/pinot/issues/12509 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Egalpin/skip indexes minor changes [pinot]

2024-03-01 Thread via GitHub
Jackie-Jiang commented on PR #12514: URL: https://github.com/apache/pinot/pull/12514#issuecomment-1974098078 Thank you for addressing the comments! Can you help contribute this new query option to the pinot doc? -- This is an automated message from the Apache Git Service. To respond to th

(pinot) branch master updated: Egalpin/skip indexes minor changes (#12514)

2024-03-01 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new f51c34fcbd Egalpin/skip indexes minor changes (#1

Re: [PR] Egalpin/skip indexes minor changes [pinot]

2024-03-01 Thread via GitHub
Jackie-Jiang merged PR #12514: URL: https://github.com/apache/pinot/pull/12514 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

(pinot) branch master updated: maven: no transfer progress (#12528)

2024-03-01 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 1bfc6ac4dd maven: no transfer progress (#12528) 1

Re: [PR] maven no transfer progress [pinot]

2024-03-01 Thread via GitHub
Jackie-Jiang merged PR #12528: URL: https://github.com/apache/pinot/pull/12528 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot

Re: [I] `VarByteChunkForwardIndexReaderV4` fails to decompress some chunks [pinot]

2024-03-01 Thread via GitHub
itschrispeck commented on issue #12534: URL: https://github.com/apache/pinot/issues/12534#issuecomment-1974066176 @richardstartin for your thoughts -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[I] `VarByteChunkForwardIndexReaderV4` fails to decompress some chunks [pinot]

2024-03-01 Thread via GitHub
itschrispeck opened a new issue, #12534: URL: https://github.com/apache/pinot/issues/12534 Within a segment, certain chunks cannot be decompressed. I can time bound a query within a single segment, and the query will succeed. e.g. for a segment with data [chunk1][chunk2][chunk3]:

Re: [PR] Metrics for Realtime Rows Fetched and Stream Consumer Create Exceptions [pinot]

2024-03-01 Thread via GitHub
zhtaoxiang commented on code in PR #12522: URL: https://github.com/apache/pinot/pull/12522#discussion_r1509633595 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java: ## @@ -1661,22 +1666,35 @@ private void makeStreamConsumer(S

Re: [PR] Enter segment preloading at partition level [pinot]

2024-03-01 Thread via GitHub
klsince commented on code in PR #12451: URL: https://github.com/apache/pinot/pull/12451#discussion_r1509628827 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java: ## @@ -359,6 +357,12 @@ public boolean isPartialUpsertEnabled() {

svn commit: r67657 - /dev/pinot/apache-pinot-1.1.0-rc0/

2024-03-01 Thread vvivekiyer
Author: vvivekiyer Date: Fri Mar 1 20:43:54 2024 New Revision: 67657 Log: Update apache-pinot-1.1.0-rc0 Added: dev/pinot/apache-pinot-1.1.0-rc0/ dev/pinot/apache-pinot-1.1.0-rc0/apache-pinot-1.1.0-bin.tar.gz (with props) dev/pinot/apache-pinot-1.1.0-rc0/apache-pinot-1.1.0-bin.tar.

Re: [I] Consuming segment flush rows significantly reduced after force commit [pinot]

2024-03-01 Thread via GitHub
sajjad-moradi commented on issue #12509: URL: https://github.com/apache/pinot/issues/12509#issuecomment-1973795958 I believe as part of this [PR](https://github.com/apache/pinot/pull/12188/files), this issue was fixed. Look here: https://github.com/apache/pinot/blob/9cfce82385dfcd11ee

Re: [PR] Custom configuration property reader for segment metadata files [pinot]

2024-03-01 Thread via GitHub
abhioncbr commented on code in PR #12440: URL: https://github.com/apache/pinot/pull/12440#discussion_r1509441264 ## pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java: ## @@ -287,4 +321,35 @@ private static PropertiesConfiguration createPropertiesCo

Re: [PR] use Temurin instead of AdoptOpenJdk [pinot]

2024-03-01 Thread via GitHub
codecov-commenter commented on PR #12533: URL: https://github.com/apache/pinot/pull/12533#issuecomment-1973621247 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12533?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [I] [Flaky test] CaseTransformFunctionTest. testCaseTransformFunctionWithIntResults() [pinot]

2024-03-01 Thread via GitHub
aditya0811 commented on issue #12367: URL: https://github.com/apache/pinot/issues/12367#issuecomment-1973584832 Hey Jackie, when you say expected result is wrong, you mean the comparison in the case block in L721 in CaseTransformFunctionTest.java? ``` case FLOAT_SV_COLUMN:

Re: [PR] Add the support to extract json array elements from json index for the transform function jsonExtractIndex [pinot]

2024-03-01 Thread via GitHub
wirybeaver commented on code in PR #12466: URL: https://github.com/apache/pinot/pull/12466#discussion_r1509270121 ## pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/BaseTransformFunctionTest.java: ## @@ -158,6 +159,11 @@ public void setUp()

[PR] use Temurin instead of AdoptOpenJdk [pinot]

2024-03-01 Thread via GitHub
sullis opened a new pull request, #12533: URL: https://github.com/apache/pinot/pull/12533 - AdoptOpenJDK is obsolete - use Temurin JDK -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] Json extract index mv [pinot]

2024-03-01 Thread via GitHub
codecov-commenter commented on PR #12532: URL: https://github.com/apache/pinot/pull/12532#issuecomment-1973403465 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12532?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) R

Re: [PR] Allow passing custom record reader to be inited/closed in SegmentProcessorFramework [pinot]

2024-03-01 Thread via GitHub
swaminathanmanish commented on PR #12529: URL: https://github.com/apache/pinot/pull/12529#issuecomment-1973361126 > LGTM > > Is the agreement is to not use the context? Yes it looks like the purpose of a RecordTransformer is to transform a GenericRecord. independent of file/rec

[PR] Json extract index mv [pinot]

2024-03-01 Thread via GitHub
saurabhd336 opened a new pull request, #12532: URL: https://github.com/apache/pinot/pull/12532 inspired by JSONEXTRACTINDEX, allows extracting contexual json array elements matching a given json match predicate, using the json index eg: For a dataset like ``` {"jsonDoc": [{"n

Re: [PR] Throw exception when schema name doesn't match table name during table creation [pinot]

2024-03-01 Thread via GitHub
estebanz01 commented on PR #11591: URL: https://github.com/apache/pinot/pull/11591#issuecomment-1973253789 Hola 👋 ! sorry to comment on a merged PR, but I don't see in the release docs for 1.1.0-rc a reference to this change, which is a breaking change for our infrastructure. Is it going to

Re: [PR] [feature] use async methods to read from Pulsar topic [pinot]

2024-03-01 Thread via GitHub
enzo-dechaene commented on PR #12061: URL: https://github.com/apache/pinot/pull/12061#issuecomment-1973221062 Hii @KKcorps, can we merge this PR in the 1.1.0 release ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [I] 1.0.1 release timeline [pinot]

2024-03-01 Thread via GitHub
hpvd commented on issue #12428: URL: https://github.com/apache/pinot/issues/12428#issuecomment-1973159953 so maybe this issue can already be closed? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [I] 1.0.1 release timeline [pinot]

2024-03-01 Thread via GitHub
hpvd commented on issue #12428: URL: https://github.com/apache/pinot/issues/12428#issuecomment-1973142317 1.1.0-rc was released: https://github.com/apache/pinot/releases/tag/release-1.1.0-rc0 -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [I] 1.0.1 release timeline [pinot]

2024-03-01 Thread via GitHub
hpvd commented on issue #12428: URL: https://github.com/apache/pinot/issues/12428#issuecomment-1973139915 has moved to 1.1.0-rc https://github.com/apache/pinot/tree/release-1.1.0-rc -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Allow passing database context through `database` http header [pinot]

2024-03-01 Thread via GitHub
shounakmk219 commented on code in PR #12417: URL: https://github.com/apache/pinot/pull/12417#discussion_r1508836250 ## pinot-common/src/main/java/org/apache/pinot/common/utils/DatabaseUtils.java: ## @@ -0,0 +1,91 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Allow passing database context through `database` http header [pinot]

2024-03-01 Thread via GitHub
shounakmk219 commented on code in PR #12417: URL: https://github.com/apache/pinot/pull/12417#discussion_r1508834724 ## pinot-broker/src/main/java/org/apache/pinot/broker/api/resources/PinotBrokerRouting.java: ## @@ -61,8 +69,23 @@ public class PinotBrokerRouting { @ApiRes

Re: [PR] Allow passing database context through `database` http header [pinot]

2024-03-01 Thread via GitHub
shounakmk219 commented on code in PR #12417: URL: https://github.com/apache/pinot/pull/12417#discussion_r1508835134 ## pinot-common/src/main/java/org/apache/pinot/common/utils/DatabaseUtils.java: ## @@ -0,0 +1,91 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Add a post-validator visitor that verifies there are no cast to bytes [pinot]

2024-03-01 Thread via GitHub
gortiz commented on code in PR #12475: URL: https://github.com/apache/pinot/pull/12475#discussion_r1508720811 ## pinot-query-planner/src/main/java/org/apache/pinot/query/validate/BytesCastVisitor.java: ## @@ -0,0 +1,73 @@ +/** + * Licensed to the Apache Software Foundation (ASF)

Re: [I] String to bytes autocasting [pinot]

2024-03-01 Thread via GitHub
gortiz commented on issue #12457: URL: https://github.com/apache/pinot/issues/12457#issuecomment-1972834050 As a partial solution, @Jackie-Jiang and I are planning to merge https://github.com/apache/pinot/pull/12475, which forbids implicit or explicit casts from anything to Bytes/BINARY onl

Re: [I] Configurable heap size for query and realtime segment persistence. [pinot]

2024-03-01 Thread via GitHub
Lvnszn commented on issue #12503: URL: https://github.com/apache/pinot/issues/12503#issuecomment-1972735574 > Is the problem resolved automatically? We'll need more context in order to debug it Add a server and rebalance table to a new server then the problem was resolved -- This

(pinot) annotated tag release-1.1.0-rc0 updated (49ee95a75b -> 37bd7be690)

2024-03-01 Thread vvivekiyer
This is an automated email from the ASF dual-hosted git repository. vvivekiyer pushed a change to annotated tag release-1.1.0-rc0 in repository https://gitbox.apache.org/repos/asf/pinot.git *** WARNING: tag release-1.1.0-rc0 was modified! *** from 49ee95a75b (commit) to 37bd7be690 (ta

(pinot) branch release-1.1.0-rc updated: [maven-release-plugin] prepare for next development iteration

2024-03-01 Thread vvivekiyer
This is an automated email from the ASF dual-hosted git repository. vvivekiyer pushed a commit to branch release-1.1.0-rc in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/release-1.1.0-rc by this push: new 64f9d792f1 [maven-release

(pinot) branch release-1.1.0-rc updated: [maven-release-plugin] prepare release release-1.1.0-rc0

2024-03-01 Thread vvivekiyer
This is an automated email from the ASF dual-hosted git repository. vvivekiyer pushed a commit to branch release-1.1.0-rc in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/release-1.1.0-rc by this push: new 49ee95a75b [maven-release