Re: Status of Pulsar 2.9.0 and starting 2.9.1
Hi Enrico, Thanks for your great effort on the 2.9.0 release. Circling back to see if there is any progress of the 2.9.0 website updates [1]. Currently, the 2.9.0 doc is not available on the website and the 2.9.0 doc set has not been generated yet, any updates? Thanks [1] https://github.com/apache/pulsar/wiki/Release-process#16-update-the-site On Mon, Dec 13, 2021 at 3:32 PM Enrico Olivelli wrote: > Dave, > You are correct. > Pulsar 2.9.0 has already been released and also some people already started > to report issues. > The docker images have been deployed and we cannot change them. > > I am finishing the release process for 2.9.0 with the website updates. > > I am preparing 2.9.1. > > I propose to just skip the announcement for 2.9.0. > > If we are quick during the VOTE we can close this story within the end of > the week > > Enrico > > Il Lun 13 Dic 2021, 06:34 Dave Fisher ha scritto: > > > (1) we have published 2.9.0 at > > https://downloads.apache.org/pulsar/pulsar-2.9.0/ > > > > (2) we have published 2.9.0 artifacts through maven central. They don’t > > let anyone republish versions. > > > > There are no do overs on versions. We simply cannot redo 2.9.0 at this > > moment. > > > > All the best, > > Dave > > > > Sent from my iPhone > > > > > On Dec 12, 2021, at 8:49 PM, Sijie Guo wrote: > > > > > > My take is - if we haven't announced 2.9, I would suggest just redoing > > the > > > 2.9.0 release. > > > > > > - Sijie > > > > > >> On Sun, Dec 12, 2021 at 6:35 PM Hang Chen > wrote: > > >> > > >> I am a little confused about why we should skip 2.9.0 and not continue > > >> to release 2.9.0 by including the critical bug fixes. In fact, the > > >> Pulsar 2.9.0 release is not yet completed. > > >> > > >> For users, they will worry about whether the Pulsar release process is > > >> standardized if we skip 2.9.0. They will also worry about the release > > >> quality of Apache Pulsar if we have found the critical bugs before it > > >> is released but not included it into the release version. For Pulsar > > >> 2.9.0, it couldn't be deployed into the production environment due to > > >> the critical bug https://github.com/apache/pulsar/pull/12993 > > >> > > >> Regards, > > >> Hang > > >> > > >> Dave Fisher 于2021年12月13日周一 09:40写道： > > >>> > > >>> It can be the case that releases are not announced. For example with > > >> Tomcat a version which fails to pass the vote is skipped. > > >>> > > >>> Let’s not announce 2.9.0 and go on to 2.9.1. > > >>> > > >>> Maybe there’s some website fixes to hide 2.9.0. > > >>> > > >>> > > >>> Sent from my iPhone > > >>> > > On Dec 12, 2021, at 5:28 PM, PengHui Li wrote: > > > > Another point is we have not announced the 2.9.0 release yet. > > > > This will make users feel confused that a new release from the > Pulsar > > community with the > > serious problem(log4j bug) but after the log4j has fixed the issue > and > > provided the new release. > > > > I think we'd better contain the fix in 2.9.0 and 2.9.0 also has a > > >> critical > > bug https://github.com/apache/pulsar/pull/12993 > > which will lead the topic stop to provide services for more than > 5min. > > >> It > > looks like, hey, we have a new release here but > > it has critical security issues and known serious bugs which will > > >> seriously > > affect the core features. > > > > From the perspective of release, yes, the release vote has closed. > But > > >> I > > believe that users will not care about this matter, > > they only care about the quality of the products we provided. > > > > I would like to hear your views. > > > > Regards, > > Penghui > > > > > > > > > On Sun, Dec 12, 2021 at 6:26 PM Enrico Olivelli < > eolive...@gmail.com > > > > > >> wrote: > > > > > > I am starting 2.9.1 on Monday > > > > > > Enrico > > > > > > Il Dom 12 Dic 2021, 02:19 陳智弘 ha scritto: > > > > > >> Totally agree > > >> > > >>> PengHui Li 於 2021年12月12日 週日 08:28 寫道： > > >> > > >>> +1 > > >>> > > >>> Penghui > > >>> > > >>> Matteo Merli 于2021年12月11日 周六15:28写道： > > >>> > > At this point, if 2.9.0 is non stable, I think we should > > >> fast-forward > > to 2.9.1 which will include security fix. Though, we should > start > > 2.9.1 right now. > > > > > > -- > > Matteo Merli > > > > > > On Fri, Dec 10, 2021 at 11:23 PM Michael Marshall < > > >> mmarsh...@apache.org> > > wrote: > > > > > > +1 - thanks Enrico. > > > > > > - Michael > > > > > > On Sat, Dec 11, 2021 at 1:11 AM Lari Hotari < > lhot...@apache.org> > > >>> wrote: > > >> > > >> +1 > > >> > > >> la 11. jouluk. 2021 klo 9.07 Enrico Olivelli < > > > eolive...@gmail.com> > > >
[GitHub] [pulsar-manager] sourabhaggrawal commented on a change in pull request #434: SAML Integration for Login
sourabhaggrawal commented on a change in pull request #434:
URL: https://github.com/apache/pulsar-manager/pull/434#discussion_r771301885
##
File path:
src/main/java/org/apache/pulsar/manager/interceptor/WebAppConfigurer.java
##
@@ -51,6 +51,7 @@ public void addInterceptors(InterceptorRegistry registry) {
.excludePathPatterns("/doc.html")
// BKVM
.excludePathPatterns("/bkvm")
+.excludePathPatterns("/pulsar-manager/saml/**")
Review comment:
our endpoint is /saml/sso @shiv4289
here /saml is mapped at controller level and ** to exclude all method in
that controller from authentication.
This makes me rethink to change the excludePathPatterns as
/pulsar-manager/saml/sso to make only one method accessible without
authentication.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [pulsar-manager] sourabhaggrawal commented on a change in pull request #434: SAML Integration for Login
sourabhaggrawal commented on a change in pull request #434:
URL: https://github.com/apache/pulsar-manager/pull/434#discussion_r771312452
##
File path: src/main/java/org/apache/pulsar/manager/controller/SSOController.java
##
@@ -0,0 +1,131 @@
+/**
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.pulsar.manager.controller;
+
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import java.net.URI;
+import java.util.Map;
+import java.util.Optional;
+import javax.annotation.Resource;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.http.util.Asserts;
+import org.apache.pulsar.manager.entity.RoleBindingEntity;
+import org.apache.pulsar.manager.entity.RoleBindingRepository;
+import org.apache.pulsar.manager.entity.RolesRepository;
+import org.apache.pulsar.manager.entity.UserInfoEntity;
+import org.apache.pulsar.manager.entity.UsersRepository;
+import org.apache.pulsar.manager.service.JwtService;
+import org.apache.pulsar.manager.utils.SAMLParser;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.beans.factory.annotation.Value;
+import org.springframework.validation.annotation.Validated;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RequestMethod;
+import org.springframework.web.bind.annotation.ResponseBody;
+import org.springframework.web.bind.annotation.RestController;
+
+@Slf4j
+@RestController
+@RequestMapping(value = "/pulsar-manager/saml")
+@Api(description = "Calling the request below this class does not require
authentication because " +
+"the user has not logged in yet.")
+@Validated
+public class SSOController {
+
+
+@Autowired
+private UsersRepository usersRepository;
+
+@Autowired
+private JwtService jwtService;
+
+@Resource
+private RolesRepository rolesRepository;
+
+@Resource
+private RoleBindingRepository roleBindingRepository;
+
+@Value("${sso.redirect.host}")
+private String redirectHost;
+
+@Value("${sso.redirect.port}")
+private int redirectPort;
+
+@Value("${sso.redirect.scheme}")
+private String redirectUrlScheme;
+
+@Value("${sso.certificate}")
+private String certificate;
+
+@Value("${sso.whitelisted.domain}")
+private String whiteListedDomain;
+
+@ApiOperation(value = "Verify SAML Response")
+@ApiResponses({
+@ApiResponse(code = 200, message = "ok"),
+@ApiResponse(code = 404, message = "Not found"),
+@ApiResponse(code = 500, message = "Internal server error")
+})
+@RequestMapping(value = "/sso", method = RequestMethod.POST)
+@ResponseBody
+public void loginByIDP(HttpServletRequest request, HttpServletResponse
response) throws Exception {
+String samlResponse = request.getParameter("SAMLResponse");
+SAMLParser samlParser = new SAMLParser(certificate);
+Map userDetails =
samlParser.getUserDetailsFromSAMLResponse(samlResponse);
+
Asserts.check(StringUtils.contains(userDetails.get("email"),whiteListedDomain),"Login
with domain not supported");
Review comment:
I am wondering what we will achieve with the email format validation ,
the email is extracted from SAML Response which is served through IDP directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [VOTE] Pulsar Node.js Client Release 1.5.0 Candidate 1
+1 (binding) * check the license headers * build the source and run producer/consumer * verify checksum and signatures Thanks, Nozomi 2021年12月16日(木) 9:29 Hiroyuki Sakai : > +1 (binding) > > * check the license headers > * build the source and run producer/consumer > * verify checksum and signatures > > == > Hiroyuki Sakai > Yahoo Japan Corp. > E-mail: hsa...@yahoo-corp.jp > > > From: Masahiro Sakamoto > Sent: Tuesday, December 14, 2021 17:55 > To: dev@pulsar.apache.org > Subject: [VOTE] Pulsar Node.js Client Release 1.5.0 Candidate 1 > > Hi everyone, > Please review and vote on the release candidate #1 for the version 1.5.0, > as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > This is the first release candidate for Apache Pulsar Node.js client, > version 1.5.0. > > It fixes the following issues: > > https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar-client-node%2Fmilestone%2F7%3Fclosed%3D1&data=04%7C01%7Chsakai%40yahoo-corp.jp%7C228460548f654cc2358e08d9bedf8c0a%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637750690187071958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=l5erf4AyIdnXqXTP%2B%2FEOyePQD%2FtTfxeTxnd7yLA%2FxSo%3D&reserved=0 > > Please download the source files and review this release candidate: > - Review release notes > - Download the source package (verify shasum and asc) and follow the > README.md to build and run the Pulsar Node.js client. > > The vote will be open for at least 72 hours. It is adopted by majority > approval, with at least 3 PMC affirmative votes. > > Source files: > > https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2Fpulsar-client-node%2Fpulsar-client-node-1.5.0-candidate-1%2F&data=04%7C01%7Chsakai%40yahoo-corp.jp%7C228460548f654cc2358e08d9bedf8c0a%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637750690187071958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=WAX42CoKhaMbUv3nmyK%2FSGCnm5LlRiGBW1XStJ9mzZI%3D&reserved=0 > > Pulsar's KEYS file containing PGP keys we use to sign the release: > > https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fpulsar%2FKEYS&data=04%7C01%7Chsakai%40yahoo-corp.jp%7C228460548f654cc2358e08d9bedf8c0a%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637750690187071958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=1NNW8t4M7NsgMo6FHK9%2FydD2FwObWeG3IaXWSm%2FU9jo%3D&reserved=0 > > SHA-512 checksum: > 5c05c5368fb822165dde5c3aa758d600eca4452e1cc160f9b53311e27436e4cd52fbc55ca6382bb56019e4b0a3a6a409f73eec7c19844fb60715e41348fd8c09 > pulsar-client-node-1.5.0.tar.gz > > The tag to be voted upon: > v1.5.0-rc.1 > > https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fpulsar-client-node%2Freleases%2Ftag%2Fv1.5.0-rc.1&data=04%7C01%7Chsakai%40yahoo-corp.jp%7C228460548f654cc2358e08d9bedf8c0a%7Ca208d369cd4e4f87b11998eaf31df2c3%7C1%7C0%7C637750690187071958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Hs8Z%2BBw%2BmXnO5RYc2AwFgPnrwC%2BWVyS4VKJ4IpArTq8%3D&reserved=0 > > Masahiro Sakamoto > Yahoo Japan Corp. > E-mail: massa...@yahoo-corp.jp >
Re: [VOTE] Apache Pulsar 2.9.1 candidate 2
Peng Hui, Il giorno ven 17 dic 2021 alle ore 08:09 PengHui Li ha scritto: > Checked: > > - Build from the src > - Check signatures > - Follow the validation process > > But when I try to verify PulsarSQL, got following exceptions: > > ``` > 2021-12-17T14:58:18.958+0800 ERROR remote-task-callback-3 > io.prestosql.execution.StageStateMachine Stage > 20211217_065818_1_cahiv.1 failed > com.google.common.util.concurrent.UncheckedExecutionException: > java.nio.BufferUnderflowException > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2051) > at com.google.common.cache.LocalCache.get(LocalCache.java:3951) > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) > at > > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935) > at > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.getSchemaByVersion(PulsarSqlSchemaInfoProvider.java:76) > at > > org.apache.pulsar.sql.presto.PulsarRecordCursor.advanceNextPosition(PulsarRecordCursor.java:485) > at > > io.prestosql.spi.connector.RecordPageSource.getNextPage(RecordPageSource.java:90) > at > > io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:302) > at io.prestosql.operator.Driver.processInternal(Driver.java:379) > at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283) > at io.prestosql.operator.Driver.tryWithLock(Driver.java:675) > at io.prestosql.operator.Driver.processFor(Driver.java:276) > at > > io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075) > at > > io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163) > at > > io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484) > at > io.prestosql.$gen.Presto_332__testversion20211217_065757_2.run(Unknown > Source) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.nio.BufferUnderflowException > at java.nio.Buffer.nextGetIndex(Buffer.java:532) > at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:417) > at > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.loadSchema(PulsarSqlSchemaInfoProvider.java:106) > at > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.access$000(PulsarSqlSchemaInfoProvider.java:49) > at > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider$1.load(PulsarSqlSchemaInfoProvider.java:61) > at > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider$1.load(PulsarSqlSchemaInfoProvider.java:58) > at > > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529) > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278) > at > > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155) > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045) > ... 18 more > ``` > > An issue can be found here https://github.com/apache/pulsar/issues/12284, > This doesn't look like a regression introduced in 2.9.1 As this is a security related release I would continue the release process. What do you think ? Enrico > my test steps are very simple: > > 1. Start presto worker, `bin/pulsar sql-worker run` > 2. Produce some messages, `bin/pulsar-client produce -m "hello" -n 10 > test_wordcount_src` > 3. Query the data from the topic, `select * from > pulsar."public/default"."test_wordcount_src";` > > Not able to query the produced data and get errors in the Pulsar SQL > worker. > > Penghui > > On Fri, Dec 17, 2021 at 5:33 AM Matteo Merli wrote: > > > +1 > > > > Checked: > > * Signatures > > * Bin distribution: > > - NOTICE, README, LICENSE > > - Start standalone service and producer/consumer test > > * Src distribution: > > - NOTICE, README, LICENSE > > - Compile and unit tests > > - Start standalone service > > * Checked staging maven repository artifacts > > * Checked docker images > > > > > > Matteo > > > > -- > > Matteo Merli > > > > > > > > > > On Thu, Dec 16, 2021 at 12:53 PM Enrico Olivelli > > wrote: > > > > > > I have pushed the docker images to my personal dockehub account > > > > > > eolivelli/pulsar:2.9.1rc2 > > > eolivelli/pulsar-all:2.9.1rc2 > > > > > > Enrico > > > > > > Il Gio 16 Dic 2021, 15:57 Nicolò Boschi ha > > scritto: > > > > > > > +1 (non binding) > > > > > > > > Checks: > > > > - Checksum and signatures > > > > - Apache Rat check passes > > > > - OWASP check passes (I created this PR for fix a false positive > > > > https://github.com/apache/pulsar/pull/13364) > > > > - Compile from source w JDK11 > > > > - Build docker image from source > > > > - Run Pulsar standalone and produce-consume from CLI > > > > - verified the presence of Log4j 2.16.0 jar in docker and tarball > > > > > > > > Il giorno gio 16 d
Re: Status of Pulsar 2.9.0 and starting 2.9.1
Il giorno ven 17 dic 2021 alle ore 09:45 Yu ha scritto: > Hi Enrico, > > Thanks for your great effort on the 2.9.0 release. > > Circling back to see if there is any progress of the 2.9.0 website updates > [1]. > > Currently, the 2.9.0 doc is not available on the website and the 2.9.0 doc > set has not been generated yet, any updates? Thanks > I will do it together with 2.9.1 because 2.9.0 is broken, so it is better to not do much buzz around it Enrico > > [1] > https://github.com/apache/pulsar/wiki/Release-process#16-update-the-site > > > On Mon, Dec 13, 2021 at 3:32 PM Enrico Olivelli > wrote: > > > Dave, > > You are correct. > > Pulsar 2.9.0 has already been released and also some people already > started > > to report issues. > > The docker images have been deployed and we cannot change them. > > > > I am finishing the release process for 2.9.0 with the website updates. > > > > I am preparing 2.9.1. > > > > I propose to just skip the announcement for 2.9.0. > > > > If we are quick during the VOTE we can close this story within the end of > > the week > > > > Enrico > > > > Il Lun 13 Dic 2021, 06:34 Dave Fisher ha > scritto: > > > > > (1) we have published 2.9.0 at > > > https://downloads.apache.org/pulsar/pulsar-2.9.0/ > > > > > > (2) we have published 2.9.0 artifacts through maven central. They don’t > > > let anyone republish versions. > > > > > > There are no do overs on versions. We simply cannot redo 2.9.0 at this > > > moment. > > > > > > All the best, > > > Dave > > > > > > Sent from my iPhone > > > > > > > On Dec 12, 2021, at 8:49 PM, Sijie Guo wrote: > > > > > > > > My take is - if we haven't announced 2.9, I would suggest just > redoing > > > the > > > > 2.9.0 release. > > > > > > > > - Sijie > > > > > > > >> On Sun, Dec 12, 2021 at 6:35 PM Hang Chen > > wrote: > > > >> > > > >> I am a little confused about why we should skip 2.9.0 and not > continue > > > >> to release 2.9.0 by including the critical bug fixes. In fact, the > > > >> Pulsar 2.9.0 release is not yet completed. > > > >> > > > >> For users, they will worry about whether the Pulsar release process > is > > > >> standardized if we skip 2.9.0. They will also worry about the > release > > > >> quality of Apache Pulsar if we have found the critical bugs before > it > > > >> is released but not included it into the release version. For Pulsar > > > >> 2.9.0, it couldn't be deployed into the production environment due > to > > > >> the critical bug https://github.com/apache/pulsar/pull/12993 > > > >> > > > >> Regards, > > > >> Hang > > > >> > > > >> Dave Fisher 于2021年12月13日周一 09:40写道： > > > >>> > > > >>> It can be the case that releases are not announced. For example > with > > > >> Tomcat a version which fails to pass the vote is skipped. > > > >>> > > > >>> Let’s not announce 2.9.0 and go on to 2.9.1. > > > >>> > > > >>> Maybe there’s some website fixes to hide 2.9.0. > > > >>> > > > >>> > > > >>> Sent from my iPhone > > > >>> > > > On Dec 12, 2021, at 5:28 PM, PengHui Li > wrote: > > > > > > Another point is we have not announced the 2.9.0 release yet. > > > > > > This will make users feel confused that a new release from the > > Pulsar > > > community with the > > > serious problem(log4j bug) but after the log4j has fixed the issue > > and > > > provided the new release. > > > > > > I think we'd better contain the fix in 2.9.0 and 2.9.0 also has a > > > >> critical > > > bug https://github.com/apache/pulsar/pull/12993 > > > which will lead the topic stop to provide services for more than > > 5min. > > > >> It > > > looks like, hey, we have a new release here but > > > it has critical security issues and known serious bugs which will > > > >> seriously > > > affect the core features. > > > > > > From the perspective of release, yes, the release vote has closed. > > But > > > >> I > > > believe that users will not care about this matter, > > > they only care about the quality of the products we provided. > > > > > > I would like to hear your views. > > > > > > Regards, > > > Penghui > > > > > > > > > > > > > On Sun, Dec 12, 2021 at 6:26 PM Enrico Olivelli < > > eolive...@gmail.com > > > > > > > >> wrote: > > > > > > > > I am starting 2.9.1 on Monday > > > > > > > > Enrico > > > > > > > > Il Dom 12 Dic 2021, 02:19 陳智弘 ha > scritto: > > > > > > > >> Totally agree > > > >> > > > >>> PengHui Li 於 2021年12月12日 週日 08:28 寫道： > > > >> > > > >>> +1 > > > >>> > > > >>> Penghui > > > >>> > > > >>> Matteo Merli 于2021年12月11日 周六15:28写道： > > > >>> > > > At this point, if 2.9.0 is non stable, I think we should > > > >> fast-forward > > > to 2.9.1 which will include security fix. Though, we should > > start > > > 2.9.1 right now. > > > > > > > > > -- > > > Matteo Merli >
Re: [VOTE] Apache Pulsar 2.9.1 candidate 2
Hi Enrico, I'm ok, it only happens when the message is without a schema version. So I'm not giving -1. Thanks, Penghui On Fri, Dec 17, 2021 at 7:33 PM Enrico Olivelli wrote: > Peng Hui, > > Il giorno ven 17 dic 2021 alle ore 08:09 PengHui Li > ha > scritto: > > > Checked: > > > > - Build from the src > > - Check signatures > > - Follow the validation process > > > > But when I try to verify PulsarSQL, got following exceptions: > > > > ``` > > 2021-12-17T14:58:18.958+0800 ERROR remote-task-callback-3 > > io.prestosql.execution.StageStateMachine Stage > > 20211217_065818_1_cahiv.1 failed > > com.google.common.util.concurrent.UncheckedExecutionException: > > java.nio.BufferUnderflowException > > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2051) > > at com.google.common.cache.LocalCache.get(LocalCache.java:3951) > > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) > > at > > > > > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935) > > at > > > > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.getSchemaByVersion(PulsarSqlSchemaInfoProvider.java:76) > > at > > > > > org.apache.pulsar.sql.presto.PulsarRecordCursor.advanceNextPosition(PulsarRecordCursor.java:485) > > at > > > > > io.prestosql.spi.connector.RecordPageSource.getNextPage(RecordPageSource.java:90) > > at > > > > > io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:302) > > at io.prestosql.operator.Driver.processInternal(Driver.java:379) > > at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283) > > at io.prestosql.operator.Driver.tryWithLock(Driver.java:675) > > at io.prestosql.operator.Driver.processFor(Driver.java:276) > > at > > > > > io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075) > > at > > > > > io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163) > > at > > > > > io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484) > > at > > > io.prestosql.$gen.Presto_332__testversion20211217_065757_2.run(Unknown > > Source) > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > Caused by: java.nio.BufferUnderflowException > > at java.nio.Buffer.nextGetIndex(Buffer.java:532) > > at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:417) > > at > > > > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.loadSchema(PulsarSqlSchemaInfoProvider.java:106) > > at > > > > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.access$000(PulsarSqlSchemaInfoProvider.java:49) > > at > > > > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider$1.load(PulsarSqlSchemaInfoProvider.java:61) > > at > > > > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider$1.load(PulsarSqlSchemaInfoProvider.java:58) > > at > > > > > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529) > > at > > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278) > > at > > > > > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155) > > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045) > > ... 18 more > > ``` > > > > An issue can be found here https://github.com/apache/pulsar/issues/12284 > , > > > > This doesn't look like a regression introduced in 2.9.1 > As this is a security related release I would continue the release process. > > What do you think ? > > Enrico > > > > my test steps are very simple: > > > > 1. Start presto worker, `bin/pulsar sql-worker run` > > 2. Produce some messages, `bin/pulsar-client produce -m "hello" -n 10 > > test_wordcount_src` > > 3. Query the data from the topic, `select * from > > pulsar."public/default"."test_wordcount_src";` > > > > Not able to query the produced data and get errors in the Pulsar SQL > > worker. > > > > Penghui > > > > On Fri, Dec 17, 2021 at 5:33 AM Matteo Merli wrote: > > > > > +1 > > > > > > Checked: > > > * Signatures > > > * Bin distribution: > > > - NOTICE, README, LICENSE > > > - Start standalone service and producer/consumer test > > > * Src distribution: > > > - NOTICE, README, LICENSE > > > - Compile and unit tests > > > - Start standalone service > > > * Checked staging maven repository artifacts > > > * Checked docker images > > > > > > > > > Matteo > > > > > > -- > > > Matteo Merli > > > > > > > > > > > > > > > On Thu, Dec 16, 2021 at 12:53 PM Enrico Olivelli > > > wrote: > > > > > > > > I have pushed the docker images to my personal dockehub account > > > > > > > > eolivelli/pulsar:2.9.1rc2 > > > > eolivelli/pulsar-all:2.9.1rc2 > > > > > > > > Enrico > > > > > > > > Il Gio 16 Dic 2021, 15:57 Nicolò Boschi ha > >
Re: [VOTE] Apache Pulsar 2.9.1 candidate 2
Hi PengHu, Will this issue be fixed in the future releases? PengHui Li 於 2021年12月17日 週五 21:53 寫道： > Hi Enrico, > > I'm ok, it only happens when the message is without a schema version. > So I'm not giving -1. > > Thanks, > Penghui > > > On Fri, Dec 17, 2021 at 7:33 PM Enrico Olivelli > wrote: > > > Peng Hui, > > > > Il giorno ven 17 dic 2021 alle ore 08:09 PengHui Li > > ha > > scritto: > > > > > Checked: > > > > > > - Build from the src > > > - Check signatures > > > - Follow the validation process > > > > > > But when I try to verify PulsarSQL, got following exceptions: > > > > > > ``` > > > 2021-12-17T14:58:18.958+0800 ERROR remote-task-callback-3 > > > io.prestosql.execution.StageStateMachine Stage > > > 20211217_065818_1_cahiv.1 failed > > > com.google.common.util.concurrent.UncheckedExecutionException: > > > java.nio.BufferUnderflowException > > > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2051) > > > at com.google.common.cache.LocalCache.get(LocalCache.java:3951) > > > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) > > > at > > > > > > > > > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935) > > > at > > > > > > > > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.getSchemaByVersion(PulsarSqlSchemaInfoProvider.java:76) > > > at > > > > > > > > > org.apache.pulsar.sql.presto.PulsarRecordCursor.advanceNextPosition(PulsarRecordCursor.java:485) > > > at > > > > > > > > > io.prestosql.spi.connector.RecordPageSource.getNextPage(RecordPageSource.java:90) > > > at > > > > > > > > > io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:302) > > > at io.prestosql.operator.Driver.processInternal(Driver.java:379) > > > at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283) > > > at io.prestosql.operator.Driver.tryWithLock(Driver.java:675) > > > at io.prestosql.operator.Driver.processFor(Driver.java:276) > > > at > > > > > > > > > io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075) > > > at > > > > > > > > > io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163) > > > at > > > > > > > > > io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484) > > > at > > > > > > io.prestosql.$gen.Presto_332__testversion20211217_065757_2.run(Unknown > > > Source) > > > at > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > > at > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > at java.lang.Thread.run(Thread.java:748) > > > Caused by: java.nio.BufferUnderflowException > > > at java.nio.Buffer.nextGetIndex(Buffer.java:532) > > > at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:417) > > > at > > > > > > > > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.loadSchema(PulsarSqlSchemaInfoProvider.java:106) > > > at > > > > > > > > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.access$000(PulsarSqlSchemaInfoProvider.java:49) > > > at > > > > > > > > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider$1.load(PulsarSqlSchemaInfoProvider.java:61) > > > at > > > > > > > > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider$1.load(PulsarSqlSchemaInfoProvider.java:58) > > > at > > > > > > > > > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529) > > > at > > > > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278) > > > at > > > > > > > > > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155) > > > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045) > > > ... 18 more > > > ``` > > > > > > An issue can be found here > https://github.com/apache/pulsar/issues/12284 > > , > > > > > > > This doesn't look like a regression introduced in 2.9.1 > > As this is a security related release I would continue the release > process. > > > > What do you think ? > > > > Enrico > > > > > > > my test steps are very simple: > > > > > > 1. Start presto worker, `bin/pulsar sql-worker run` > > > 2. Produce some messages, `bin/pulsar-client produce -m "hello" -n 10 > > > test_wordcount_src` > > > 3. Query the data from the topic, `select * from > > > pulsar."public/default"."test_wordcount_src";` > > > > > > Not able to query the produced data and get errors in the Pulsar SQL > > > worker. > > > > > > Penghui > > > > > > On Fri, Dec 17, 2021 at 5:33 AM Matteo Merli > wrote: > > > > > > > +1 > > > > > > > > Checked: > > > > * Signatures > > > > * Bin distribution: > > > > - NOTICE, README, LICENSE > > > > - Start standalone service and producer/consumer test > > > > * Src distribution: > > > > - NOTICE, README, LICENSE > > > > - Compile and unit tests > > > > - Start standalone service > > > > * Checked staging ma
Re: [VOTE] Apache Pulsar 2.9.1 candidate 2
Thank you for the rc! +1 (non-binding) * verify sha512 checksums * verify gpg signatures * build pulsar-all docker image * execute Fallout distributed system test (produce / receive 10k messages) Building the docker image as indicated in the README: mvn clean install -DskipTests mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests I still get the error described in this PR#11951 ( https://github.com/apache/pulsar/pull/11951) which I suppose has been cherry picked in 2.9.1. Giving enough permissions to /docker/pulsar/scripts/ in the src package and then building the docker image again solved the issue. I also tested the problem with the image Enrico provided, this way: docker run -it --entrypoint bash eolivelli/pulsar-all:2.9.1rc2 ls -al /pulsar/bin and that one does have the right permissions. Enrico, did you build the image using the same mvn commands (^^^), or is there some other way to build it? Thank you, Max On Fri, 17 Dec 2021 at 16:18, 陳智弘 wrote: > Hi PengHu, > > Will this issue be fixed in the future releases? > > PengHui Li 於 2021年12月17日 週五 21:53 寫道： > > > Hi Enrico, > > > > I'm ok, it only happens when the message is without a schema version. > > So I'm not giving -1. > > > > Thanks, > > Penghui > > > > > > On Fri, Dec 17, 2021 at 7:33 PM Enrico Olivelli > > wrote: > > > > > Peng Hui, > > > > > > Il giorno ven 17 dic 2021 alle ore 08:09 PengHui Li < > peng...@apache.org> > > > ha > > > scritto: > > > > > > > Checked: > > > > > > > > - Build from the src > > > > - Check signatures > > > > - Follow the validation process > > > > > > > > But when I try to verify PulsarSQL, got following exceptions: > > > > > > > > ``` > > > > 2021-12-17T14:58:18.958+0800 ERROR remote-task-callback-3 > > > > io.prestosql.execution.StageStateMachine Stage > > > > 20211217_065818_1_cahiv.1 failed > > > > com.google.common.util.concurrent.UncheckedExecutionException: > > > > java.nio.BufferUnderflowException > > > > at > > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2051) > > > > at com.google.common.cache.LocalCache.get(LocalCache.java:3951) > > > > at > com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) > > > > at > > > > > > > > > > > > > > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935) > > > > at > > > > > > > > > > > > > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.getSchemaByVersion(PulsarSqlSchemaInfoProvider.java:76) > > > > at > > > > > > > > > > > > > > org.apache.pulsar.sql.presto.PulsarRecordCursor.advanceNextPosition(PulsarRecordCursor.java:485) > > > > at > > > > > > > > > > > > > > io.prestosql.spi.connector.RecordPageSource.getNextPage(RecordPageSource.java:90) > > > > at > > > > > > > > > > > > > > io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:302) > > > > at io.prestosql.operator.Driver.processInternal(Driver.java:379) > > > > at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283) > > > > at io.prestosql.operator.Driver.tryWithLock(Driver.java:675) > > > > at io.prestosql.operator.Driver.processFor(Driver.java:276) > > > > at > > > > > > > > > > > > > > io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075) > > > > at > > > > > > > > > > > > > > io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163) > > > > at > > > > > > > > > > > > > > io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484) > > > > at > > > > > > > > > > io.prestosql.$gen.Presto_332__testversion20211217_065757_2.run(Unknown > > > > Source) > > > > at > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > > > at > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > at java.lang.Thread.run(Thread.java:748) > > > > Caused by: java.nio.BufferUnderflowException > > > > at java.nio.Buffer.nextGetIndex(Buffer.java:532) > > > > at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:417) > > > > at > > > > > > > > > > > > > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.loadSchema(PulsarSqlSchemaInfoProvider.java:106) > > > > at > > > > > > > > > > > > > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.access$000(PulsarSqlSchemaInfoProvider.java:49) > > > > at > > > > > > > > > > > > > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider$1.load(PulsarSqlSchemaInfoProvider.java:61) > > > > at > > > > > > > > > > > > > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider$1.load(PulsarSqlSchemaInfoProvider.java:58) > > > > at > > > > > > > > > > > > > > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529) > > > > at > > > > > > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278) > > > > at > > > > > > > > > > > > > > com.google.common.cache.Local
Re: [VOTE] Apache Pulsar 2.9.1 candidate 2
> Will this issue be fixed in the future releases? Yes, 2.8.2 and 2.9.2 will fix the problem. Penghui On Sat, Dec 18, 2021 at 3:28 AM Massimiliano Mirelli < massimilianomirelli...@gmail.com> wrote: > Thank you for the rc! > > +1 (non-binding) > > * verify sha512 checksums > * verify gpg signatures > * build pulsar-all docker image > * execute Fallout distributed system test (produce / receive 10k messages) > > Building the docker image as indicated in the README: > > mvn clean install -DskipTests > mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests > > I still get the error described in this PR#11951 ( > https://github.com/apache/pulsar/pull/11951) which I suppose has been > cherry picked in 2.9.1. > > Giving enough permissions to /docker/pulsar/scripts/ in the src package and > then building the docker image again solved the issue. > > I also tested the problem with the image Enrico provided, this way: > > docker run -it --entrypoint bash eolivelli/pulsar-all:2.9.1rc2 > ls -al /pulsar/bin > > and that one does have the right permissions. > > Enrico, did you build the image using the same mvn commands (^^^), or is > there some other way to build it? > > Thank you, > Max > > On Fri, 17 Dec 2021 at 16:18, 陳智弘 wrote: > > > Hi PengHu, > > > > Will this issue be fixed in the future releases? > > > > PengHui Li 於 2021年12月17日 週五 21:53 寫道： > > > > > Hi Enrico, > > > > > > I'm ok, it only happens when the message is without a schema version. > > > So I'm not giving -1. > > > > > > Thanks, > > > Penghui > > > > > > > > > On Fri, Dec 17, 2021 at 7:33 PM Enrico Olivelli > > > wrote: > > > > > > > Peng Hui, > > > > > > > > Il giorno ven 17 dic 2021 alle ore 08:09 PengHui Li < > > peng...@apache.org> > > > > ha > > > > scritto: > > > > > > > > > Checked: > > > > > > > > > > - Build from the src > > > > > - Check signatures > > > > > - Follow the validation process > > > > > > > > > > But when I try to verify PulsarSQL, got following exceptions: > > > > > > > > > > ``` > > > > > 2021-12-17T14:58:18.958+0800 ERROR remote-task-callback-3 > > > > > io.prestosql.execution.StageStateMachine Stage > > > > > 20211217_065818_1_cahiv.1 failed > > > > > com.google.common.util.concurrent.UncheckedExecutionException: > > > > > java.nio.BufferUnderflowException > > > > > at > > > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2051) > > > > > at com.google.common.cache.LocalCache.get(LocalCache.java:3951) > > > > > at > > com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) > > > > > at > > > > > > > > > > > > > > > > > > > > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.getSchemaByVersion(PulsarSqlSchemaInfoProvider.java:76) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.pulsar.sql.presto.PulsarRecordCursor.advanceNextPosition(PulsarRecordCursor.java:485) > > > > > at > > > > > > > > > > > > > > > > > > > > io.prestosql.spi.connector.RecordPageSource.getNextPage(RecordPageSource.java:90) > > > > > at > > > > > > > > > > > > > > > > > > > > io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:302) > > > > > at io.prestosql.operator.Driver.processInternal(Driver.java:379) > > > > > at > io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283) > > > > > at io.prestosql.operator.Driver.tryWithLock(Driver.java:675) > > > > > at io.prestosql.operator.Driver.processFor(Driver.java:276) > > > > > at > > > > > > > > > > > > > > > > > > > > io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075) > > > > > at > > > > > > > > > > > > > > > > > > > > io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163) > > > > > at > > > > > > > > > > > > > > > > > > > > io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484) > > > > > at > > > > > > > > > > > > > > > io.prestosql.$gen.Presto_332__testversion20211217_065757_2.run(Unknown > > > > > Source) > > > > > at > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > > > > at > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > > at java.lang.Thread.run(Thread.java:748) > > > > > Caused by: java.nio.BufferUnderflowException > > > > > at java.nio.Buffer.nextGetIndex(Buffer.java:532) > > > > > at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:417) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.loadSchema(PulsarSqlSchemaInfoProvider.java:106) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.pulsar.sql.presto.PulsarSqlSchemaInfoProvider.access$000(PulsarSqlSchemaInfoProvider.java:49) > > > > > at > > > > > > > >
