On Fri, Dec 20, 2024 at 10:58 AM David Handermann
<exceptionfact...@apache.org> wrote:
>
> Nissim and Bryan,
>
> On closer evaluation, this appears to be a threading issue with the
> new StandardProxiedEntityEncoder and its use of
> CharsetEncoder.canEncode(). With CharsetEncoder being stateful, that
> explains why this can happen intermittently.
>
> I will cancel this release candidate build and put together a Jira and
> pull request to correct behavior.
>
> Thanks for reporting this issue!
>
> Regards,
> David Handermann
>
> On Fri, Dec 20, 2024 at 10:40 AM Bryan Bende <bbe...@gmail.com> wrote:
> >
> > I just got the same error while on the primary/coordinator node,
> > investigating...
> >
> > On Fri, Dec 20, 2024 at 10:34 AM David Handermann
> > <exceptionfact...@apache.org> wrote:
> > >
> > > Hi Nissim,
> > >
> > > Thanks for providing the stack trace. This stack trace seems to point
> > > to some other issue, as the failure itself relates to character set
> > > parsing. It sounds like one node is returning some type of error
> > > response from another node during the request replication process.
> > >
> > > If you can look at the user and app logs across all cluster nodes,
> > > that should provide some additional background for the source of the
> > > error.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Fri, Dec 20, 2024 at 9:12 AM Nissim Shiman <nshi...@yahoo.com.invalid>
> > > wrote:
> > > >
> > > > Hello Team,
> > > >
> > > > I am running into an issue when running on a three node cluster. If
> > > > the ui is brought up on one of the non-primary nodes, and refreshed, it
> > > > give an error:
> > > >
> > > > Invalid State
> > > > Current state = CODING_END, new state = CODING
> > > >
> > > > I am interested if others see anything similar.
> > > >
> > > > Code was compiled and run on RHEL8 with java-21-openjdk-21.0.5
> > > >
> > > > The nifi-user log has this:
> > > >
> > > > 2024-12-20 14:56:55,456 WARN [NiFi Web Server-156]
> > > > o.a.n.w.a.c.IllegalStateExceptionMapper
> > > > java.lang.IllegalStateException: Current state = CODING_END, new state
> > > > = CODING. Returning Conflict response.java.lang.IllegalStateException:
> > > > Current state = CODING_END, new state = CODING at
> > > > java.base/java.nio.charset.CharsetEncoder.throwIllegalStateException(CharsetEncoder.java:1012)
> > > > at
> > > > java.base/java.nio.charset.CharsetEncoder.canEncode(CharsetEncoder.java:924)
> > > > at
> > > > java.base/java.nio.charset.CharsetEncoder.canEncode(CharsetEncoder.java:1005)
> > > > at
> > > > org.apache.nifi.security.proxied.entity.StandardProxiedEntityEncoder.getSanitizedIdentity(StandardProxiedEntityEncoder.java:76)
> > > > at
> > > > org.apache.nifi.security.proxied.entity.StandardProxiedEntityEncoder.getEncodedEntity(StandardProxiedEntityEncoder.java:64)
> > > > at
> > > > java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
> > > > at
> > > > java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
> > > > at
> > > > java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
> > > > at
> > > > java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
> > > > at
> > > > java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
> > > > at
> > > > java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> > > > at
> > > > java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
> > > > at
> > > > org.apache.nifi.web.security.ProxiedEntitiesUtils.buildProxiedEntitiesChainString(ProxiedEntitiesUtils.java:114)
> > > > at
> > > > org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.updateRequestHeaders(ThreadPoolRequestReplicator.java:249)
> > > > at
> > > > org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.forwardToCoordinator(ThreadPoolRequestReplicator.java:333)
> > > > at
> > > > org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.forwardToCoordinator(ThreadPoolRequestReplicator.java:324)
> > > > at
> > > > org.apache.nifi.web.api.ApplicationResource.replicateNodeResponse(ApplicationResource.java:1039)
> > > > at
> > > > org.apache.nifi.web.api.ApplicationResource.replicateNodeResponse(ApplicationResource.java:1008)
> > > > at
> > > > org.apache.nifi.web.api.ApplicationResource.replicate(ApplicationResource.java:999)
> > > > at
> > > > org.apache.nifi.web.api.ApplicationResource.replicate(ApplicationResource.java:983)
> > > > at
> > > > org.apache.nifi.web.api.ApplicationResource.replicate(ApplicationResource.java:958)
> > > > at
> > > > org.apache.nifi.web.api.FlowResource.getFlow(FlowResource.java:499)
> > > > at
> > > > java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:580)
> > > > at
> > > > org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
> > > > at
> > > > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
> > > > at
> > > > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
> > > > at
> > > > org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
> > > > at
> > > > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
> > > > at
> > > > org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
> > > > at
> > > > org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
> > > > at
> > > > org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
> > > > at
> > > > org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)
> > > > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
> > > > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
> > > > at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
> > > > at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
> > > > at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
> > > > at
> > > > org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:266)
> > > > at
> > > > org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:253)
> > > > at
> > > > org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:696)
> > > > at
> > > > org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:397)
> > > > at
> > > > org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:349)
> > > > at
> > > > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
> > > > at
> > > > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:312)
> > > > at
> > > > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
> > > > at
> > > > org.eclipse.jetty.ee10.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1379)
> > > > at
> > > > org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:736)
> > > > at
> > > > org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1619)
> > > > at
> > > > org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231)
> > > > at
> > > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:365)
> > > > at
> > > > org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:101)
> > > > at
> > > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
> > > > at
> > > > org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
> > > > at
> > > > org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
> > > > at
> > > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
> > > > at
> > > > org.apache.nifi.web.security.log.AuthenticationUserFilter.doFilterInternal(AuthenticationUserFilter.java:57)
> > > > at
> > > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
> > > > at
> > > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
> > > > at
> > > > org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
> > > > at
> > > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
> > > > at
> > > > org.springframework.security.oauth2.server.resource.web.authentication.BearerTokenAuthenticationFilter.doFilterInternal(BearerTokenAuthenticationFilter.java:145)
> > > > at
> > > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
> > > > at
> > > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
> > > > at
> > > > org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:94)
> > > > at
> > > > org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:56)
> > > > at
> > > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
> > > > at
> > > > org.apache.nifi.web.security.csrf.CsrfCookieFilter.doFilterInternal(CsrfCookieFilter.java:43)
> > > > at
> > > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
> > > > at
> > > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
> > > > at
> > > > org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:117)
> > > > at
> > > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
> > > > at
> > > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
> > > > at
> > > > org.apache.nifi.web.security.csrf.SkipReplicatedCsrfFilter.doFilterInternal(SkipReplicatedCsrfFilter.java:59)
> > > > at
> > > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
> > > > at
> > > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
> > > > at
> > > > org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)
> > > > at
> > > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
> > > > at
> > > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
> > > > at
> > > > org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233)
> > > > at
> > > > org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191)
> > > > at
> > > > org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:362)
> > > > at
> > > > org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:278)
> > > > at
> > > > org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
> > > > at
> > > > org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1591)
> > > > at
> > > > org.apache.nifi.web.filter.ExceptionFilter.doFilter(ExceptionFilter.java:44)
> > > > at
> > > > org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:208)
> > > > at
> > > > org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1591)
> > > > at
> > > > org.eclipse.jetty.ee10.servlets.DoSFilter.doFilterChain(DoSFilter.java:462)
> > > > at
> > > > org.apache.nifi.web.server.filter.DataTransferExcludedDoSFilter.doFilterChain(DataTransferExcludedDoSFilter.java:51)
> > > > at
> > > > org.eclipse.jetty.ee10.servlets.DoSFilter.doFilter(DoSFilter.java:317)
> > > > at
> > > > org.eclipse.jetty.ee10.servlets.DoSFilter.doFilter(DoSFilter.java:282)
> > > > at
> > > > org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
> > > > at
> > > > org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1591)
> > > > at
> > > > org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
> > > > at
> > > > org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
> > > > at
> > > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
> > > > at
> > > > org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
> > > > at
> > > > org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1591)
> > > > at
> > > > org.apache.nifi.web.server.log.RequestAuthenticationFilter.doFilterInternal(RequestAuthenticationFilter.java:59)
> > > > at
> > > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
> > > > at
> > > > org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
> > > > at
> > > > org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1591)
> > > > at
> > > > org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1552)
> > > > at
> > > > org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:819)
> > > > at
> > > > org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:436)
> > > > at
> > > > org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:469)
> > > > at
> > > > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)
> > > > at
> > > > org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)
> > > > at
> > > > org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1060)
> > > > at
> > > > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:181)
> > > > at
> > > > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:181)
> > > > at org.eclipse.jetty.server.Server.handle(Server.java:182)
> > > > at
> > > > org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:662)
> > > > at
> > > > org.eclipse.jetty.util.thread.Invocable$ReadyTask.run(Invocable.java:175)
> > > > at
> > > > org.eclipse.jetty.http2.server.internal.HttpStreamOverHTTP2$1.run(HttpStreamOverHTTP2.java:135)
> > > > at
> > > > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)
> > > > at
> > > > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)
> > > > at
> > > > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
> > > > at
> > > > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)
> > > > at
> > > > org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:209)
> > > > at
> > > > org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:156)
> > > > at
> > > > org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:442)
> > > > at
> > > > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
> > > > at
> > > > org.eclipse.jetty.io.ssl.SslConnection$SslEndPoint.onFillable(SslConnection.java:575)
> > > > at
> > > > org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:390)
> > > > at
> > > > org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:150)
> > > > at
> > > > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
> > > > at
> > > > org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
> > > > at
> > > > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)
> > > > at
> > > > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)
> > > > at
> > > > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
> > > > at
> > > > org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)
> > > > at
> > > > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)
> > > > at
> > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)
> > > > at
> > > > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)
> > > > at
> > > > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)
> > > > at java.base/java.lang.Thread.run(Thread.java:1583)
> > > >
> > > >
> > > > On Thursday, December 19, 2024 at 09:02:34 PM UTC, David Handermann
> > > > <exceptionfact...@apache.org> wrote:
> > > >
> > > > Team,
> > > >
> > > > I am pleased to be calling this vote for the source release of Apache
> > > > NiFi 2.1.0.
> > > >
> > > > Please review the following guide for how to verify a release candidate
> > > > build:
> > > >
> > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification
> > > >
> > > > The source being voted on the and the convenience binaries are
> > > > available on the Apache Distribution Repository:
> > > >
> > > > https://dist.apache.org/repos/dist/dev/nifi/nifi-2.1.0
> > > >
> > > > The build artifacts are available on the Apache Nexus Repository:
> > > >
> > > > https://repository.apache.org/content/repositories/orgapachenifi-1303/
> > > >
> > > > Git Tag: nifi-2.1.0-RC2
> > > > Git Commit ID: 2f91793272f48efb80d18a01d7086f406b6d9968
> > > > GitHub Commit Link:
> > > > https://github.com/apache/nifi/commit/2f91793272f48efb80d18a01d7086f406b6d9968
> > > >
> > > > Checksums of nifi-2.1.0-source-release.zip
> > > >
> > > > SHA512:
> > > > 374227bbdf9df45ce30266d1e5d86fbd022f88876ac765a29b65dde3f9da7f675cf4e4e313e9b6f77067b73ea7a7cc82ce4599cac04b7a38175c81e46ef1ed2d
> > > >
> > > > Release artifacts are signed with the following key:
> > > >
> > > > https://people.apache.org/keys/committer/exceptionfactory.asc
> > > >
> > > > KEYS file is available on the Apache Distribution Repository:
> > > >
> > > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > > >
> > > > Issues resolved for this version: 88
> > > >
> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12355215
> > > >
> > > > Release note highlights can be found on the project wiki:
> > > >
> > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.1.0
> > > >
> > > > The vote will be open for 72 hours.
> > > >
> > > > Please download the release candidate and evaluate the necessary items
> > > > including checking hashes, signatures, build from source, and test.
> > > > Then please vote:
> > > >
> > > > [] +1 Release this package as nifi-2.1.0
> > > > [] +0 no opinion
> > > > [] -1 Do not release this package because...
> > > >
