[DISCUSS] [PIP-160] Batch writing ledger for transaction operation
Hi Pulsar community: I open a pip to discuss "Batch writing ledger for transaction operation" Proposal Link: https://github.com/apache/pulsar/issues/15370 ## Motivation Before reading the background, I suggest you read section “Transaction Flow” of [PIP-31: Transactional Streaming]( https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx ) ### Normal Flow vs. Transaction Flow  In *Figure 1. Normal Flow vs. Transaction Flow*: - The gray square boxes represent logical components. - All the blue boxes represent logs. The logs are usually Managed ledger - Each arrow represents the request flow or message flow. These operations occur in sequence indicated by the numbers next to each arrow. - The black arrows indicate those shared by transaction and normal flow. - The blue arrows represent normal-message-specific flow. - The orange arrows represent transaction-message-specific flow. - The sections below are numbered to match the operations showed in the diagram(differ from [PIP-31: Transactional Streaming]( https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx )) 2.4a Write logs to ledger which Acknowledgement State is PENDING_ACK [Acknowledgement State Machine]( https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#bookmark=id.4bikq6sjiy8u) tells about the changes of the Acknowledge State and why we need persistent “The Log which the Acknowledgement State is PENDING_ACK”. 2.4a’ Mark messages is no longer useful with current subscription Update `Cursor` to mark the messages as DELETED. So they can be deleted. 3.2b Mark messages is no longer useful with current subscription The implementation here is exactly the same as 2.4a’, except that the execution is triggered later, after the Transaction has been committed. ### Analyze the performance cost of transaction As you can see Figure 1. Normal Flow vs. Transaction Flow]: 2.4a 'and 3.2b are exactly the same logic, so the remaining orange arrows are the additional performance overhead of all transactions. In terms of whether or not each transaction is executed multiple times, we can split the flow into two classes(Optimizing a process that is executed multiple times will yield more benefits): - Executed once each transaction: flow-1.x and flow-3.x - Executed multiple times each transaction: flow-2.x So optimizing the flow 2.x with a lot of execution is a good choice. Let's split flow-2.x into two groups: those that cost more and those that cost less: - No disk written: flow-2.1 and fow-2.3 - Disk written: fow-2.1a, fow-2.3a, flow-2.4a >From the previous analysis, we found that optimizing flow-2.1a, flow-2.3a, flow-2.4a would bring the most benefits, and batch writes would be an excellent solution for multiple disk writes. Flow-2.1a and Flow-2.3a are both manipulations written into the transaction log, we can combine them in one batch; 2.4a is the operation of writing pending ACK log, we combine multiple 2.4a's into one batch for processing. As we can see from “Transaction Flow” of [PIP-31: Transactional Streaming]( https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx), these instructions are strictly sequential (guaranteed by the client): - flow-1.x end before flow-2.x start - flow-2.x end before flow-3.x start - flow-3.1a end before flow-3.3a start Therefore, the broker does not need to worry about the dependency of these flows, we can also put flow-1a flow-31a and flow-3.3a into The Transaction Log Batch too. ## Goal Provide a mechanism for Transaction Log Store and Pending Ack Store: accept multiple write requests, buffer all those records, and persist to a single BK entry(aka “Batched Entry”). This will improve broker->BK throughput. - Allow users to specify control of the max size, max record of The Buffer. - Allow users to specify control max delay time of The Write Request. - Multiple raw data can be recovered from Batched Entry. - Configurable “batched implementation” and “common implementation” switch. ## Approach ### Buffer requests and write Bookie Create a new protobuf record called “Batched Transaction Data” with an array inside. When receive a request, we put it in the array. Request: ``` [Request 1]{ data, callback } [Request 2]]{ data, callback } … … [Request N]]{ data, callback } ``` Buffer: ``` [BatchedTransactionData]{ list=[Request 1, Request 2 … Request N] } ``` Write Bookie: ``` LedgerHandle async write ( BatchedTransactionData to byteBuf ) LedgerHandle callback: ledgerId=1, entryId=1 ``` Request-Callback: ``` Callback 1: {ledgerId=1, entryId=1, batchSize=N, batchIndex=0} Callback 2: {ledgerId=1, entryId=1, batchSize=N, batchIndex=1} … … Callback N: {ledgerId=1, entryId=1, batchSize=N, batchIndex=N-1} ``` ### Delet
Re: [DISCUSS] [PIP-160] Batch writing ledger for transaction operation
I have read the PIP, and overall I agree with the design. Good work ! I am not sure I understand the part of making it configurable via a classname. I believe it is better to simply have a flag "transactionEnableBatchWrites". Otherwise the matrix of possible implementations will grow without limits. Enrico Il giorno ven 10 giu 2022 alle ore 11:35 Yubiao Feng ha scritto: > > Hi Pulsar community: > > I open a pip to discuss "Batch writing ledger for transaction operation" > > Proposal Link: https://github.com/apache/pulsar/issues/15370 > > ## Motivation > > Before reading the background, I suggest you read section “Transaction > Flow” of [PIP-31: Transactional Streaming]( > https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx > ) > > ### Normal Flow vs. Transaction Flow >  > In *Figure 1. Normal Flow vs. Transaction Flow*: > - The gray square boxes represent logical components. > - All the blue boxes represent logs. The logs are usually Managed ledger > - Each arrow represents the request flow or message flow. These operations > occur in sequence indicated by the numbers next to each arrow. > - The black arrows indicate those shared by transaction and normal flow. > - The blue arrows represent normal-message-specific flow. > - The orange arrows represent transaction-message-specific flow. > - The sections below are numbered to match the operations showed in the > diagram(differ from [PIP-31: Transactional Streaming]( > https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx > )) > > > 2.4a Write logs to ledger which Acknowledgement State is PENDING_ACK > [Acknowledgement State Machine]( > https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#bookmark=id.4bikq6sjiy8u) > tells about the changes of the Acknowledge State and why we need persistent > “The Log which the Acknowledgement State is PENDING_ACK”. > 2.4a’ Mark messages is no longer useful with current subscription > Update `Cursor` to mark the messages as DELETED. So they can be deleted. > 3.2b Mark messages is no longer useful with current subscription > The implementation here is exactly the same as 2.4a’, except that the > execution is triggered later, after the Transaction has been committed. > > > ### Analyze the performance cost of transaction > As you can see Figure 1. Normal Flow vs. > Transaction Flow]: 2.4a 'and 3.2b are exactly the same logic, so the > remaining orange arrows are the additional performance overhead of all > transactions. > In terms of whether or not each transaction is executed multiple times, we > can split the flow into two classes(Optimizing a process that is executed > multiple times will yield more benefits): > - Executed once each transaction: flow-1.x and flow-3.x > - Executed multiple times each transaction: flow-2.x > > So optimizing the flow 2.x with a lot of execution is a good choice. Let's > split flow-2.x into two groups: those that cost more and those that cost > less: > - No disk written: flow-2.1 and fow-2.3 > - Disk written: fow-2.1a, fow-2.3a, flow-2.4a > > From the previous analysis, we found that optimizing flow-2.1a, flow-2.3a, > flow-2.4a would bring the most benefits, and batch writes would be an > excellent solution for multiple disk writes. Flow-2.1a and Flow-2.3a are > both manipulations written into the transaction log, we can combine them in > one batch; 2.4a is the operation of writing pending ACK log, we combine > multiple 2.4a's into one batch for processing. > As we can see from “Transaction Flow” of [PIP-31: Transactional Streaming]( > https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx), > these instructions are strictly sequential (guaranteed by the client): > - flow-1.x end before flow-2.x start > - flow-2.x end before flow-3.x start > - flow-3.1a end before flow-3.3a start > > Therefore, the broker does not need to worry about the dependency of these > flows, we can also put flow-1a flow-31a and flow-3.3a into The Transaction > Log Batch too. > > ## Goal > Provide a mechanism for Transaction Log Store and Pending Ack Store: accept > multiple write requests, buffer all those records, and persist to a single > BK entry(aka “Batched Entry”). This will improve broker->BK throughput. > - Allow users to specify control of the max size, max record of The Buffer. > - Allow users to specify control max delay time of The Write Request. > - Multiple raw data can be recovered from Batched Entry. > - Configurable “batched implementation” and “common implementation” switch. > > ## Approach > ### Buffer requests and write Bookie > Create a new protobuf record called “Batched Transaction Data” with an > array inside. When receive a request, we put it in the array. > > Request: >
Re: [DISCUSS] [PIP-160] Batch writing ledger for transaction operation
I think this is a nice optimization of transaction internal components. But I have two little questions: 1. I noticed you write conf transactionMetadataStoreProviderClassName is configured in broker.conf. Should this be added to `Configuration Changes`? 2. This proposal will add a batch mechanism for pending ack, but this proposal is no detailed description for pending ack similar to the description of metadata log batch. On Fri, Jun 10, 2022 at 7:15 PM Enrico Olivelli wrote: > I have read the PIP, and overall I agree with the design. > Good work ! > > I am not sure I understand the part of making it configurable via a > classname. > I believe it is better to simply have a flag > "transactionEnableBatchWrites". > Otherwise the matrix of possible implementations will grow without limits. > > Enrico > > Il giorno ven 10 giu 2022 alle ore 11:35 Yubiao Feng > ha scritto: > > > > Hi Pulsar community: > > > > I open a pip to discuss "Batch writing ledger for transaction operation" > > > > Proposal Link: https://github.com/apache/pulsar/issues/15370 > > > > ## Motivation > > > > Before reading the background, I suggest you read section “Transaction > > Flow” of [PIP-31: Transactional Streaming]( > > > https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx > > ) > > > > ### Normal Flow vs. Transaction Flow > > >  > > In *Figure 1. Normal Flow vs. Transaction Flow*: > > - The gray square boxes represent logical components. > > - All the blue boxes represent logs. The logs are usually Managed ledger > > - Each arrow represents the request flow or message flow. These > operations > > occur in sequence indicated by the numbers next to each arrow. > > - The black arrows indicate those shared by transaction and normal flow. > > - The blue arrows represent normal-message-specific flow. > > - The orange arrows represent transaction-message-specific flow. > > - The sections below are numbered to match the operations showed in the > > diagram(differ from [PIP-31: Transactional Streaming]( > > > https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx > > )) > > > > > > 2.4a Write logs to ledger which Acknowledgement State is PENDING_ACK > > [Acknowledgement State Machine]( > > > https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#bookmark=id.4bikq6sjiy8u > ) > > tells about the changes of the Acknowledge State and why we need > persistent > > “The Log which the Acknowledgement State is PENDING_ACK”. > > 2.4a’ Mark messages is no longer useful with current subscription > > Update `Cursor` to mark the messages as DELETED. So they can be deleted. > > 3.2b Mark messages is no longer useful with current subscription > > The implementation here is exactly the same as 2.4a’, except that the > > execution is triggered later, after the Transaction has been committed. > > > > > > ### Analyze the performance cost of transaction > > As you can see Figure 1. Normal Flow > vs. > > Transaction Flow]: 2.4a 'and 3.2b are exactly the same logic, so the > > remaining orange arrows are the additional performance overhead of all > > transactions. > > In terms of whether or not each transaction is executed multiple times, > we > > can split the flow into two classes(Optimizing a process that is executed > > multiple times will yield more benefits): > > - Executed once each transaction: flow-1.x and flow-3.x > > - Executed multiple times each transaction: flow-2.x > > > > So optimizing the flow 2.x with a lot of execution is a good choice. > Let's > > split flow-2.x into two groups: those that cost more and those that cost > > less: > > - No disk written: flow-2.1 and fow-2.3 > > - Disk written: fow-2.1a, fow-2.3a, flow-2.4a > > > > From the previous analysis, we found that optimizing flow-2.1a, > flow-2.3a, > > flow-2.4a would bring the most benefits, and batch writes would be an > > excellent solution for multiple disk writes. Flow-2.1a and Flow-2.3a are > > both manipulations written into the transaction log, we can combine them > in > > one batch; 2.4a is the operation of writing pending ACK log, we combine > > multiple 2.4a's into one batch for processing. > > As we can see from “Transaction Flow” of [PIP-31: Transactional > Streaming]( > > > https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx > ), > > these instructions are strictly sequential (guaranteed by the client): > > - flow-1.x end before flow-2.x start > > - flow-2.x end before flow-3.x start > > - flow-3.1a end before flow-3.3a start > > > > Therefore, the broker does not need to worry about the dependency of > these > > flows, we can also put flow-1a flow-31a and flow-3.3a into The > Transaction > > Log Batch too. > > > > ## Goal > > Provide a mechan
[GitHub] [pulsar-manager] daniel-bartz opened a new issue, #464: JWT/TLS authentication issue creating en new Environment (pulsar-manager 0.3.0)
daniel-bartz opened a new issue, #464: URL: https://github.com/apache/pulsar-manager/issues/464 When I want to add a new _Environment_ via the web interface, nothing happen and I get the following error in the backend logs: ``` Jun 10 14:10:52 hostname pulsar-manager[6494]: 2022-06-10 14:10:52.707 ERROR 6494 --- [nio-7750-exec-6] o.a.c.c.C.[.[.[/].[dispatcherServlet]: Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.lang.NoSuchFieldError: id_alg_AEADChaCha20Poly1305] with root cause Jun 10 14:10:52 hostname pulsar-manager[6494]: java.lang.NoSuchFieldError: id_alg_AEADChaCha20Poly1305 ``` I built (tried both jdk8 & 11) the _pulsar-manager_ using `./gradlew build -x test` I tried to run it on both jdk8 & 11 and the same issue appeared. Notes: - I was using same approach for _pulsar-manager_ 0.2.0 on jdk11and it worked very nicely. - I tried to use the binary provided here https://archive.apache.org/dist/pulsar/pulsar-manager/pulsar-manager-0.3.0/ but could not manage to get it working with PostgreSQL. - I also tried to remove the TLS_CHACHA20_POLY1305_SHA256 cipher on the broker side but it did not help. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-manager] eolivelli commented on issue #464: JWT/TLS authentication issue creating en new Environment (pulsar-manager 0.3.0)
eolivelli commented on issue #464: URL: https://github.com/apache/pulsar-manager/issues/464#issuecomment-1152639969 Can you paste the full stack trace please? It looks like the problem is related only to TLS Probably there is some third party library to upgrade -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-manager] eolivelli commented on issue #464: JWT/TLS authentication issue creating en new Environment (pulsar-manager 0.3.0)
eolivelli commented on issue #464: URL: https://github.com/apache/pulsar-manager/issues/464#issuecomment-1152640851 https://stackoverflow.com/questions/47126562/bouncycastleprovider-throws-java-lang-nosuchfielderror-id-hmacwithsha3-224 You can try to upgrade BouncyCastle dependency to fix the problem -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-manager] daniel-bartz commented on issue #464: JWT/TLS authentication issue creating en new Environment (pulsar-manager 0.3.0)
daniel-bartz commented on issue #464: URL: https://github.com/apache/pulsar-manager/issues/464#issuecomment-1152681071 Here is the full stacktrace, so you can see I use a rather new BC lib ``` Jun 10 19:46:56 hostname pulsar-manager[49051]: 2022-06-10 19:46:56.586 ERROR 49051 --- [nio-7750-exec-6] o.a.c.c.C.[.[.[/].[dispatcherServlet]: Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.lang.NoSuchFieldError: id_alg_AEADChaCha20Poly1305] with root cause Jun 10 19:46:56 hostname pulsar-manager[49051]: java.lang.NoSuchFieldError: id_alg_AEADChaCha20Poly1305 Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at org.bouncycastle.jcajce.provider.symmetric.ChaCha$Mappings.configure(Unknown Source) ~[bcprov-ext-jdk15on-1.66.jar:1.66.00.0] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at org.bouncycastle.jce.provider.BouncyCastleProvider.loadAlgorithms(Unknown Source) ~[bcprov-ext-jdk15on-1.66.jar:1.66.00.0] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at org.bouncycastle.jce.provider.BouncyCastleProvider.setup(Unknown Source) ~[bcprov-ext-jdk15on-1.66.jar:1.66.00.0] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at org.bouncycastle.jce.provider.BouncyCastleProvider.access$000(Unknown Source) ~[bcprov-ext-jdk15on-1.66.jar:1.66.00.0] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at org.bouncycastle.jce.provider.BouncyCastleProvider$1.run(Unknown Source) ~[bcprov-ext-jdk15on-1.66.jar:1.66.00.0] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_332] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at org.bouncycastle.jce.provider.BouncyCastleProvider.(Unknown Source) ~[bcprov-ext-jdk15on-1.66.jar:1.66.00.0] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_332] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_332] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_332] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_332] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at java.lang.Class.newInstance(Class.java:442) ~[na:1.8.0_332] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at org.apache.pulsar.common.util.SecurityUtility.getBCProviderFromClassPath(SecurityUtility.java:125) ~[pulsar-common-2.7.0.jar:2.7.0] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at org.apache.pulsar.common.util.SecurityUtility.getProvider(SecurityUtility.java:102) ~[pulsar-common-2.7.0.jar:2.7.0] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at org.apache.pulsar.common.util.SecurityUtility.(SecurityUtility.java:69) ~[pulsar-common-2.7.0.jar:2.7.0] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.(AsyncHttpConnector.java:152) ~[pulsar-client-admin-original-2.7.0.jar:2.7.0] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnectorProvider.getConnector(AsyncHttpConnectorProvider.java:50) ~[pulsar-client-admin-original-2.7.0.jar:2.7.0] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at org.apache.pulsar.client.admin.PulsarAdmin.(PulsarAdmin.java:200) ~[pulsar-client-admin-original-2.7.0.jar:2.7.0] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at org.apache.pulsar.client.admin.internal.PulsarAdminBuilderImpl.build(PulsarAdminBuilderImpl.java:46) ~[pulsar-client-admin-original-2.7.0.jar:2.7.0] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at org.apache.pulsar.manager.service.impl.PulsarAdminServiceImpl.createPulsarAdmin(PulsarAdminServiceImpl.java:146) ~[pulsar-manager-master.jar:na] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at org.apache.pulsar.manager.service.impl.PulsarAdminServiceImpl.getPulsarAdmin(PulsarAdminServiceImpl.java:73) ~[pulsar-manager-master.jar:na] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at org.apache.pulsar.manager.service.impl.PulsarAdminServiceImpl.clusters(PulsarAdminServiceImpl.java:83) ~[pulsar-manager-master.jar:na] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at org.apache.pulsar.manager.controller.EnvironmentsController.addEnvironment(EnvironmentsController.java:124) ~[pulsar-manager-master.jar:na] Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at org.apache.pulsar.manager.controller.EnvironmentsController$$FastClassBySpringCGLIB$$8f0ec792.invoke() ~[pulsar-manager-master.jar:na] Jun 10 19:46:56 hostname pul
Re: Pulsar Flaky test report 2022-06-02 for PR builds in CI
Hi, Can I get this PR reviewed? This PR is to make some of the tests less flaky. https://github.com/apache/pulsar/pull/16011 Thanks, Heesung On Thu, Jun 2, 2022 at 5:06 AM Lari Hotari wrote: > Dear Pulsar community members, > > Here's a report of the flaky tests in Pulsar CI during the observation > period of 2022-05-26 to 2022-06-02 . > The full report is available as a Google Sheet, > > https://docs.google.com/spreadsheets/d/165FHpHjs5fHccSsmQM4beeg6brn-zfUjcrXf6xAu4yQ/edit?usp=sharing > > The report contains a subset of the test failures. > The flaky tests are observed from builds of merged PRs. > The GitHub Actions logs will be checked for builds where the SHA of the > head of the PR matches the SHA which got merged. > This ensures that all found exceptions are real flakes, since no changes > were made to the PR to make the tests pass later > so that the PR was merged successfully. > > Here are the most flaky test methods: > Test method nameNumber of build failures due to this test > org.apache.pulsar.tests.integration.functions.java.PulsarFunctionsJavaThreadTest.testJavaLoggingFunction > 32 > org.apache.pulsar.client.impl.MessageImplTest.testMessageBrokerAndEntryMetadataTimestampMissed > 18 > org.apache.pulsar.client.impl.MultiTopicsConsumerImplTest.testParallelSubscribeAsync > 12 > org.apache.pulsar.functions.worker.PulsarFunctionTlsTest.tearDown 10 > org.apache.pulsar.client.impl.BinaryProtoLookupServiceTest.maxLookupRedirectsTest1 > 8 > org.apache.pulsar.tests.integration.functions.python.PulsarFunctionsPythonProcessTest.testPythonExclamationFunction >8 > org.apache.pulsar.broker.admin.PersistentTopicsTest.testTriggerCompactionTopic > 7 > org.apache.pulsar.broker.service.RackAwareTest.testRackUpdate 6 > org.apache.pulsar.metadata.bookkeeper.PulsarLedgerIdGeneratorTest.testGenerateLedgerId > 5 > org.apache.pulsar.broker.service.RackAwareTest.testPlacement5 > org.apache.pulsar.client.impl.ClientCnxTest.testClientCnxTimeout5 > org.apache.pulsar.tests.integration.functions.PulsarStateTest.testPythonWordCountFunction > 5 > org.apache.pulsar.metadata.ZKSessionTest.testSessionLost4 > org.apache.pulsar.tests.integration.io.sources.debezium.PulsarDebeziumOracleSourceTest.testDebeziumOracleDbSource > 4 > org.apache.pulsar.client.impl.ConnectionTimeoutTest.testLowTimeout 4 > org.apache.pulsar.client.impl.ProducerCloseTest.brokerCloseTopicTest3 > > Markdown formatted summary reports for each test class can be accessed at > > https://github.com/lhotari/pulsar-flakes/tree/master/2022-05-26-to-2022-06-02 > The summary report links are now available in the Google sheet > > https://docs.google.com/spreadsheets/d/165FHpHjs5fHccSsmQM4beeg6brn-zfUjcrXf6xAu4yQ/edit?usp=sharing > > We need more help in addressing the flaky tests. Please join the efforts > so that we can get CI to a more stable state. > > To coordinate the work, > 1) please search for an existing issues or search for all flaky issues with > "flaky" or the test class name (without package) in the search: > > https://github.com/apache/pulsar/issues?q=is%3Aopen+flaky+sort%3Aupdated-desc > 2) If there isn't an issue for a particular flaky test failure that you'd > like to fix, please create an issue using the "Flaky test" template at > https://github.com/apache/pulsar/issues/new/choose > 3) Please comment on the issue that you are working on it. > > We have a few active contributors working on the flaky tests, thanks for > the contributions. > > I'm looking forward to more contributors joining the efforts. Please join > the #testing channel on Slack if you'd like to ask questions and tips about > reproducing flaky tests locally and how to fix them. > Sharing stories about fixing flaky tests is also helpful for sharing the > knowledge about how flaky tests can be fixed. That's also a valuable way to > contribute. > Some flaky tests might be actual real production code bugs. Fixing > the flaky test might result in fixing a real production code bug. > > Current contributors, please keep up the good work! > New contributors, you are welcome to join the efforts! You will learn > about Pulsar and its internals as a side effect. If you'd love to learn > Pulsar internals and Pulsar OSS development, start by fixing flaky tests. :) > > BR, -Lari >
[GitHub] [pulsar-client-node] thomasechen commented on pull request #81: Support seting topic schema when creating producers/consumers
thomasechen commented on PR #81: URL: https://github.com/apache/pulsar-client-node/pull/81#issuecomment-1152851431 @sijie Anu update on this request and it is really important to the world because numbers of nodejs developmer become more amd more -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org