[DISCUSS] [PIP-160] Batch writing ledger for transaction operation

2022-06-10 Thread Yubiao Feng
Hi Pulsar community:

I open a pip to discuss "Batch writing ledger for transaction operation"

Proposal Link: https://github.com/apache/pulsar/issues/15370

## Motivation

Before reading the background, I suggest you read section “Transaction
Flow” of [PIP-31: Transactional Streaming](
https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx
)

###  Normal Flow vs. Transaction Flow 
![MG3](
https://user-images.githubusercontent.com/25195800/172985866-25e496a4-ea93-42ec-aa0d-e6a02aa0635e.jpeg
)
In *Figure 1. Normal Flow vs. Transaction Flow*:
- The gray square boxes represent logical components.
- All the blue boxes represent logs. The logs are usually Managed ledger
- Each arrow represents the request flow or message flow. These operations
occur in sequence indicated by the numbers next to each arrow.
- The black arrows indicate those shared by transaction and normal flow.
- The blue arrows represent normal-message-specific flow.
- The orange arrows represent transaction-message-specific flow.
- The sections below are numbered to match the operations showed in the
diagram(differ from [PIP-31: Transactional Streaming](
https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx
))


 2.4a Write logs to ledger which Acknowledgement State is PENDING_ACK
[Acknowledgement State Machine](
https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#bookmark=id.4bikq6sjiy8u)
tells about the changes of the Acknowledge State and why we need persistent
“The Log which the Acknowledgement State is PENDING_ACK”.
 2.4a’ Mark messages is no longer useful with current subscription
Update `Cursor` to mark the messages as DELETED. So they can be deleted.
 3.2b Mark messages is no longer useful with current subscription
The implementation here is exactly the same as 2.4a’, except that the
execution is triggered later, after the Transaction has been committed.


### Analyze the performance cost of transaction
As you can see Figure 1. Normal Flow vs.
Transaction Flow]: 2.4a 'and 3.2b are exactly the same logic, so the
remaining orange arrows are the additional performance overhead of all
transactions.
In terms of whether or not each transaction is executed multiple times, we
can split the flow into two classes(Optimizing a process that is executed
multiple times will yield more benefits):
- Executed once each transaction: flow-1.x and flow-3.x
- Executed multiple times each transaction: flow-2.x

So optimizing the flow 2.x with a lot of execution is a good choice. Let's
split flow-2.x into two groups: those that cost more and those that cost
less:
- No disk written: flow-2.1 and fow-2.3
- Disk written: fow-2.1a, fow-2.3a, flow-2.4a

>From the previous analysis, we found that optimizing flow-2.1a, flow-2.3a,
flow-2.4a would bring the most benefits, and batch writes would be an
excellent solution for multiple disk writes. Flow-2.1a and Flow-2.3a are
both manipulations written into the transaction log, we can combine them in
one batch; 2.4a is the operation of writing pending ACK log, we combine
multiple 2.4a's into one batch for processing.
As we can see from “Transaction Flow” of [PIP-31: Transactional Streaming](
https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx),
these instructions are strictly sequential (guaranteed by the client):
- flow-1.x end before flow-2.x start
- flow-2.x end before flow-3.x start
- flow-3.1a end before flow-3.3a start

Therefore, the broker does not need to worry about the dependency of these
flows, we can also put flow-1a flow-31a and flow-3.3a into The Transaction
Log Batch too.

## Goal
Provide a mechanism for Transaction Log Store and Pending Ack Store: accept
multiple write requests, buffer all those records, and persist to a single
BK entry(aka “Batched Entry”). This will improve broker->BK throughput.
- Allow users to specify control of the max size, max record of The Buffer.
- Allow users to specify control max delay time of The Write Request.
- Multiple raw data can be recovered from Batched Entry.
- Configurable “batched implementation” and “common implementation” switch.

## Approach
### Buffer requests and write Bookie
Create a new protobuf record called “Batched Transaction Data” with an
array inside. When receive a request, we put it in the array.

Request:
```
[Request 1]{ data, callback }
[Request 2]]{ data, callback }
…
…
[Request N]]{ data, callback }
```
Buffer:
```
[BatchedTransactionData]{ list=[Request 1, Request 2 … Request N] }
```
Write Bookie:
```
LedgerHandle async write ( BatchedTransactionData to byteBuf )
LedgerHandle callback: ledgerId=1, entryId=1
```
Request-Callback:
```
Callback 1: {ledgerId=1, entryId=1, batchSize=N, batchIndex=0}
Callback 2: {ledgerId=1, entryId=1, batchSize=N, batchIndex=1}
…
…
Callback N: {ledgerId=1, entryId=1, batchSize=N, batchIndex=N-1}
```

### Delet

Re: [DISCUSS] [PIP-160] Batch writing ledger for transaction operation

2022-06-10 Thread Enrico Olivelli
I have read the PIP, and overall I agree with the design.
Good work !

I am not sure I understand the part of making it configurable via a classname.
I believe it is better to simply have a flag "transactionEnableBatchWrites".
Otherwise the matrix of possible implementations will grow without limits.

Enrico

Il giorno ven 10 giu 2022 alle ore 11:35 Yubiao Feng
 ha scritto:
>
> Hi Pulsar community:
>
> I open a pip to discuss "Batch writing ledger for transaction operation"
>
> Proposal Link: https://github.com/apache/pulsar/issues/15370
>
> ## Motivation
>
> Before reading the background, I suggest you read section “Transaction
> Flow” of [PIP-31: Transactional Streaming](
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx
> )
>
> ###  Normal Flow vs. Transaction Flow 
> ![MG3](
> https://user-images.githubusercontent.com/25195800/172985866-25e496a4-ea93-42ec-aa0d-e6a02aa0635e.jpeg
> )
> In *Figure 1. Normal Flow vs. Transaction Flow*:
> - The gray square boxes represent logical components.
> - All the blue boxes represent logs. The logs are usually Managed ledger
> - Each arrow represents the request flow or message flow. These operations
> occur in sequence indicated by the numbers next to each arrow.
> - The black arrows indicate those shared by transaction and normal flow.
> - The blue arrows represent normal-message-specific flow.
> - The orange arrows represent transaction-message-specific flow.
> - The sections below are numbered to match the operations showed in the
> diagram(differ from [PIP-31: Transactional Streaming](
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx
> ))
>
>
>  2.4a Write logs to ledger which Acknowledgement State is PENDING_ACK
> [Acknowledgement State Machine](
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#bookmark=id.4bikq6sjiy8u)
> tells about the changes of the Acknowledge State and why we need persistent
> “The Log which the Acknowledgement State is PENDING_ACK”.
>  2.4a’ Mark messages is no longer useful with current subscription
> Update `Cursor` to mark the messages as DELETED. So they can be deleted.
>  3.2b Mark messages is no longer useful with current subscription
> The implementation here is exactly the same as 2.4a’, except that the
> execution is triggered later, after the Transaction has been committed.
>
>
> ### Analyze the performance cost of transaction
> As you can see Figure 1. Normal Flow vs.
> Transaction Flow]: 2.4a 'and 3.2b are exactly the same logic, so the
> remaining orange arrows are the additional performance overhead of all
> transactions.
> In terms of whether or not each transaction is executed multiple times, we
> can split the flow into two classes(Optimizing a process that is executed
> multiple times will yield more benefits):
> - Executed once each transaction: flow-1.x and flow-3.x
> - Executed multiple times each transaction: flow-2.x
>
> So optimizing the flow 2.x with a lot of execution is a good choice. Let's
> split flow-2.x into two groups: those that cost more and those that cost
> less:
> - No disk written: flow-2.1 and fow-2.3
> - Disk written: fow-2.1a, fow-2.3a, flow-2.4a
>
> From the previous analysis, we found that optimizing flow-2.1a, flow-2.3a,
> flow-2.4a would bring the most benefits, and batch writes would be an
> excellent solution for multiple disk writes. Flow-2.1a and Flow-2.3a are
> both manipulations written into the transaction log, we can combine them in
> one batch; 2.4a is the operation of writing pending ACK log, we combine
> multiple 2.4a's into one batch for processing.
> As we can see from “Transaction Flow” of [PIP-31: Transactional Streaming](
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx),
> these instructions are strictly sequential (guaranteed by the client):
> - flow-1.x end before flow-2.x start
> - flow-2.x end before flow-3.x start
> - flow-3.1a end before flow-3.3a start
>
> Therefore, the broker does not need to worry about the dependency of these
> flows, we can also put flow-1a flow-31a and flow-3.3a into The Transaction
> Log Batch too.
>
> ## Goal
> Provide a mechanism for Transaction Log Store and Pending Ack Store: accept
> multiple write requests, buffer all those records, and persist to a single
> BK entry(aka “Batched Entry”). This will improve broker->BK throughput.
> - Allow users to specify control of the max size, max record of The Buffer.
> - Allow users to specify control max delay time of The Write Request.
> - Multiple raw data can be recovered from Batched Entry.
> - Configurable “batched implementation” and “common implementation” switch.
>
> ## Approach
> ### Buffer requests and write Bookie
> Create a new protobuf record called “Batched Transaction Data” with an
> array inside. When receive a request, we put it in the array.
>
> Request:
> 

Re: [DISCUSS] [PIP-160] Batch writing ledger for transaction operation

2022-06-10 Thread Xiangying Meng
I think this is a nice optimization of transaction internal components.
But I have two little questions:
1. I noticed you write conf transactionMetadataStoreProviderClassName is
configured in broker.conf. Should this be added to `Configuration Changes`?
2. This proposal will add a batch mechanism for pending ack, but this
proposal is no detailed description for pending ack similar to the
description of metadata log batch.

On Fri, Jun 10, 2022 at 7:15 PM Enrico Olivelli  wrote:

> I have read the PIP, and overall I agree with the design.
> Good work !
>
> I am not sure I understand the part of making it configurable via a
> classname.
> I believe it is better to simply have a flag
> "transactionEnableBatchWrites".
> Otherwise the matrix of possible implementations will grow without limits.
>
> Enrico
>
> Il giorno ven 10 giu 2022 alle ore 11:35 Yubiao Feng
>  ha scritto:
> >
> > Hi Pulsar community:
> >
> > I open a pip to discuss "Batch writing ledger for transaction operation"
> >
> > Proposal Link: https://github.com/apache/pulsar/issues/15370
> >
> > ## Motivation
> >
> > Before reading the background, I suggest you read section “Transaction
> > Flow” of [PIP-31: Transactional Streaming](
> >
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx
> > )
> >
> > ###  Normal Flow vs. Transaction Flow
> 
> > ![MG3](
> >
> https://user-images.githubusercontent.com/25195800/172985866-25e496a4-ea93-42ec-aa0d-e6a02aa0635e.jpeg
> > )
> > In *Figure 1. Normal Flow vs. Transaction Flow*:
> > - The gray square boxes represent logical components.
> > - All the blue boxes represent logs. The logs are usually Managed ledger
> > - Each arrow represents the request flow or message flow. These
> operations
> > occur in sequence indicated by the numbers next to each arrow.
> > - The black arrows indicate those shared by transaction and normal flow.
> > - The blue arrows represent normal-message-specific flow.
> > - The orange arrows represent transaction-message-specific flow.
> > - The sections below are numbered to match the operations showed in the
> > diagram(differ from [PIP-31: Transactional Streaming](
> >
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx
> > ))
> >
> >
> >  2.4a Write logs to ledger which Acknowledgement State is PENDING_ACK
> > [Acknowledgement State Machine](
> >
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#bookmark=id.4bikq6sjiy8u
> )
> > tells about the changes of the Acknowledge State and why we need
> persistent
> > “The Log which the Acknowledgement State is PENDING_ACK”.
> >  2.4a’ Mark messages is no longer useful with current subscription
> > Update `Cursor` to mark the messages as DELETED. So they can be deleted.
> >  3.2b Mark messages is no longer useful with current subscription
> > The implementation here is exactly the same as 2.4a’, except that the
> > execution is triggered later, after the Transaction has been committed.
> >
> >
> > ### Analyze the performance cost of transaction
> > As you can see Figure 1. Normal Flow
> vs.
> > Transaction Flow]: 2.4a 'and 3.2b are exactly the same logic, so the
> > remaining orange arrows are the additional performance overhead of all
> > transactions.
> > In terms of whether or not each transaction is executed multiple times,
> we
> > can split the flow into two classes(Optimizing a process that is executed
> > multiple times will yield more benefits):
> > - Executed once each transaction: flow-1.x and flow-3.x
> > - Executed multiple times each transaction: flow-2.x
> >
> > So optimizing the flow 2.x with a lot of execution is a good choice.
> Let's
> > split flow-2.x into two groups: those that cost more and those that cost
> > less:
> > - No disk written: flow-2.1 and fow-2.3
> > - Disk written: fow-2.1a, fow-2.3a, flow-2.4a
> >
> > From the previous analysis, we found that optimizing flow-2.1a,
> flow-2.3a,
> > flow-2.4a would bring the most benefits, and batch writes would be an
> > excellent solution for multiple disk writes. Flow-2.1a and Flow-2.3a are
> > both manipulations written into the transaction log, we can combine them
> in
> > one batch; 2.4a is the operation of writing pending ACK log, we combine
> > multiple 2.4a's into one batch for processing.
> > As we can see from “Transaction Flow” of [PIP-31: Transactional
> Streaming](
> >
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx
> ),
> > these instructions are strictly sequential (guaranteed by the client):
> > - flow-1.x end before flow-2.x start
> > - flow-2.x end before flow-3.x start
> > - flow-3.1a end before flow-3.3a start
> >
> > Therefore, the broker does not need to worry about the dependency of
> these
> > flows, we can also put flow-1a flow-31a and flow-3.3a into The
> Transaction
> > Log Batch too.
> >
> > ## Goal
> > Provide a mechan

[GitHub] [pulsar-manager] daniel-bartz opened a new issue, #464: JWT/TLS authentication issue creating en new Environment (pulsar-manager 0.3.0)

2022-06-10 Thread GitBox


daniel-bartz opened a new issue, #464:
URL: https://github.com/apache/pulsar-manager/issues/464

   When I want to add a new _Environment_ via the web interface, nothing happen 
and I get the following error in the backend logs:
   ```
   Jun 10 14:10:52 hostname pulsar-manager[6494]: 2022-06-10 14:10:52.707 ERROR 
6494 --- [nio-7750-exec-6] o.a.c.c.C.[.[.[/].[dispatcherServlet]: 
Servlet.service() for servlet [dispatcherServlet] in context with path [] threw 
exception [Handler dispatch failed; nested exception is 
java.lang.NoSuchFieldError: id_alg_AEADChaCha20Poly1305] with root cause
   Jun 10 14:10:52 hostname pulsar-manager[6494]: java.lang.NoSuchFieldError: 
id_alg_AEADChaCha20Poly1305
   ```
   
   I built (tried both jdk8 & 11) the _pulsar-manager_ using `./gradlew build 
-x test`
   I tried to run it on both jdk8 & 11 and the same issue appeared.
   
   
   Notes:
   - I was using same approach for _pulsar-manager_ 0.2.0 on jdk11and it worked 
very nicely.
   - I tried to use the binary provided here 
https://archive.apache.org/dist/pulsar/pulsar-manager/pulsar-manager-0.3.0/ but 
could not manage to get it working with PostgreSQL.
   - I also tried to remove the TLS_CHACHA20_POLY1305_SHA256 cipher on the 
broker side but it did not help.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[GitHub] [pulsar-manager] eolivelli commented on issue #464: JWT/TLS authentication issue creating en new Environment (pulsar-manager 0.3.0)

2022-06-10 Thread GitBox


eolivelli commented on issue #464:
URL: https://github.com/apache/pulsar-manager/issues/464#issuecomment-1152639969

   Can you paste the full stack trace please?
   
   It looks like the problem is related only to TLS
   
   Probably there is some third party library to upgrade


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[GitHub] [pulsar-manager] eolivelli commented on issue #464: JWT/TLS authentication issue creating en new Environment (pulsar-manager 0.3.0)

2022-06-10 Thread GitBox


eolivelli commented on issue #464:
URL: https://github.com/apache/pulsar-manager/issues/464#issuecomment-1152640851

   
https://stackoverflow.com/questions/47126562/bouncycastleprovider-throws-java-lang-nosuchfielderror-id-hmacwithsha3-224
   
   
   You can try to upgrade BouncyCastle dependency to fix the problem


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



[GitHub] [pulsar-manager] daniel-bartz commented on issue #464: JWT/TLS authentication issue creating en new Environment (pulsar-manager 0.3.0)

2022-06-10 Thread GitBox


daniel-bartz commented on issue #464:
URL: https://github.com/apache/pulsar-manager/issues/464#issuecomment-1152681071

   Here is the full stacktrace, so you can see I use a rather new BC lib
   ```
   Jun 10 19:46:56 hostname pulsar-manager[49051]: 2022-06-10 19:46:56.586 
ERROR 49051 --- [nio-7750-exec-6] o.a.c.c.C.[.[.[/].[dispatcherServlet]: 
Servlet.service() for servlet [dispatcherServlet] in context with path [] threw 
exception [Handler dispatch failed; nested exception is 
java.lang.NoSuchFieldError: id_alg_AEADChaCha20Poly1305] with root cause
   Jun 10 19:46:56 hostname pulsar-manager[49051]: java.lang.NoSuchFieldError: 
id_alg_AEADChaCha20Poly1305
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
org.bouncycastle.jcajce.provider.symmetric.ChaCha$Mappings.configure(Unknown 
Source) ~[bcprov-ext-jdk15on-1.66.jar:1.66.00.0]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
org.bouncycastle.jce.provider.BouncyCastleProvider.loadAlgorithms(Unknown 
Source) ~[bcprov-ext-jdk15on-1.66.jar:1.66.00.0]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
org.bouncycastle.jce.provider.BouncyCastleProvider.setup(Unknown Source) 
~[bcprov-ext-jdk15on-1.66.jar:1.66.00.0]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
org.bouncycastle.jce.provider.BouncyCastleProvider.access$000(Unknown Source) 
~[bcprov-ext-jdk15on-1.66.jar:1.66.00.0]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
org.bouncycastle.jce.provider.BouncyCastleProvider$1.run(Unknown Source) 
~[bcprov-ext-jdk15on-1.66.jar:1.66.00.0]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_332]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
org.bouncycastle.jce.provider.BouncyCastleProvider.(Unknown Source) 
~[bcprov-ext-jdk15on-1.66.jar:1.66.00.0]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
~[na:1.8.0_332]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 ~[na:1.8.0_332]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 ~[na:1.8.0_332]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_332]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
java.lang.Class.newInstance(Class.java:442) ~[na:1.8.0_332]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
org.apache.pulsar.common.util.SecurityUtility.getBCProviderFromClassPath(SecurityUtility.java:125)
 ~[pulsar-common-2.7.0.jar:2.7.0]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
org.apache.pulsar.common.util.SecurityUtility.getProvider(SecurityUtility.java:102)
 ~[pulsar-common-2.7.0.jar:2.7.0]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
org.apache.pulsar.common.util.SecurityUtility.(SecurityUtility.java:69) 
~[pulsar-common-2.7.0.jar:2.7.0]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.(AsyncHttpConnector.java:152)
 ~[pulsar-client-admin-original-2.7.0.jar:2.7.0]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
org.apache.pulsar.client.admin.internal.http.AsyncHttpConnectorProvider.getConnector(AsyncHttpConnectorProvider.java:50)
 ~[pulsar-client-admin-original-2.7.0.jar:2.7.0]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
org.apache.pulsar.client.admin.PulsarAdmin.(PulsarAdmin.java:200) 
~[pulsar-client-admin-original-2.7.0.jar:2.7.0]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
org.apache.pulsar.client.admin.internal.PulsarAdminBuilderImpl.build(PulsarAdminBuilderImpl.java:46)
 ~[pulsar-client-admin-original-2.7.0.jar:2.7.0]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
org.apache.pulsar.manager.service.impl.PulsarAdminServiceImpl.createPulsarAdmin(PulsarAdminServiceImpl.java:146)
 ~[pulsar-manager-master.jar:na]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
org.apache.pulsar.manager.service.impl.PulsarAdminServiceImpl.getPulsarAdmin(PulsarAdminServiceImpl.java:73)
 ~[pulsar-manager-master.jar:na]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
org.apache.pulsar.manager.service.impl.PulsarAdminServiceImpl.clusters(PulsarAdminServiceImpl.java:83)
 ~[pulsar-manager-master.jar:na]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
org.apache.pulsar.manager.controller.EnvironmentsController.addEnvironment(EnvironmentsController.java:124)
 ~[pulsar-manager-master.jar:na]
   Jun 10 19:46:56 hostname pulsar-manager[49051]: #011at 
org.apache.pulsar.manager.controller.EnvironmentsController$$FastClassBySpringCGLIB$$8f0ec792.invoke()
 ~[pulsar-manager-master.jar:na]
   Jun 10 19:46:56 hostname pul

Re: Pulsar Flaky test report 2022-06-02 for PR builds in CI

2022-06-10 Thread Heesung Sohn
Hi,

Can I get this PR reviewed?

This PR is to make some of the tests less flaky.

https://github.com/apache/pulsar/pull/16011

Thanks,
Heesung



On Thu, Jun 2, 2022 at 5:06 AM Lari Hotari  wrote:

> Dear Pulsar community members,
>
> Here's a report of the flaky tests in Pulsar CI during the observation
> period of 2022-05-26 to 2022-06-02 .
> The full report is available as a Google Sheet,
>
> https://docs.google.com/spreadsheets/d/165FHpHjs5fHccSsmQM4beeg6brn-zfUjcrXf6xAu4yQ/edit?usp=sharing
>
> The report contains a subset of the test failures.
> The flaky tests are observed from builds of merged PRs.
> The GitHub Actions logs will be checked for builds where the SHA of the
> head of the PR matches the SHA which got merged.
> This ensures that all found exceptions are real flakes, since no changes
> were made to the PR to make the tests pass later
> so that the PR was merged successfully.
>
> Here are the most flaky test methods:
> Test method nameNumber of build failures due to this test
> org.apache.pulsar.tests.integration.functions.java.PulsarFunctionsJavaThreadTest.testJavaLoggingFunction
>   32
> org.apache.pulsar.client.impl.MessageImplTest.testMessageBrokerAndEntryMetadataTimestampMissed
> 18
> org.apache.pulsar.client.impl.MultiTopicsConsumerImplTest.testParallelSubscribeAsync
>   12
> org.apache.pulsar.functions.worker.PulsarFunctionTlsTest.tearDown   10
> org.apache.pulsar.client.impl.BinaryProtoLookupServiceTest.maxLookupRedirectsTest1
> 8
> org.apache.pulsar.tests.integration.functions.python.PulsarFunctionsPythonProcessTest.testPythonExclamationFunction
>8
> org.apache.pulsar.broker.admin.PersistentTopicsTest.testTriggerCompactionTopic
> 7
> org.apache.pulsar.broker.service.RackAwareTest.testRackUpdate   6
> org.apache.pulsar.metadata.bookkeeper.PulsarLedgerIdGeneratorTest.testGenerateLedgerId
> 5
> org.apache.pulsar.broker.service.RackAwareTest.testPlacement5
> org.apache.pulsar.client.impl.ClientCnxTest.testClientCnxTimeout5
> org.apache.pulsar.tests.integration.functions.PulsarStateTest.testPythonWordCountFunction
>  5
> org.apache.pulsar.metadata.ZKSessionTest.testSessionLost4
> org.apache.pulsar.tests.integration.io.sources.debezium.PulsarDebeziumOracleSourceTest.testDebeziumOracleDbSource
>  4
> org.apache.pulsar.client.impl.ConnectionTimeoutTest.testLowTimeout  4
> org.apache.pulsar.client.impl.ProducerCloseTest.brokerCloseTopicTest3
>
> Markdown formatted summary reports for each test class can be accessed at
>
> https://github.com/lhotari/pulsar-flakes/tree/master/2022-05-26-to-2022-06-02
> The summary report links are now available in the Google sheet
>
> https://docs.google.com/spreadsheets/d/165FHpHjs5fHccSsmQM4beeg6brn-zfUjcrXf6xAu4yQ/edit?usp=sharing
>
> We need more help in addressing the flaky tests. Please join the efforts
> so that we can get CI to a more stable state.
>
> To coordinate the work,
> 1) please search for an existing issues or search for all flaky issues with
> "flaky" or the test class name (without package) in the search:
>
> https://github.com/apache/pulsar/issues?q=is%3Aopen+flaky+sort%3Aupdated-desc
> 2) If there isn't an issue for a particular flaky test failure that you'd
> like to fix, please create an issue using the "Flaky test" template at
> https://github.com/apache/pulsar/issues/new/choose
> 3) Please comment on the issue that you are working on it.
>
> We have a few active contributors working on the flaky tests, thanks for
> the contributions.
>
> I'm looking forward to more contributors joining the efforts. Please join
> the #testing channel on Slack if you'd like to ask questions and tips about
> reproducing flaky tests locally and how to fix them.
> Sharing stories about fixing flaky tests is also helpful for sharing the
> knowledge about how flaky tests can be fixed. That's also a valuable way to
> contribute.
> Some flaky tests might be actual real production code bugs. Fixing
> the flaky test might result in fixing a real production code bug.
>
> Current contributors, please keep up the good work!
> New contributors, you are welcome to join the efforts! You will learn
> about Pulsar and its internals as a side effect.  If you'd love to learn
> Pulsar internals and Pulsar OSS development, start by fixing flaky tests. :)
>
> BR, -Lari
>


[GitHub] [pulsar-client-node] thomasechen commented on pull request #81: Support seting topic schema when creating producers/consumers

2022-06-10 Thread GitBox


thomasechen commented on PR #81:
URL: 
https://github.com/apache/pulsar-client-node/pull/81#issuecomment-1152851431

   @sijie Anu update on this request and it is really important to the world 
because numbers of nodejs developmer become more amd more


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org