date:20220616

[GitHub] [pulsar-client-node] nkurihar commented on pull request #217: Suggest pulsar-admin-client in README.md

2022-06-16 Thread GitBox



nkurihar commented on PR #217:
URL: 
https://github.com/apache/pulsar-client-node/pull/217#issuecomment-1157335037

   Thank you for your contribution.
   
   However, IMO we sholud clarify whether the library is made by the third 
party or not to avoid users' misunderstanding and confusion. 
   
   That's why I think it's nice to put the document on 
https://pulsar.apache.org/docs/next/client-libraries#third-party-clients (which 
is also suggested at 
https://github.com/apache/pulsar/pull/15839#pullrequestreview-1005282792 ) 
instead of README in pulsar-client-node.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-node] visortelle commented on pull request #217: Suggest pulsar-admin-client in README.md

2022-06-16 Thread GitBox



visortelle commented on PR #217:
URL: 
https://github.com/apache/pulsar-client-node/pull/217#issuecomment-1157342139

   Thank you. I'll create a PR at the suggested place.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-node] visortelle closed pull request #217: Suggest pulsar-admin-client in README.md

2022-06-16 Thread GitBox



visortelle closed pull request #217: Suggest pulsar-admin-client in README.md
URL: https://github.com/apache/pulsar-client-node/pull/217


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-node] nearzxide10 commented on issue #87: npm install fails on MacOS

2022-06-16 Thread GitBox

nearzxide10 commented on issue #87:
URL:
https://github.com/apache/pulsar-client-node/issues/87#issuecomment-1157411697

nodeJs version : 16.15.1
npm version : 8.11.0
mac OS: 12.4 (M1 max)

I installed the cpp client first and export PULSAR_CPP_DIR.. can't npm
install too

```
npm ERR! code 1
npm ERR! path
/Users/jackliu/Project/pulsar/pulsar_test/node_modules/pulsar-client
npm ERR! command failed
npm ERR! command sh -c node-pre-gyp install --fallback-to-build
npm ERR! CC(target) Release/obj.target/nothing/../node-addon-api/nothing.o
npm ERR! LIBTOOL-STATIC Release/nothing.a
npm ERR! CXX(target) Release/obj.target/Pulsar/src/addon.o
npm ERR! CXX(target) Release/obj.target/Pulsar/src/Message.o
npm ERR! CXX(target) Release/obj.target/Pulsar/src/MessageId.o
npm ERR! CXX(target) Release/obj.target/Pulsar/src/Authentication.o
npm ERR! CXX(target) Release/obj.target/Pulsar/src/Client.o
npm ERR! CXX(target) Release/obj.target/Pulsar/src/Producer.o
npm ERR! CXX(target) Release/obj.target/Pulsar/src/ProducerConfig.o
npm ERR! CXX(target) Release/obj.target/Pulsar/src/Consumer.o
npm ERR! CXX(target) Release/obj.target/Pulsar/src/ConsumerConfig.o
npm ERR! CXX(target) Release/obj.target/Pulsar/src/Reader.o
npm ERR! CXX(target) Release/obj.target/Pulsar/src/ReaderConfig.o
npm ERR! SOLINK_MODULE(target) Release/Pulsar.node
npm ERR! Failed to execute
'/Users/jackliu/.nvm/versions/node/v16.15.1/bin/node
/Users/jackliu/.nvm/versions/node/v16.15.1/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js
build --fallback-to-build
--module=/Users/jackliu/Project/pulsar/pulsar_test/node_modules/pulsar-client/build/Release/libpulsar.node
--module_name=libpulsar
--module_path=/Users/jackliu/Project/pulsar/pulsar_test/node_modules/pulsar-client/build/Release
--napi_version=8 --node_abi_napi=napi --napi_build_version=0
--node_napi_label=node-v93' (1)
npm ERR! node-pre-gyp info it worked if it ends with ok
npm ERR! node-pre-gyp info using node-pre-gyp@1.0.9
npm ERR! node-pre-gyp info using node@16.15.1 | darwin | arm64
npm ERR! node-pre-gyp info check checked for
"/Users/jackliu/Project/pulsar/pulsar_test/node_modules/pulsar-client/build/Release/libpulsar.node"
(not found)
npm ERR! node-pre-gyp http GET
https://pulsar.apache.org/docs/en/client-libraries-cpp/libpulsar-v1.6.2-node-v93-darwin-arm64.tar.gz
npm ERR! node-pre-gyp ERR! install response status 404 Not Found on
https://pulsar.apache.org/docs/en/client-libraries-cpp/libpulsar-v1.6.2-node-v93-darwin-arm64.tar.gz
npm ERR! node-pre-gyp WARN Pre-built binaries not installable for
pulsar-client@1.6.2 and node@16.15.1 (node-v93 ABI, unknown) (falling back to
source compile with node-gyp)
npm ERR! node-pre-gyp WARN Hit error response status 404 Not Found on
https://pulsar.apache.org/docs/en/client-libraries-cpp/libpulsar-v1.6.2-node-v93-darwin-arm64.tar.gz
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using node-gyp@9.0.0
npm ERR! gyp info using node@16.15.1 | darwin | arm64
npm ERR! gyp info ok
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using node-gyp@9.0.0
npm ERR! gyp info using node@16.15.1 | darwin | arm64
npm ERR! gyp info find Python using Python version 3.9.13 found at
"/opt/homebrew/opt/python@3.9/bin/python3.9"
npm ERR! gyp info spawn /opt/homebrew/opt/python@3.9/bin/python3.9
npm ERR! gyp info spawn args [
npm ERR! gyp info spawn args
'/Users/jackliu/.nvm/versions/node/v16.15.1/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py',
npm ERR! gyp info spawn args 'binding.gyp',
npm ERR! gyp info spawn args '-f',
npm ERR! gyp info spawn args 'make',
npm ERR! gyp info spawn args '-I',
npm ERR! gyp info spawn args
'/Users/jackliu/Project/pulsar/pulsar_test/node_modules/pulsar-client/build/config.gypi',
npm ERR! gyp info spawn args '-I',
npm ERR! gyp info spawn args
'/Users/jackliu/.nvm/versions/node/v16.15.1/lib/node_modules/npm/node_modules/node-gyp/addon.gypi',
npm ERR! gyp info spawn args '-I',
npm ERR! gyp info spawn args
'/Users/jackliu/Library/Caches/node-gyp/16.15.1/include/node/common.gypi',
npm ERR! gyp info spawn args '-Dlibrary=shared_library',
npm ERR! gyp info spawn args '-Dvisibility=default',
npm ERR! gyp info spawn args
'-Dnode_root_dir=/Users/jackliu/Library/Caches/node-gyp/16.15.1',
npm ERR! gyp info spawn args
'-Dnode_gyp_dir=/Users/jackliu/.nvm/versions/node/v16.15.1/lib/node_modules/npm/node_modules/node-gyp',
npm ERR! gyp info spawn args
'-Dnode_lib_file=/Users/jackliu/Library/Caches/node-gyp/16.15.1/<(target_arch)/node.lib',
npm ERR! gyp info spawn args
'-Dmodule_root_dir=/Users/jackliu/Project/pulsar/pulsar_test/node_modules/pulsar-client',
npm ERR! gyp info spawn args '-Dnode_engine=v8',
npm ERR! gyp info spawn args '--depth=.',

Re: Mark delete rate limiter bug?

2022-06-16 Thread Asaf Mesika

If I understand correctly:

this.flushCursorsTask =
scheduledExecutor.scheduleAtFixedRate(catchingAndLoggingThrowables(this::flushCursors),
config.getCursorPositionFlushSeconds(),
config.getCursorPositionFlushSeconds(), TimeUnit.SECONDS);

which calls

ml.getCursors().forEach(c -> ((ManagedCursorImpl) c).flush());


which calls

asyncMarkDelete(lastMarkDeleteEntry.newPosition,
lastMarkDeleteEntry.properties, new MarkDeleteCallback() {


which gets to here

// Apply rate limiting to mark-delete operations
if (markDeleteLimiter != null && !markDeleteLimiter.tryAcquire()) {
isDirty = true;
updateLastMarkDeleteEntryToLatest(newPosition, properties);
callback.markDeleteComplete(ctx);
return;
}
internalAsyncMarkDelete(newPosition, properties, callback, ctx);


So you end up in the rate limiter check even if you come via the scheduled
route, no?


On Wed, Jun 15, 2022 at 9:11 PM Matteo Merli  wrote:

> There is a background flush process that was added for this precise reason.
>
> In broker.conf
> 
> # How frequently to flush the cursor positions that were accumulated
> due to rate limiting. (seconds).
> # Default is 60 seconds
> managedLedgerCursorPositionFlushSeconds=60
>
> --
> Matteo Merli
> 
>
> On Wed, Jun 15, 2022 at 8:45 AM Asaf Mesika  wrote:
> >
> > Hi,
> >
> > In the Managed Cursor implementation, I saw that the position state is
> > persisted in the ledger whenever a user calls acknowledge, be it
> individual
> > or cumulative.
> >
> > For performance reasons there is a rate limiter, defaulting to 1 persist
> > action per second.
> >
> > The bug:
> > 17:00:00.003 - user ack a single message --> state it written to ledger
> > 17:00:00.004 - 17:00:00.400 - use acks a single message 100 times.
> > nothing happens after that on that subscription.
> >
> > 18:00 - machine crashed.
> > You "lost" the state hence 100 messages have to be redelivered.
> >
> > I was wondering if it makes sense to add a timeout - say 10sec. If
> nothing
> > happens to the subscription, flush the state after 10sec if it's dirty.
> >
> > WDYT?
>

Re: [DISCUSS] [PIP-160] Batch writing ledger for transaction operation

2022-06-16 Thread Yubiao Feng

Hi Pulsar community:

I've rewritten proposal: "Make transactions work more efficiently by
aggregation operation for transaction log and pending ack store"

Proposal Link: https://github.com/apache/pulsar/issues/15370

## Motivation

Transaction coordinator and Pending Ack Store are two core components in
Pulsar transactions. For how they work in Pulsar transactions, see [PIP-31:
Transactional Streaming](
https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx
)

Both transaction coordinator and pending ack store use the managed ledger
to record data.The transaction coordinator records the transaction metadata
in a managed ledger(aka transaction log) to achieve transaction durability.
And the pending ack store records all the message ack operations to
guarantee the durability of message ack operations.

Currently, the transaction coordinator will append a new entry to the
transaction log for each transaction metadata/state change. For example,
for the life cycle of a transaction, both of the open transaction, add
produced partition to a transaction, add consumed subscription to a
transaction, and commit the transaction will append to the transaction log.
If there are many parallel transactions, the number of entries added to the
transaction log will also increase.

The pending ack store is similar to the transaction coordinator. Too many
parallel message acks will add more entries to the managed ledger used by
the pending ack store.

So this proposal tries to make the transaction work more efficiently by
adding aggregation operations for transaction log and pending ack store.

## Goal
Provide a mechanism allowing the Transaction Log Store and Pending Ack
Store to aggregate multiple records into a batched record and persist into
a single BK entry. This will make Pulsar transactions work more efficiently.

- Reduce the number of entries written to Bookies
- Reduce the entry indexes size of Bookie
- Reduce the number of entries read from the bookie

The new mechanism will provide the following ability to control the batch.

Adjustable thresholds: trigger BookKeeper-write when reaching any one of
the following conditions
- Max size (bytes)
- Max records count
- Max delay time
- Allow users to enable or disable batch-write.
- Dynamic enable or disable the feature.

## Approach
### Aggregate multiple records and write to Bookies
We will create a Container(aka TxLogBufferedWriter ) to buffer requests and
flush to Managed Ledger. Transaction Log Store And Pending Ack Store will
no longer write to Managed Ledger directly, Change to using
TxLogBufferedWriter to write Ledger data. The TxLogBufferedWriter caches
“write requests” for a certain number(or a certain size of request data)
and then writes them to the Managed Ledger in one go. After Managed Ledger
has written complete, The TxLogBufferedWriter responds to each
request-caller. In this process, Managed Ledger doesn't care how many
records(or what to be written) in the Entry, it just treats them as a
single block of data.

The first write-request by transaction components that write to
TxLogBufferedWriter will take a long time to receive a response because The
TxLogBufferedWriter must wait for subsequent requests to accumulate enough
data to actually start writing to the Managed Ledger. To control the
maximum latency, The TxLogBufferedWriter will mark the first request time
for each batch, and additional timing triggers writes.

The TxLogBufferedWriter does not guarantee that logs in the batch are all
from the same transaction. Instead, it does not matter which transaction
the request belongs to. One batch corresponds to one Managed Ledger
addEntryOp, so the Entry looks like the below:
![截屏2022-06-16 22 35 33](
https://user-images.githubusercontent.com/25195800/174094211-e2145887-c5fd-4cca-a4ab-a5842b92e0ef.png
)

When Transaction completes(committed or aborted), all Transaction log
records and Pending ack log records of this transaction will be
deleted(marked acknowledgment). To ensure that users can delete transaction
records accurately, So when callback to “write request”, The
TxLogBufferedWriter should tell “position(Entry position)”, “how many
requests there are in a batch( aka batchSize )” and “the location of that
record inside the batch aka batchIndex)” to the caller.

After enabling this feature, some data is non-batch, and some data is batch
which is in Ledger. When batch-data is read, it needs to specifically parse
the data that Transaction Components can process. So we append a magic num
in front of the Batched Log Data to confirm that it is a batched data and
to ensure the future scalability, we will add a version identifier in front
of the original Batched Log Data, just like this:

```
// combine multi records using protobuf record (defined below)
[Record0, Record1, … RecordN ] ⇒ [Records]
// append magic num and version at front of Batched Log (outside protobuf
serialization)
// Magic Num 2 byt

[GitHub] [pulsar-manager] toneill818 opened a new issue, #465: Docker Failed to Initialize Postgresql Database

2022-06-16 Thread GitBox



toneill818 opened a new issue, #465:
URL: https://github.com/apache/pulsar-manager/issues/465

   When running the docker container the following is produced in the logs.
   
   ```
   /pulsar-manager/startup.sh: 21: initdb: not found
   /pulsar-manager/startup.sh: 22: pg_ctl: not found
   createdb: error: could not connect to database template1: could not connect 
to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
   psql: error: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Full Name []:   Room Number []: Work Phone []:  Home Phone []:  
Other []: Is the information correct? [Y/n] Starting Pulsar Manager Front end
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] [PIP-160] Batch writing ledger for transaction operation

2022-06-16 Thread Yubiao Feng

> I am not sure I understand the part of making it configurable via a
classname.
I believe it is better to simply have a flag "transactionEnableBatchWrites".
Otherwise the matrix of possible implementations will grow without limits.

Good idea, I've modified the design and added a switch in the Configure
Changes section.

On Fri, Jun 10, 2022 at 7:14 PM Enrico Olivelli  wrote:

> I have read the PIP, and overall I agree with the design.
> Good work !
>
> I am not sure I understand the part of making it configurable via a
> classname.
> I believe it is better to simply have a flag
> "transactionEnableBatchWrites".
> Otherwise the matrix of possible implementations will grow without limits.
>
> Enrico
>
> Il giorno ven 10 giu 2022 alle ore 11:35 Yubiao Feng
>  ha scritto:
> >
> > Hi Pulsar community:
> >
> > I open a pip to discuss "Batch writing ledger for transaction operation"
> >
> > Proposal Link: https://github.com/apache/pulsar/issues/15370
> >
> > ## Motivation
> >
> > Before reading the background, I suggest you read section “Transaction
> > Flow” of [PIP-31: Transactional Streaming](
> >
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx
> > )
> >
> > ###  Normal Flow vs. Transaction Flow
> 
> > ![MG3](
> >
> https://user-images.githubusercontent.com/25195800/172985866-25e496a4-ea93-42ec-aa0d-e6a02aa0635e.jpeg
> > )
> > In *Figure 1. Normal Flow vs. Transaction Flow*:
> > - The gray square boxes represent logical components.
> > - All the blue boxes represent logs. The logs are usually Managed ledger
> > - Each arrow represents the request flow or message flow. These
> operations
> > occur in sequence indicated by the numbers next to each arrow.
> > - The black arrows indicate those shared by transaction and normal flow.
> > - The blue arrows represent normal-message-specific flow.
> > - The orange arrows represent transaction-message-specific flow.
> > - The sections below are numbered to match the operations showed in the
> > diagram(differ from [PIP-31: Transactional Streaming](
> >
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx
> > ))
> >
> >
> >  2.4a Write logs to ledger which Acknowledgement State is PENDING_ACK
> > [Acknowledgement State Machine](
> >
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#bookmark=id.4bikq6sjiy8u
> )
> > tells about the changes of the Acknowledge State and why we need
> persistent
> > “The Log which the Acknowledgement State is PENDING_ACK”.
> >  2.4a’ Mark messages is no longer useful with current subscription
> > Update `Cursor` to mark the messages as DELETED. So they can be deleted.
> >  3.2b Mark messages is no longer useful with current subscription
> > The implementation here is exactly the same as 2.4a’, except that the
> > execution is triggered later, after the Transaction has been committed.
> >
> >
> > ### Analyze the performance cost of transaction
> > As you can see Figure 1. Normal Flow
> vs.
> > Transaction Flow]: 2.4a 'and 3.2b are exactly the same logic, so the
> > remaining orange arrows are the additional performance overhead of all
> > transactions.
> > In terms of whether or not each transaction is executed multiple times,
> we
> > can split the flow into two classes(Optimizing a process that is executed
> > multiple times will yield more benefits):
> > - Executed once each transaction: flow-1.x and flow-3.x
> > - Executed multiple times each transaction: flow-2.x
> >
> > So optimizing the flow 2.x with a lot of execution is a good choice.
> Let's
> > split flow-2.x into two groups: those that cost more and those that cost
> > less:
> > - No disk written: flow-2.1 and fow-2.3
> > - Disk written: fow-2.1a, fow-2.3a, flow-2.4a
> >
> > From the previous analysis, we found that optimizing flow-2.1a,
> flow-2.3a,
> > flow-2.4a would bring the most benefits, and batch writes would be an
> > excellent solution for multiple disk writes. Flow-2.1a and Flow-2.3a are
> > both manipulations written into the transaction log, we can combine them
> in
> > one batch; 2.4a is the operation of writing pending ACK log, we combine
> > multiple 2.4a's into one batch for processing.
> > As we can see from “Transaction Flow” of [PIP-31: Transactional
> Streaming](
> >
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx
> ),
> > these instructions are strictly sequential (guaranteed by the client):
> > - flow-1.x end before flow-2.x start
> > - flow-2.x end before flow-3.x start
> > - flow-3.1a end before flow-3.3a start
> >
> > Therefore, the broker does not need to worry about the dependency of
> these
> > flows, we can also put flow-1a flow-31a and flow-3.3a into The
> Transaction
> > Log Batch too.
> >
> > ## Goal
> > Provide a mechanism for Transaction Log Store and Pending Ack Store:
> accept
> > multiple write requests, buffer all t

Re: [DISCUSS] [PIP-160] Batch writing ledger for transaction operation

2022-06-16 Thread Yubiao Feng

Hi XiangYing
>  I noticed you write conf transactionMetadataStoreProviderClassName is
configured in broker.conf. Should this be added to `Configuration Changes`?

I have rewritten the design, in the new design I removed the section which
has "transactionMetadataStoreProviderClassName", instead it, I write
new sections: "Upgrade" and "Downgrade".

> This proposal will add a batch mechanism for pending ack, but this
proposal is no detailed description for pending ack similar to the
description of metadata log batch.

In the new design, I have appended "detail of pending ack" at section
"Acknowledge entry with the aggregated records".


On Fri, Jun 10, 2022 at 9:19 PM Xiangying Meng  wrote:

> I think this is a nice optimization of transaction internal components.
> But I have two little questions:
> 1. I noticed you write conf transactionMetadataStoreProviderClassName is
> configured in broker.conf. Should this be added to `Configuration Changes`?
> 2. This proposal will add a batch mechanism for pending ack, but this
> proposal is no detailed description for pending ack similar to the
> description of metadata log batch.
>
> On Fri, Jun 10, 2022 at 7:15 PM Enrico Olivelli 
> wrote:
>
> > I have read the PIP, and overall I agree with the design.
> > Good work !
> >
> > I am not sure I understand the part of making it configurable via a
> > classname.
> > I believe it is better to simply have a flag
> > "transactionEnableBatchWrites".
> > Otherwise the matrix of possible implementations will grow without
> limits.
> >
> > Enrico
> >
> > Il giorno ven 10 giu 2022 alle ore 11:35 Yubiao Feng
> >  ha scritto:
> > >
> > > Hi Pulsar community:
> > >
> > > I open a pip to discuss "Batch writing ledger for transaction
> operation"
> > >
> > > Proposal Link: https://github.com/apache/pulsar/issues/15370
> > >
> > > ## Motivation
> > >
> > > Before reading the background, I suggest you read section “Transaction
> > > Flow” of [PIP-31: Transactional Streaming](
> > >
> >
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx
> > > )
> > >
> > > ###  Normal Flow vs. Transaction Flow
> > 
> > > ![MG3](
> > >
> >
> https://user-images.githubusercontent.com/25195800/172985866-25e496a4-ea93-42ec-aa0d-e6a02aa0635e.jpeg
> > > )
> > > In *Figure 1. Normal Flow vs. Transaction Flow*:
> > > - The gray square boxes represent logical components.
> > > - All the blue boxes represent logs. The logs are usually Managed
> ledger
> > > - Each arrow represents the request flow or message flow. These
> > operations
> > > occur in sequence indicated by the numbers next to each arrow.
> > > - The black arrows indicate those shared by transaction and normal
> flow.
> > > - The blue arrows represent normal-message-specific flow.
> > > - The orange arrows represent transaction-message-specific flow.
> > > - The sections below are numbered to match the operations showed in the
> > > diagram(differ from [PIP-31: Transactional Streaming](
> > >
> >
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx
> > > ))
> > >
> > >
> > >  2.4a Write logs to ledger which Acknowledgement State is
> PENDING_ACK
> > > [Acknowledgement State Machine](
> > >
> >
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#bookmark=id.4bikq6sjiy8u
> > )
> > > tells about the changes of the Acknowledge State and why we need
> > persistent
> > > “The Log which the Acknowledgement State is PENDING_ACK”.
> > >  2.4a’ Mark messages is no longer useful with current subscription
> > > Update `Cursor` to mark the messages as DELETED. So they can be
> deleted.
> > >  3.2b Mark messages is no longer useful with current subscription
> > > The implementation here is exactly the same as 2.4a’, except that the
> > > execution is triggered later, after the Transaction has been committed.
> > >
> > >
> > > ### Analyze the performance cost of transaction
> > > As you can see Figure 1. Normal Flow
> > vs.
> > > Transaction Flow]: 2.4a 'and 3.2b are exactly the same logic, so
> the
> > > remaining orange arrows are the additional performance overhead of all
> > > transactions.
> > > In terms of whether or not each transaction is executed multiple times,
> > we
> > > can split the flow into two classes(Optimizing a process that is
> executed
> > > multiple times will yield more benefits):
> > > - Executed once each transaction: flow-1.x and flow-3.x
> > > - Executed multiple times each transaction: flow-2.x
> > >
> > > So optimizing the flow 2.x with a lot of execution is a good choice.
> > Let's
> > > split flow-2.x into two groups: those that cost more and those that
> cost
> > > less:
> > > - No disk written: flow-2.1 and fow-2.3
> > > - Disk written: fow-2.1a, fow-2.3a, flow-2.4a
> > >
> > > From the previous analysis, we found that optimizing flow-2.1a,
> > flow-2.3a,
> > > flow-2.4a would bring the most benefits, and batch writes

[GitHub] [pulsar-client-node] tuteng commented on issue #87: npm install fails on MacOS

2022-06-16 Thread GitBox



tuteng commented on issue #87:
URL: 
https://github.com/apache/pulsar-client-node/issues/87#issuecomment-1158192533

   > nodeJs version : 16.15.1 npm version : 8.11.0 mac OS: 12.4 (M1 max)
   > 
   > I installed the cpp client first and export PULSAR_CPP_DIR.. can't npm 
install too
   > 
   > ```
   > npm ERR! code 1
   > npm ERR! path 
/Users/jackliu/Project/pulsar/pulsar_test/node_modules/pulsar-client
   > npm ERR! command failed
   > npm ERR! command sh -c node-pre-gyp install --fallback-to-build
   > npm ERR! CC(target) Release/obj.target/nothing/../node-addon-api/nothing.o
   > npm ERR!   LIBTOOL-STATIC Release/nothing.a
   > npm ERR!   CXX(target) Release/obj.target/Pulsar/src/addon.o
   > npm ERR!   CXX(target) Release/obj.target/Pulsar/src/Message.o
   > npm ERR!   CXX(target) Release/obj.target/Pulsar/src/MessageId.o
   > npm ERR!   CXX(target) Release/obj.target/Pulsar/src/Authentication.o
   > npm ERR!   CXX(target) Release/obj.target/Pulsar/src/Client.o
   > npm ERR!   CXX(target) Release/obj.target/Pulsar/src/Producer.o
   > npm ERR!   CXX(target) Release/obj.target/Pulsar/src/ProducerConfig.o
   > npm ERR!   CXX(target) Release/obj.target/Pulsar/src/Consumer.o
   > npm ERR!   CXX(target) Release/obj.target/Pulsar/src/ConsumerConfig.o
   > npm ERR!   CXX(target) Release/obj.target/Pulsar/src/Reader.o
   > npm ERR!   CXX(target) Release/obj.target/Pulsar/src/ReaderConfig.o
   > npm ERR!   SOLINK_MODULE(target) Release/Pulsar.node
   > npm ERR! Failed to execute 
'/Users/jackliu/.nvm/versions/node/v16.15.1/bin/node 
/Users/jackliu/.nvm/versions/node/v16.15.1/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js
 build --fallback-to-build 
--module=/Users/jackliu/Project/pulsar/pulsar_test/node_modules/pulsar-client/build/Release/libpulsar.node
 --module_name=libpulsar 
--module_path=/Users/jackliu/Project/pulsar/pulsar_test/node_modules/pulsar-client/build/Release
 --napi_version=8 --node_abi_napi=napi --napi_build_version=0 
--node_napi_label=node-v93' (1)
   > npm ERR! node-pre-gyp info it worked if it ends with ok
   > npm ERR! node-pre-gyp info using node-pre-gyp@1.0.9
   > npm ERR! node-pre-gyp info using node@16.15.1 | darwin | arm64
   > npm ERR! node-pre-gyp info check checked for 
"/Users/jackliu/Project/pulsar/pulsar_test/node_modules/pulsar-client/build/Release/libpulsar.node"
 (not found)
   > npm ERR! node-pre-gyp http GET 
https://pulsar.apache.org/docs/en/client-libraries-cpp/libpulsar-v1.6.2-node-v93-darwin-arm64.tar.gz
   > npm ERR! node-pre-gyp ERR! install response status 404 Not Found on 
https://pulsar.apache.org/docs/en/client-libraries-cpp/libpulsar-v1.6.2-node-v93-darwin-arm64.tar.gz
   > npm ERR! node-pre-gyp WARN Pre-built binaries not installable for 
pulsar-client@1.6.2 and node@16.15.1 (node-v93 ABI, unknown) (falling back to 
source compile with node-gyp)
   > npm ERR! node-pre-gyp WARN Hit error response status 404 Not Found on 
https://pulsar.apache.org/docs/en/client-libraries-cpp/libpulsar-v1.6.2-node-v93-darwin-arm64.tar.gz
   > npm ERR! gyp info it worked if it ends with ok
   > npm ERR! gyp info using node-gyp@9.0.0
   > npm ERR! gyp info using node@16.15.1 | darwin | arm64
   > npm ERR! gyp info ok
   > npm ERR! gyp info it worked if it ends with ok
   > npm ERR! gyp info using node-gyp@9.0.0
   > npm ERR! gyp info using node@16.15.1 | darwin | arm64
   > npm ERR! gyp info find Python using Python version 3.9.13 found at 
"/opt/homebrew/opt/python@3.9/bin/python3.9"
   > npm ERR! gyp info spawn /opt/homebrew/opt/python@3.9/bin/python3.9
   > npm ERR! gyp info spawn args [
   > npm ERR! gyp info spawn args   
'/Users/jackliu/.nvm/versions/node/v16.15.1/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py',
   > npm ERR! gyp info spawn args   'binding.gyp',
   > npm ERR! gyp info spawn args   '-f',
   > npm ERR! gyp info spawn args   'make',
   > npm ERR! gyp info spawn args   '-I',
   > npm ERR! gyp info spawn args   
'/Users/jackliu/Project/pulsar/pulsar_test/node_modules/pulsar-client/build/config.gypi',
   > npm ERR! gyp info spawn args   '-I',
   > npm ERR! gyp info spawn args   
'/Users/jackliu/.nvm/versions/node/v16.15.1/lib/node_modules/npm/node_modules/node-gyp/addon.gypi',
   > npm ERR! gyp info spawn args   '-I',
   > npm ERR! gyp info spawn args   
'/Users/jackliu/Library/Caches/node-gyp/16.15.1/include/node/common.gypi',
   > npm ERR! gyp info spawn args   '-Dlibrary=shared_library',
   > npm ERR! gyp info spawn args   '-Dvisibility=default',
   > npm ERR! gyp info spawn args   
'-Dnode_root_dir=/Users/jackliu/Library/Caches/node-gyp/16.15.1',
   > npm ERR! gyp info spawn args   
'-Dnode_gyp_dir=/Users/jackliu/.nvm/versions/node/v16.15.1/lib/node_modules/npm/node_modules/node-gyp',
   > npm ERR! gyp info spawn args   
'-Dnode_lib_file=/Users/jackliu/Library/Caches/node-gyp/16.15.1/<(target_arch)/node.lib',
   > npm ERR! gyp info spawn args   
'-Dmodule_root_dir=/Users/jackliu/Project/pulsar/pulsar_test/node_modules/pulsar-client',

[VOTE] PIP-177: Add the classLoader field for SchemaDefinition

2022-06-16 Thread Cong Zhao

Hi Pulsar Community,


I would like to start a VOTE on "Add the classLoader field for
SchemaDefinition" (PIP-177).


The proposal can be read at https://github.com/apache/pulsar/issues/16058

and the discussion thead is available at

https://lists.apache.org/thread/3wjvmpzo3pq1ff62f4cops7ckyrcgfhf


Voting will stay open for at least 48h.


Thanks,

Cong Zhao

[GitHub] [pulsar-client-node] nearzxide10 commented on issue #87: npm install fails on MacOS

2022-06-16 Thread GitBox



nearzxide10 commented on issue #87:
URL: 
https://github.com/apache/pulsar-client-node/issues/87#issuecomment-1158382450

   > > nodeJs version : 16.15.1 npm version : 8.11.0 mac OS: 12.4 (M1 max)
   > > I installed the cpp client first and export PULSAR_CPP_DIR.. can't npm 
install too
   > > ```
   > > npm ERR! code 1
   > > npm ERR! path 
/Users/jackliu/Project/pulsar/pulsar_test/node_modules/pulsar-client
   > > npm ERR! command failed
   > > npm ERR! command sh -c node-pre-gyp install --fallback-to-build
   > > npm ERR! CC(target) 
Release/obj.target/nothing/../node-addon-api/nothing.o
   > > npm ERR!   LIBTOOL-STATIC Release/nothing.a
   > > npm ERR!   CXX(target) Release/obj.target/Pulsar/src/addon.o
   > > npm ERR!   CXX(target) Release/obj.target/Pulsar/src/Message.o
   > > npm ERR!   CXX(target) Release/obj.target/Pulsar/src/MessageId.o
   > > npm ERR!   CXX(target) Release/obj.target/Pulsar/src/Authentication.o
   > > npm ERR!   CXX(target) Release/obj.target/Pulsar/src/Client.o
   > > npm ERR!   CXX(target) Release/obj.target/Pulsar/src/Producer.o
   > > npm ERR!   CXX(target) Release/obj.target/Pulsar/src/ProducerConfig.o
   > > npm ERR!   CXX(target) Release/obj.target/Pulsar/src/Consumer.o
   > > npm ERR!   CXX(target) Release/obj.target/Pulsar/src/ConsumerConfig.o
   > > npm ERR!   CXX(target) Release/obj.target/Pulsar/src/Reader.o
   > > npm ERR!   CXX(target) Release/obj.target/Pulsar/src/ReaderConfig.o
   > > npm ERR!   SOLINK_MODULE(target) Release/Pulsar.node
   > > npm ERR! Failed to execute 
'/Users/jackliu/.nvm/versions/node/v16.15.1/bin/node 
/Users/jackliu/.nvm/versions/node/v16.15.1/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js
 build --fallback-to-build 
--module=/Users/jackliu/Project/pulsar/pulsar_test/node_modules/pulsar-client/build/Release/libpulsar.node
 --module_name=libpulsar 
--module_path=/Users/jackliu/Project/pulsar/pulsar_test/node_modules/pulsar-client/build/Release
 --napi_version=8 --node_abi_napi=napi --napi_build_version=0 
--node_napi_label=node-v93' (1)
   > > npm ERR! node-pre-gyp info it worked if it ends with ok
   > > npm ERR! node-pre-gyp info using node-pre-gyp@1.0.9
   > > npm ERR! node-pre-gyp info using node@16.15.1 | darwin | arm64
   > > npm ERR! node-pre-gyp info check checked for 
"/Users/jackliu/Project/pulsar/pulsar_test/node_modules/pulsar-client/build/Release/libpulsar.node"
 (not found)
   > > npm ERR! node-pre-gyp http GET 
https://pulsar.apache.org/docs/en/client-libraries-cpp/libpulsar-v1.6.2-node-v93-darwin-arm64.tar.gz
   > > npm ERR! node-pre-gyp ERR! install response status 404 Not Found on 
https://pulsar.apache.org/docs/en/client-libraries-cpp/libpulsar-v1.6.2-node-v93-darwin-arm64.tar.gz
   > > npm ERR! node-pre-gyp WARN Pre-built binaries not installable for 
pulsar-client@1.6.2 and node@16.15.1 (node-v93 ABI, unknown) (falling back to 
source compile with node-gyp)
   > > npm ERR! node-pre-gyp WARN Hit error response status 404 Not Found on 
https://pulsar.apache.org/docs/en/client-libraries-cpp/libpulsar-v1.6.2-node-v93-darwin-arm64.tar.gz
   > > npm ERR! gyp info it worked if it ends with ok
   > > npm ERR! gyp info using node-gyp@9.0.0
   > > npm ERR! gyp info using node@16.15.1 | darwin | arm64
   > > npm ERR! gyp info ok
   > > npm ERR! gyp info it worked if it ends with ok
   > > npm ERR! gyp info using node-gyp@9.0.0
   > > npm ERR! gyp info using node@16.15.1 | darwin | arm64
   > > npm ERR! gyp info find Python using Python version 3.9.13 found at 
"/opt/homebrew/opt/python@3.9/bin/python3.9"
   > > npm ERR! gyp info spawn /opt/homebrew/opt/python@3.9/bin/python3.9
   > > npm ERR! gyp info spawn args [
   > > npm ERR! gyp info spawn args   
'/Users/jackliu/.nvm/versions/node/v16.15.1/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py',
   > > npm ERR! gyp info spawn args   'binding.gyp',
   > > npm ERR! gyp info spawn args   '-f',
   > > npm ERR! gyp info spawn args   'make',
   > > npm ERR! gyp info spawn args   '-I',
   > > npm ERR! gyp info spawn args   
'/Users/jackliu/Project/pulsar/pulsar_test/node_modules/pulsar-client/build/config.gypi',
   > > npm ERR! gyp info spawn args   '-I',
   > > npm ERR! gyp info spawn args   
'/Users/jackliu/.nvm/versions/node/v16.15.1/lib/node_modules/npm/node_modules/node-gyp/addon.gypi',
   > > npm ERR! gyp info spawn args   '-I',
   > > npm ERR! gyp info spawn args   
'/Users/jackliu/Library/Caches/node-gyp/16.15.1/include/node/common.gypi',
   > > npm ERR! gyp info spawn args   '-Dlibrary=shared_library',
   > > npm ERR! gyp info spawn args   '-Dvisibility=default',
   > > npm ERR! gyp info spawn args   
'-Dnode_root_dir=/Users/jackliu/Library/Caches/node-gyp/16.15.1',
   > > npm ERR! gyp info spawn args   
'-Dnode_gyp_dir=/Users/jackliu/.nvm/versions/node/v16.15.1/lib/node_modules/npm/node_modules/node-gyp',
   > > npm ERR! gyp info spawn args   
'-Dnode_lib_file=/Users/jackliu/Library/Caches/node-gyp/16.15.1/<(target_arch)/node.lib',
   > > npm ERR! gyp inf

[GitHub] [pulsar-client-node] equanz commented on issue #87: npm install fails on MacOS

2022-06-16 Thread GitBox



equanz commented on issue #87:
URL: 
https://github.com/apache/pulsar-client-node/issues/87#issuecomment-1158384683

   @nearzxide10
   > ```
   > npm ERR! ld: library not found for -lpulsar
   > npm ERR! clang: error: linker command failed with exit code 1 (use -v to 
see invocation)
   > ```
   
   It seems clang doesn't link the `pulsar` library yet.
   Temporally, you set `LIBRARY_PATH` environment variable to the 
libpulsar.dylib path like below.
   
   ```
   export LIBRARY_PATH=$(brew --prefix)/lib${LIBRARY_PATH+:${LIBRARY_PATH}}
   # or simply
   export LIBRARY_PATH=$(brew --prefix)/lib
   ```
   
   I think the root cause of this issue is below. I'll try to fix it.
   
https://github.com/apache/pulsar-client-node/blob/d8d24eb5f563869ba0028b8a1ae8646601252a93/binding.gyp#L105-L108
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-node] nearzxide10 commented on issue #87: npm install fails on MacOS

2022-06-16 Thread GitBox



nearzxide10 commented on issue #87:
URL: 
https://github.com/apache/pulsar-client-node/issues/87#issuecomment-1158391286

   > @nearzxide10
   > 
   > > ```
   > > npm ERR! ld: library not found for -lpulsar
   > > npm ERR! clang: error: linker command failed with exit code 1 (use -v to 
see invocation)
   > > ```
   > 
   > It seems clang doesn't link the `pulsar` library yet. Temporally, you set 
`LIBRARY_PATH` environment variable to the libpulsar.dylib path like below.
   > 
   > ```
   > export LIBRARY_PATH=$(brew --prefix)/lib${LIBRARY_PATH+:${LIBRARY_PATH}}
   > # or simply
   > export LIBRARY_PATH=$(brew --prefix)/lib
   > ```
   > 
   > I think the root cause of this issue is below. I'll try to fix it.
   > 
   > 
https://github.com/apache/pulsar-client-node/blob/d8d24eb5f563869ba0028b8a1ae8646601252a93/binding.gyp#L105-L108
   
   Thanks, it work for me 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] Reject partitioned topic creation when the topic name contains the `partition` keyword.

2022-06-16 Thread mattison chao

I pushed a pull request[1]. That can avoid creating partitioned metadata
when the topic local name includes `-partition-` keywords.

[1] https://github.com/apache/pulsar/pull/16104

[GitHub] [pulsar-connectors] dependabot[bot] opened a new pull request, #3: Bump fastjson from 1.2.73 to 1.2.83 in /pulsar-io/canal

2022-06-16 Thread GitBox



dependabot[bot] opened a new pull request, #3:
URL: https://github.com/apache/pulsar-connectors/pull/3

   Bumps [fastjson](https://github.com/alibaba/fastjson) from 1.2.73 to 1.2.83.
   
   Release notes
   Sourced from https://github.com/alibaba/fastjson/releases";>fastjson's 
releases.
   
   FASTJSON 1.2.83版本发布（安全修复）
   这是一个安全修复版本，修复最近收到在特定场景下可以绕过autoType关闭限制的漏洞，建议fastjson用户尽快采取安全措施保障系统安全。
   https://github.com/alibaba/fastjson/wiki/security_update_20220523";>安全修复方案
 ：https://github.com/alibaba/fastjson/wiki/security_update_20220523";>https://github.com/alibaba/fastjson/wiki/security_update_20220523
   Issues
   
   安全加固
   修复JDK17下setAccessible报错的问题 https://github-redirect.dependabot.com/alibaba/fastjson/issues/4077";>#4077
   
   
   下载 https://repo1.maven.org/maven2/com/alibaba/fastjson/1.2.83/";>https://repo1.maven.org/maven2/com/alibaba/fastjson/1.2.83/
   文档 https://github.com/alibaba/fastjson/wiki/%E5%B8%B8%E8%A7%81%E9%97%AE%E9%A2%98";>https://github.com/alibaba/fastjson/wiki/%E5%B8%B8%E8%A7%81%E9%97%AE%E9%A2%98
   源码 https://github.com/alibaba/fastjson/tree/1.2.83";>https://github.com/alibaba/fastjson/tree/1.2.83
   
   fastjson 1.2.79版本发布，BUG修复
   这又是一个bug fixed的版本，大家按需升级
   Issues
   
   修复引入MethodInheritanceComparator导致某些场景序列化报错的问题
   增强JDK 9兼容
   修复JSONArray/JSONObject的equals方法在内部对象map/list相同时不直接返回true的问题
   
   相关链接
   
   下载 https://repo1.maven.org/maven2/com/alibaba/fastjson/1.2.79/";>https://repo1.maven.org/maven2/com/alibaba/fastjson/1.2.79/
   文档 https://github.com/alibaba/fastjson/wiki/%E5%B8%B8%E8%A7%81%E9%97%AE%E9%A2%98";>https://github.com/alibaba/fastjson/wiki/%E5%B8%B8%E8%A7%81%E9%97%AE%E9%A2%98
   源码 https://github.com/alibaba/fastjson/tree/1.2.79";>https://github.com/alibaba/fastjson/tree/1.2.79
   
   fastjson 1.2.76版本发布，BUG修复增强兼容
   这又是一个bug fixed的版本，大家按需升级
   Issues
   
   修复一些直接抛RuntimeException的问题 https://github-redirect.dependabot.com/alibaba/fastjson/issues/3631";>#3631
   parser自动识别gzip bytes https://github-redirect.dependabot.com/alibaba/fastjson/issues/3614";>#3614
   修复Throwable继承类属性不支持自动类型转换问题 https://github-redirect.dependabot.com/alibaba/fastjson/issues/3217";>#3217
   修复PrettyFormat情况下引用计算不对的问题 https://github-redirect.dependabot.com/alibaba/fastjson/issues/3672";>#3672
   修复AutoType不兼容LinkedHashMap的问题
   增强对Enum类型的自定类型转换
   修复deserializeUsing在泛型某些场景不能正常工作的问题 https://github-redirect.dependabot.com/alibaba/fastjson/issues/3693";>#3693
   提升JSONReader性能，减少小对象创建 https://github-redirect.dependabot.com/alibaba/fastjson/issues/3627";>#3627
   增强对JSONPath对filter的支持 https://github-redirect.dependabot.com/alibaba/fastjson/issues/3629";>#3629
   JSONPath支持忽略NullValue的选项 https://github-redirect.dependabot.com/alibaba/fastjson/issues/3607";>#3607
   增强对定制化enum的支持 https://github-redirect.dependabot.com/alibaba/fastjson/issues/3601";>#3601
   增强对java.time.Instant和org.joda.time.Instant的支持 https://github-redirect.dependabot.com/alibaba/fastjson/issues/3539";>#3539
   修复Parser某些场景不能识别引用的问题
   
   相关链接
   
   下载 https://repo1.maven.org/maven2/com/alibaba/fastjson/1.2.76/";>https://repo1.maven.org/maven2/com/alibaba/fastjson/1.2.76/
   文档 https://github.com/alibaba/fastjson/wiki/%E5%B8%B8%E8%A7%81%E9%97%AE%E9%A2%98";>https://github.com/alibaba/fastjson/wiki/%E5%B8%B8%E8%A7%81%E9%97%AE%E9%A2%98
   源码 https://github.com/alibaba/fastjson/tree/1.2.76";>https://github.com/alibaba/fastjson/tree/1.2.76
   
   fastjson 1.2.75版本发布，例行Bug修复
   
   
   ... (truncated)
   
   
   Commits
   
   https://github.com/alibaba/fastjson/commit/26f13f84fdd522de10678e43f55fde918ab7b347";>26f13f8
 1.2.83
   https://github.com/alibaba/fastjson/commit/8f3410f81cbd437f7c459f8868445d50ad301f15";>8f3410f
 bug fix for autotype
   https://github.com/alibaba/fastjson/commit/cd3c2de6968f0046be1ac4cc7475627ebefbcf64";>cd3c2de
 improved jdk8 java.time support
   https://github.com/alibaba/fastjson/commit/c63866eaca27441721a82ad8a2553b929e1e3360";>c63866e
 remove unused import
   https://github.com/alibaba/fastjson/commit/35db4adad70c32089542f23c272def1ad920a60d";>35db4ad
 bug fix for autoType
   https://github.com/alibaba/fastjson/commit/3f009e10a6905778b42dd3f8cad2de3b194a62d4";>3f009e1
 Merge pull request https://github-redirect.dependabot.com/alibaba/fastjson/issues/4085";>#4085
 from hengyunabc/fix_setAccessible
   https://github.com/alibaba/fastjson/commit/dd3de5f84bcbf0c13ad60df0c3feb9d564727447";>dd3de5f
 fix InaccessibleObjectException in jdk17. https://github-redirect.dependabot.com/alibaba/fastjson/issues/4077";>#4077
   https://github.com/alibaba/fastjson/commit/a234f9abba58add6e5ef9d44b0c24bf49a583905";>a234f9a
 Merge pull request https://github-redirect.dependabot.com/alibaba/fastjson/issues/4084";>#4084
 from alibaba/revert-4078-master
   https://github.com/alibaba/fastjson/commit/0814909139735ea4c7648e755630fd49da7be059";>0814909
 Revert "fix InaccessibleObjectException in jdk17. https://github-redirect.dependabot.com/alibaba/fastjson/issues/4077";>#4077"
   https://github.com/alibaba/fastjson/comm

Re: [VOTE] PIP-165: Auto release client useless connections

2022-06-16 Thread Yubiao Feng

Hi  Matteo, Enrico, LinLin

I would appreciate it if you could give me some comments.

Thanks
Yubiao.Feng

On Wed, Jun 15, 2022 at 2:47 PM Hang Chen  wrote:

> +1
>
> Thanks,
> Hang
>
> Zike Yang  于2022年6月15日周三 10:31写道：
> >
> > +1
> >
> >
> > Zike Yang
> >
> > On Wed, Jun 15, 2022 at 10:28 AM Haiting Jiang 
> > wrote:
> >
> > > +1
> > >
> > > Haiting
> > >
> > > On 2022/06/14 10:28:22 Xiangying Meng wrote:
> > > > +1
> > > >
> > > > On Tue, Jun 14, 2022 at 3:52 PM Gavin Gao 
> > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > >
> > > > > On 2022/06/01 07:11:51 Yubiao Feng wrote:
> > > > > > Hi Pulsar Community:
> > > > > >
> > > > > > There were some mistakes in the last email, so I have corrected
> them.
> > > > > >
> > > > > > I would like to start a VOTE on "Auto release client useless
> > > connections"
> > > > > > (PIP-165).
> > > > > >
> > > > > > Proposal Link: [PIP-165] Auto release client useless connections
> ·
> > > Issue
> > > > > > #15516 · apache/pulsar (github.com)
> > > > > > 
> > > > > >
> > > > > > Discuss Link: [DISCUSS] [PIP-165] Auto release client useless
> > > > > > connections-Apache Mail Archives
> > > > > > <
> https://lists.apache.org/thread/t6h98qs2coc56z06tw38hdlljl67ft4n>
> > > > > >
> > > > > > Voting will stay open for at least 48h.
> > > > > > Thanks, Yubiao Feng
> > > > > >
> > > > > > On Wed, Jun 1, 2022 at 2:40 PM Yubiao Feng <
> > > yubiao.f...@streamnative.io>
> > > > > > wrote:
> > > > > >
> > > > > > > [VOTE] PIP-165: Auto release client useless connectionsHi
> Pulsar
> > > > > > > Community, I would like to start a VOTE on "Auto release client
> > > useless
> > > > > > > connections" (PIP-165). The proposal can be read at
> > > > > > > https://github.com/apache/pulsar/issues/15516
> > > > > > >  and the
> discussion
> > > > > thead
> > > > > > > is available at
> > > > > > >
> https://lists.apache.org/thread/t6h98qs2coc56z06tw38hdlljl67ft4n
> > > > > > >   Voting will stay open for at least 48h. Thanks, Yubiao Feng
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Re: [DISCUSS] [PIP-160] Batch writing ledger for transaction operation

2022-06-16 Thread Yubiao Feng

Hi  Enrico

> I am not sure I understand the part of making it configurable via a
classname.
I believe it is better to simply have a flag "transactionEnableBatchWrites".
Otherwise the matrix of possible implementations will grow without limits.

Good idea, I've modified the design and added a switch in the Configure
Changes section. Could you take a look again.

On Fri, Jun 10, 2022 at 7:14 PM Enrico Olivelli  wrote:

> I have read the PIP, and overall I agree with the design.
> Good work !
>
> I am not sure I understand the part of making it configurable via a
> classname.
> I believe it is better to simply have a flag
> "transactionEnableBatchWrites".
> Otherwise the matrix of possible implementations will grow without limits.
>
> Enrico
>
> Il giorno ven 10 giu 2022 alle ore 11:35 Yubiao Feng
>  ha scritto:
> >
> > Hi Pulsar community:
> >
> > I open a pip to discuss "Batch writing ledger for transaction operation"
> >
> > Proposal Link: https://github.com/apache/pulsar/issues/15370
> >
> > ## Motivation
> >
> > Before reading the background, I suggest you read section “Transaction
> > Flow” of [PIP-31: Transactional Streaming](
> >
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx
> > )
> >
> > ###  Normal Flow vs. Transaction Flow
> 
> > ![MG3](
> >
> https://user-images.githubusercontent.com/25195800/172985866-25e496a4-ea93-42ec-aa0d-e6a02aa0635e.jpeg
> > )
> > In *Figure 1. Normal Flow vs. Transaction Flow*:
> > - The gray square boxes represent logical components.
> > - All the blue boxes represent logs. The logs are usually Managed ledger
> > - Each arrow represents the request flow or message flow. These
> operations
> > occur in sequence indicated by the numbers next to each arrow.
> > - The black arrows indicate those shared by transaction and normal flow.
> > - The blue arrows represent normal-message-specific flow.
> > - The orange arrows represent transaction-message-specific flow.
> > - The sections below are numbered to match the operations showed in the
> > diagram(differ from [PIP-31: Transactional Streaming](
> >
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx
> > ))
> >
> >
> >  2.4a Write logs to ledger which Acknowledgement State is PENDING_ACK
> > [Acknowledgement State Machine](
> >
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#bookmark=id.4bikq6sjiy8u
> )
> > tells about the changes of the Acknowledge State and why we need
> persistent
> > “The Log which the Acknowledgement State is PENDING_ACK”.
> >  2.4a’ Mark messages is no longer useful with current subscription
> > Update `Cursor` to mark the messages as DELETED. So they can be deleted.
> >  3.2b Mark messages is no longer useful with current subscription
> > The implementation here is exactly the same as 2.4a’, except that the
> > execution is triggered later, after the Transaction has been committed.
> >
> >
> > ### Analyze the performance cost of transaction
> > As you can see Figure 1. Normal Flow
> vs.
> > Transaction Flow]: 2.4a 'and 3.2b are exactly the same logic, so the
> > remaining orange arrows are the additional performance overhead of all
> > transactions.
> > In terms of whether or not each transaction is executed multiple times,
> we
> > can split the flow into two classes(Optimizing a process that is executed
> > multiple times will yield more benefits):
> > - Executed once each transaction: flow-1.x and flow-3.x
> > - Executed multiple times each transaction: flow-2.x
> >
> > So optimizing the flow 2.x with a lot of execution is a good choice.
> Let's
> > split flow-2.x into two groups: those that cost more and those that cost
> > less:
> > - No disk written: flow-2.1 and fow-2.3
> > - Disk written: fow-2.1a, fow-2.3a, flow-2.4a
> >
> > From the previous analysis, we found that optimizing flow-2.1a,
> flow-2.3a,
> > flow-2.4a would bring the most benefits, and batch writes would be an
> > excellent solution for multiple disk writes. Flow-2.1a and Flow-2.3a are
> > both manipulations written into the transaction log, we can combine them
> in
> > one batch; 2.4a is the operation of writing pending ACK log, we combine
> > multiple 2.4a's into one batch for processing.
> > As we can see from “Transaction Flow” of [PIP-31: Transactional
> Streaming](
> >
> https://docs.google.com/document/d/145VYp09JKTw9jAT-7yNyFU255FptB2_B2Fye100ZXDI/edit#heading=h.bm5ainqxosrx
> ),
> > these instructions are strictly sequential (guaranteed by the client):
> > - flow-1.x end before flow-2.x start
> > - flow-2.x end before flow-3.x start
> > - flow-3.1a end before flow-3.3a start
> >
> > Therefore, the broker does not need to worry about the dependency of
> these
> > flows, we can also put flow-1a flow-31a and flow-3.3a into The
> Transaction
> > Log Batch too.
> >
> > ## Goal
> > Provide a mechanism for Transaction Log Store and Pending Ack Store:
> accept

[GitHub] [pulsar-client-node] equanz opened a new pull request, #220: Split conditions between OS in binding.gyp

2022-06-16 Thread GitBox



equanz opened a new pull request, #220:
URL: https://github.com/apache/pulsar-client-node/pull/220

   # Motivation
   On macOS and Apple Silicon, we can't build pulsar-client-node if using 
default environment variables like below.
   
https://github.com/apache/pulsar-client-node/issues/87#issuecomment-1158192533
   
   ```
   % echo $LIBRARY_PATH
   
   
   % PULSAR_CPP_DIR=/opt/homebrew/Cellar/libpulsar/2.10.0 npm install
   ...
   ld: library not found for -lpulsar
   clang: error: linker command failed with exit code 1 (use -v to see 
invocation)
   make: *** [Debug/Pulsar.node] Error 1
   gyp ERR! build error 
   gyp ERR! stack Error: `make` failed with exit code: 2
   gyp ERR! stack at ChildProcess.onExit 
(/opt/homebrew/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
   gyp ERR! stack at ChildProcess.emit (node:events:527:28)
   gyp ERR! stack at ChildProcess._handle.onexit 
(node:internal/child_process:291:12)
   gyp ERR! System Darwin 21.5.0
   gyp ERR! command "/opt/homebrew/Cellar/node/18.3.0/bin/node" 
"/opt/homebrew/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" 
"build" "--fallback-to-build" "--debug" 
"--module=/Users/yumizush/src/pulsar-client-node/build/Release/libpulsar.node" 
"--module_name=libpulsar" 
"--module_path=/Users/yumizush/src/pulsar-client-node/build/Release" 
"--napi_version=8" "--node_abi_napi=napi" "--napi_build_version=0" 
"--node_napi_label=node-v108"
   gyp ERR! cwd /Users/yumizush/src/pulsar-client-node
   gyp ERR! node -v v18.3.0
   gyp ERR! node-gyp -v v9.0.0
   gyp ERR! not ok 
   ...
   ```
   
   The Root cause of the issue is clang tries to link `libpulsar` libraries 
through `-lpulsar` even if `OS==mac` .
   Therefore, If you don't add `libpulsar` to the library path, then fail on 
any architectures.
   
   I want to fix it.
   
   # Modification
   * Split target conditions between OS in `binding.gyp`
 - cf. 
https://github.com/nodejs/node-gyp/blob/v8.4.1/gyp/pylib/gyp/input.py#L1187-L1196
 - cf. https://docs.python.org/3/library/functions.html#compile
   * Add an example script for macOS and Apple Silicon to the README.md
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] urfreespace merged pull request #119: fix: url trailing slash

2022-06-16 Thread GitBox



urfreespace merged PR #119:
URL: https://github.com/apache/pulsar-site/pull/119


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] MarvinCai opened a new issue, #272: Allow broker/proxy to define extra envs.

2022-06-16 Thread GitBox



MarvinCai opened a new issue, #272:
URL: https://github.com/apache/pulsar-helm-chart/issues/272

   **Is your feature request related to a problem? Please describe.**
   Allow broker/proxy to define extra envs so user can add env like Pod IP or 
Pod Name.
   
   **Describe the solution you'd like**
   Allow broker/proxy to define extra envs.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-helm-chart] MarvinCai opened a new pull request, #273: Support defining extra env for broker and proxy statefulsset.

2022-06-16 Thread GitBox



MarvinCai opened a new pull request, #273:
URL: https://github.com/apache/pulsar-helm-chart/pull/273

   Fixes #272
   
   ### Motivation
   Allow broker/proxy to define extra envs so user can add env like Pod IP or 
Pod Name.
   
   ### Modifications
   Add extra env in broker/proxy sts if defined.
   
   ### Verifying this change
   - [x] Make sure that the change passes the CI checks.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-client-node] nkurihar commented on pull request #217: Suggest pulsar-admin-client in README.md

[GitHub] [pulsar-client-node] visortelle commented on pull request #217: Suggest pulsar-admin-client in README.md

[GitHub] [pulsar-client-node] visortelle closed pull request #217: Suggest pulsar-admin-client in README.md

[GitHub] [pulsar-client-node] nearzxide10 commented on issue #87: npm install fails on MacOS

Re: Mark delete rate limiter bug?

Re: [DISCUSS] [PIP-160] Batch writing ledger for transaction operation

[GitHub] [pulsar-manager] toneill818 opened a new issue, #465: Docker Failed to Initialize Postgresql Database

Re: [DISCUSS] [PIP-160] Batch writing ledger for transaction operation

Re: [DISCUSS] [PIP-160] Batch writing ledger for transaction operation

[GitHub] [pulsar-client-node] tuteng commented on issue #87: npm install fails on MacOS

[VOTE] PIP-177: Add the classLoader field for SchemaDefinition

[GitHub] [pulsar-client-node] nearzxide10 commented on issue #87: npm install fails on MacOS

[GitHub] [pulsar-client-node] equanz commented on issue #87: npm install fails on MacOS

[GitHub] [pulsar-client-node] nearzxide10 commented on issue #87: npm install fails on MacOS

Re: [DISCUSS] Reject partitioned topic creation when the topic name contains the `partition` keyword.

[GitHub] [pulsar-connectors] dependabot[bot] opened a new pull request, #3: Bump fastjson from 1.2.73 to 1.2.83 in /pulsar-io/canal

Re: [VOTE] PIP-165: Auto release client useless connections

Re: [DISCUSS] [PIP-160] Batch writing ledger for transaction operation

[GitHub] [pulsar-client-node] equanz opened a new pull request, #220: Split conditions between OS in binding.gyp

[GitHub] [pulsar-site] urfreespace merged pull request #119: fix: url trailing slash

[GitHub] [pulsar-helm-chart] MarvinCai opened a new issue, #272: Allow broker/proxy to define extra envs.

[GitHub] [pulsar-helm-chart] MarvinCai opened a new pull request, #273: Support defining extra env for broker and proxy statefulsset.

22 matches

Site Navigation

Mail list logo

Footer information