2020-02-12 14:34:26 UTC - Ashwin Kapur: Hi. Just looking at the pulsar-client-cpp source. Couple of thoughts and questions. 1) Patching protobuf and using the patched version seems to make building needlessly difficult. Since your goal is to use something protobuf like but without needless memory allocations, why not use google flatbuffers <https://google.github.io/flatbuffers/> instead. Designed precisely for this use case instead of patching something designed for a different purpose. If you do in fact do this, use flatbuffers 1.10 or build from source from the repo. The maven distribution of 1.11 is borked and has been for a long time. 2) You seem to be using CMake using the old style of CMake which means you have to specify every directory of headers etc. If you use the new style of Cmake which means you state your dependencies and let CMake figure everything out, see for example <https://gist.github.com/mbinna/c61dbb39bca0e4fb7d1f73b0d66a4fd1>, this makes using tools like Conan and vcpkg really easy and makes the build much simpler. Would you be open to a pull request that did 2. ---- 2020-02-12 14:37:31 UTC - Ashwin Kapur: In the short term, instead of 1, would it be possible to check in PulsarApi.pb.h and <http://PulsarApi.pb.cc|PulsarApi.pb.cc> into the repository. ---- 2020-02-12 14:39:28 UTC - Ashwin Kapur: If the PulsarApi.pb files aren't committed to github, it makes writing a conan or vcpkg file exponentially harder. ---- 2020-02-12 14:56:17 UTC - Ashwin Kapur: Adding on the the benefit of 1. With flatbuffers you don't just have no memory allocation, you also don't have a serialization and de serialization step. At least in C++, serialization is just writing to specific locations in memory. Deserialization is just reading from a specific memory location. Not sure how this is implemented in Java given that you don't really have the same low level facilities. ---- 2020-02-12 15:14:24 UTC - Ashwin Kapur: So it seems someone may have already asked for the <http://PulsarApi.pb.cc|PulsarApi.pb.cc> and PulsarApi.pb.h to be committed to the repo. Quoting from the bottom of protobuf/README.md ### For C++ Client:
The pre-generated C++ code is at `pulsar-client-cpp/lib/PulsarApi.pb.cc` and `pulsar-client-cpp/lib/PulsarApi.pb.h`. ---- 2020-02-12 15:14:44 UTC - Ashwin Kapur: The code is not there in the github repository or the source distribution. ---- 2020-02-12 15:22:16 UTC - Ashwin Kapur: Also from protobuf/README.md in the apache-pulsar-2.5.0 source distribution To re-generate the `PulsarApi.java` code you need to apply a patch to the protobuf generator. Patch is found in `build/docker/protobuf.patch`. #### Commands for creating the pre-generated Java code ```shell export PULSAR_HOME=<Path where you cloned the pulsar repo> cd $HOME git clone <https://github.com/google/protobuf.git> cd ${HOME}/protobuf git checkout v2.4.1 ### Apply patch patch -p1 < ${PULSAR_HOME}/build/docker/protobuf.patch ### Compile protobuf autoreconf --install ./configure make ### Re-generate PulsarApi cd ${PULSAR_HOME}/pulsar-common/ export PROTOC=${HOME}/protobuf/src/protoc ./generate_protobuf.sh ``` Or you can use the pre-built protoc included in `pulsar-build` docker image to generate java protobuf files. ---- 2020-02-12 15:22:59 UTC - Ashwin Kapur: Now if I go to the pulsar source directory there is no directory called build and find . -name protobuf.patch gives me no files ---- 2020-02-12 17:00:57 UTC - Yannis Weishaupt: @Yannis Weishaupt has joined the channel ---- 2020-02-12 17:20:58 UTC - Sijie Guo: @Ashwin Kapur the project started when flatbuffer doesn’t exist. so for backward compatibility issue, we stayed with protobuf at this moment. For sure we can consider using flatbuffer for future releases, it requires a huge effort for changing all these implementations and it also need consider backward compatibility. It might be doable through the current protocol handler mechanism to add a flatbuffer based protocol handler. just to correct one thing, we only used a customize protoc for java code generation. for the cpp and go code are using the organic protoc. in c++ client build process, cmake will automatically generate the files. We can consider adding those generated files to the repo as needed. If you want to contribute a pull request for it, that would be great! If you run `cmake` in the cpp client project, the files are generated under `./pulsar-client-cpp/generated/lib/PulsarApi.pb.cc` and `./pulsar-client-cpp/generated/lib/PulsarApi.pb.h` ---- 2020-02-12 17:49:36 UTC - Ali Ahmed: @Ashwin Kapur I did take a stab it <https://github.com/microsoft/vcpkg/pull/5693> ---- 2020-02-12 17:51:19 UTC - Ali Ahmed: took a look conan didn’t really try it ---- 2020-02-12 23:38:16 UTC - Eugen: <https://pulsar.apache.org/docs/en/functions-state/> says Functions State is in developer preview since 2.1.0 - is that still so, or is this considered stable now? ---- 2020-02-13 00:56:05 UTC - Sijie Guo: unfortunately, it is still in developer preview. we are lagging behind in this area. ---- 2020-02-13 01:25:52 UTC - Eugen: gotcha! ----
