"Yes", except Arrow has a linker script that hides the symbols, so you would
have to carve out an exception to get it to work. If you unhide one symbol, you
can make it work again (though I don't think this will let you actually use the
Protobuf types, only write your own Protobufs referencing Arrow Protobuf
types): https://github.com/lidavidm/arrow-14771-demo
Also on Windows, it just cannot work, because protoc doesn't properly implement
support for injecting a dllimport/dllexport visibility macro (even though it
claims to), at least that was the case last I tried.
On Mon, Feb 13, 2023, at 16:49, Li Jin wrote:
> "
> In this case though, it's just that we purposely hide symbols by default.
> If there's a use case, we could unhide this specific symbol (we did it for
> one other Protobuf symbol) which would let you externally generate and use
> the headers (as long as you take care not to actually include the generated
> sources).
> "
> My knowledge about these symbols hiding is lacky so I would like to take
> this opportunity and learn about it if possible.
>
> How is the hide/unhide done here? My understanding is that these protobuf
> symbols are included in the libarrow_fight.so, but not exposed via any
> public header, and therefore it is "hidden". But if the user protoc the
> header files, they can still access the symbols in libarrow_fight.so?
>
>
> On Mon, Feb 13, 2023 at 3:28 PM Paul Nienaber
> <paul.niena...@improving.com.invalid> wrote:
>
>> Yes I can build normally. Good point about this being in a public header,
>> as it would consequently be in any further application translation units...
>> I'll look into whether we can refactor what's been done already.
>> ________________________________
>> From: David Li <lidav...@apache.org>
>> Sent: Monday, February 13, 2023 12:15 PM
>> To: dev@arrow.apache.org <dev@arrow.apache.org>
>> Subject: Re: Build issues (Protobuf internal symbols)
>>
>> Oh, sorry, this is for Arrow development...
>>
>> A public header should not include an internal header (this includes the
>> Protobuf-generated headers). But can you build the project without
>> modification already?
>>
>> On Mon, Feb 13, 2023, at 15:02, Paul Nienaber wrote:
>> > Hi David,
>> >
>> > This is for Arrow feature development, where the changes have been made
>> > in client.h. Am I possibly running into symbol visibility guards I'm
>> > not expecting here and didn't spot yet?
>> >
>> > Paul
>> > ________________________________
>> > From: David Li <lidav...@apache.org>
>> > Sent: Monday, February 13, 2023 11:11 AM
>> > To: dev@arrow.apache.org <dev@arrow.apache.org>
>> > Subject: Re: Build issues (Protobuf internal symbols)
>> >
>> > The Protobuf-generated code from Arrow is not meant for external
>> > consumption, and in general, the way Protobuf works will make this
>> > complicated. What is the use case here?
>> >
>> > (In fact, that header shouldn't even be available for you to include.
>> > It seems you're linking/including it from a build of Arrow, but it
>> > won't be there once Arrow is actually installed.)
>> >
>> > In this case though, it's just that we purposely hide symbols by
>> > default. If there's a use case, we could unhide this specific symbol
>> > (we did it for one other Protobuf symbol) which would let you
>> > externally generate and use the headers (as long as you take care not
>> > to actually include the generated sources).
>> >
>> > On Mon, Feb 13, 2023, at 14:05, Paul Nienaber wrote:
>> >> Hi folks,
>> >>
>> >> I'm not familiar yet with all of the details of how Protobuf is set up
>> >> to be built and included or whether there's a subtle difference between
>> >> the CMake targets (headers vs cc sources), but we've recently gone and
>> >> started touching some of the Protobuf types from a C++ header and are
>> >> seeing build issues stemming from Protobuf-internal symbols/macros not
>> >> being properly in scope.
>> >>
>> >> Working with /cpp/src/arrow/flight/sql/client.h we've included
>> >> "arrow/flight/sql/FlightSql.pb.h" (this may itself be an issue, but
>> >> using <google/protobuf/any.pb.g was not exposing generated types from
>> >> sql/FlightSql), and we're seeing the following:
>> >>
>> >> [1/4] Building CXX object
>> >> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
>> >> FAILED:
>> >> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
>> >> /opt/homebrew/bin/ccache
>> >> /Library/Developer/CommandLineTools/usr/bin/c++
>> >> -DARROW_EXTRA_ERROR_CONTEXT -DARROW_FLIGHT_SQL_EXPORTING
>> >> -DARROW_HAVE_NEON -DARROW_WITH_RE2 -DARROW_WITH_TIMING_TESTS
>> >> -DARROW_WITH_ZLIB
>> >> -DGRPC_NAMESPACE_FOR_TLS_CREDENTIALS_OPTIONS=grpc::experimental
>> >> -DGRPC_USE_CERTIFICATE_VERIFIER
>> >> -DGRPC_USE_TLS_CHANNEL_CREDENTIALS_OPTIONS -DURI_STATIC_BUILD
>> >> -I/Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src
>> >> -I/Users/pauln/bq/git/arrow/cpp/src
>> >> -I/Users/pauln/bq/git/arrow/cpp/src/generated -isystem
>> >> /opt/homebrew/include -isystem
>> >> /Users/pauln/bq/git/arrow/cpp/thirdparty/flatbuffers/include -isystem
>> >> /Users/pauln/bq/git/arrow/cpp/thirdparty/hadoop/include -isystem
>> >> /opt/homebrew/opt/openssl@1.1/include -isystem
>> >>
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/jemalloc_ep-prefix/src
>> >> -fno-aligned-new -Qunused-arguments -fcolor-diagnostics -Wall -Wextra
>> >> -Wdocumentation -Wshorten-64-to-32 -Wno-missing-braces
>> >> -Wno-unused-parameter -Wno-constant-logical-operand
>> >> -Wno-return-stack-address -Wno-unknown-warning-option -Wno-pass-failed
>> >> -march=armv8-a -g -Werror -O0 -ggdb -arch arm64 -isysroot
>> >> /Library/Developer/CommandLineTools/SDKs/MacOSX13.1.sdk -fPIC
>> >> -std=c++17 -MD -MT
>> >> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
>> >> -MF
>> >>
>> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o.d
>> >> -o
>> >> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
>> >> -c /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.cc
>> >> In file included from
>> >> /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.cc:22:
>> >> In file included from
>> >> /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.h:24:
>> >>
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:47:3:
>> >> error: expected expression
>> >> static const uint32_t offsets[];
>> >> ^
>> >>
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:46:32:
>> >> error: variable has incomplete type 'struct ARROW_FLIGHT_SQL_EXPORT'
>> >> struct ARROW_FLIGHT_SQL_EXPORT TableStruct_FlightSql_2eproto {
>> >> ^
>> >>
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:46:8:
>> >> note: forward declaration of 'ARROW_FLIGHT_SQL_EXPORT'
>> >> struct ARROW_FLIGHT_SQL_EXPORT TableStruct_FlightSql_2eproto {
>> >> ^
>> >>
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:49:73:
>> >> error: no member named 'DescriptorTable' in namespace
>> >> 'google::protobuf::internal'
>> >> ARROW_FLIGHT_SQL_EXPORT extern const
>> >> ::PROTOBUF_NAMESPACE_ID::internal::DescriptorTable
>> >> descriptor_table_FlightSql_2eproto;
>> >>
>> >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
>> >>
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:49:88:
>> >> error: expected ';' after top level declarator
>> >> ARROW_FLIGHT_SQL_EXPORT extern const
>> >> ::PROTOBUF_NAMESPACE_ID::internal::DescriptorTable
>> >> descriptor_table_FlightSql_2eproto;
>> >> [...]
>> >>
>> >>
>> >> Any insight or suggestions are welcome here!
>> >>
>> >> Cheers
>> >> Paul
>>
