An update about this issue (also on JIRA at https://issues.apache.org/jira/browse/ARROW-10105) Hoping for some suggestions on how to deal with C++ issues with gRPC versioning.
I have the Java version of this feature working and tested. The C++ client has some challenges: The experimental TlsCredentials interface has an option to specify that you want to disable server certificate checks. You also must supply a callback that is supposed to represent how you the client should verify the server instead. There are two problems with this interface though: 1. The caller needs to supply a CA roots PEM file to this. This would be unexpected since the point of using this is to skip over passing in certificate information. Furthermore, the SslCredentials interface that's currently being used for SSL automatically chooses a default CA roots PEM file if none is supplied. gRPC ships one as part of its installation process that it uses. There does not appear to be a programmatic way to get to the path to the shipped with gRPC. We may need to supply a default/dummy PEM file for the purpose of skipping server cert verification. I've verified that if I do supply a PEM file, everything works correctly and the server verification gets skipped. The Python client changes associated with this also work. 2. The TlsCredentials class and related Options, etc classes are in a different namespace depending on gRPC version. It first shows up in gRPC 1.25 in the grpc_impl::experimental namespace. It got moved to grpc::experimental namespace starting in gRPC 1.32. What's problematic is that we can't get the gRPC version at compile-time, so we can't choose the appropriate namespace for looking these up. My preference is to just switch entirely to 1.32, however I'm seeing issues in some CI jobs that don't seem to correctly use 1.32 such as: https://ci.ursalabs.org/#/builders/66/builds/11440 The CentOS 5.11 build also can't seem to build gRPC 1.32. It fails compiling some code in linux/tcp.h, which may be related to using an older Linux kernel: https://github.com/apache/arrow/pull/8325/checks?check_run_id=1208358439): CMakeFiles/grpc_unsecure.dir/src/core/lib/iomgr/socket_utils_common_posix.cc.o -c src/core/lib/iomgr/socket_utils_common_posix.cc In file included from /usr/include/asm-x86_64/byteorder.h:30:0, from /usr/include/asm/byteorder.h:5, from /usr/include/linux/tcp.h:21, from src/core/lib/iomgr/socket_utils_common_posix.cc:34: /usr/include/linux/byteorder/little_endian.h:43:19: error: ‘__le64’ does not name a type static __inline__ __le64 __cpu_to_le64p(const __u64 *p) ^ /usr/include/linux/byteorder/little_endian.h:47:46: error: ‘__le64’ does not name a type static __inline__ __u64 __le64_to_cpup(const __le64 *p) ^ /usr/include/linux/byteorder/little_endian.h:67:19: error: ‘__be64’ does not name a type static __inline__ __be64 __cpu_to_be64p(const __u64 *p) ^ /usr/include/linux/byteorder/little_endian.h:71:46: error: ‘__be64’ does not name a type static __inline__ __u64 __be64_to_cpup(const __be64 *p) On Fri, Oct 2, 2020 at 5:36 AM David Li <li.david...@gmail.com> wrote: > Hey James, > > Where gRPC comes from depends on your setup; the C++ development > guide[1] describes it all, but I personally use Conda with conda-forge > which gets me gRPC 1.32. > > The dependency version again depends on where you're looking - for > bundled builds it's set in thirdparty/versions.txt[2], for the other > CI builds it's set in various config files. > > Best, > David > > [1]: https://arrow.apache.org/docs/developers/cpp/building.html > [2]: > https://arrow.apache.org/docs/developers/cpp/building.html#bundled-dependency-versions > > On 10/2/20, James Duong <jam...@bitquilltech.com> wrote: > > Hi David, > > > > I've created a WIP PR illustrating what I'm planning to do. However I'm > > still trying to figure out a few things: > > > > - In my environment, gRPC is coming from miniconda3 for C++, and seems > > to be an older version. In more recent versions, gRPC has an > > experimental > > API TlsCredentials, which has the option to disable cert validation. > Is > > it > > normal that this comes from miniconda, or is it just my setup? > > - See: > > > > > https://github.com/grpc/grpc/blob/a6a5cbfee2ff2709a8bc66c3d4b428db48ae46af/src/cpp/common/tls_credentials_options.cc#L284 > > . This shows up starting with 1.30.x > > - Where do we specify which version of gRPC we depend on? In java > > it's in maven and looks to be 1.30.2. > > - The Java implementation looks fairly straightforward. > > - Python implementation is pending. > > - Tests are pending. > > > > > > On Wed, Sep 30, 2020 at 8:12 AM James Duong <jam...@bitquilltech.com> > > wrote: > > > >> Will do David. > >> > >> I've created a JIRA for the TLS_SNI extension as well: > >> https://issues.apache.org/jira/browse/ARROW-10144 > >> > >> On Wed, Sep 30, 2020 at 7:52 AM David Li <li.david...@gmail.com> wrote: > >> > >>> Hi James, > >>> > >>> Please feel free to tag me as a reviewer if needed. > >>> > >>> Best, > >>> David > >>> > >>> On 9/28/20, James Duong <jam...@bitquilltech.com> wrote: > >>> > Hi Arrow-Dev, > >>> > > >>> > I've created a JIRA for supporting using Flight clients to connect to > >>> > Flight servers without verifying the server certificate. > >>> > > >>> > This is something I've seen done frequently for other connectivity > >>> > tools > >>> > such as JDBC drivers. > >>> > > >>> > I'll be looking to implement this one for the and hopefully get it in > >>> for > >>> > the next release: > >>> > > https://issues.apache.org/jira/plugins/servlet/mobile#issue/ARROW-10105 > >>> > > >>> > Related, I'm also planning to add support for using the TLS_SNI > >>> extension. > >>> > > >>> > >> > >> > >> -- > >> > >> *James Duong* > >> Lead Software Developer > >> Bit Quill Technologies Inc. > >> Direct: +1.604.562.6082 | jam...@bitquilltech.com > >> https://www.bitquilltech.com > >> > >> This email message is for the sole use of the intended recipient(s) and > >> may contain confidential and privileged information. Any unauthorized > >> review, use, disclosure, or distribution is prohibited. If you are not > >> the > >> intended recipient, please contact the sender by reply email and destroy > >> all copies of the original message. Thank you. > >> > > > > > > -- > > > > *James Duong* > > Lead Software Developer > > Bit Quill Technologies Inc. > > Direct: +1.604.562.6082 | jam...@bitquilltech.com > > https://www.bitquilltech.com > > > > This email message is for the sole use of the intended recipient(s) and > may > > contain confidential and privileged information. Any unauthorized > review, > > use, disclosure, or distribution is prohibited. If you are not the > > intended recipient, please contact the sender by reply email and destroy > > all copies of the original message. Thank you. > > > -- *James Duong* Lead Software Developer Bit Quill Technologies Inc. Direct: +1.604.562.6082 | jam...@bitquilltech.com https://www.bitquilltech.com This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure, or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. Thank you.
