For #1, can we embed a dummy root to make gRPC happy, since we're only using it in the case that we're disabling validation? (Also, is there a bug report for gRPC/would you like to file one?)
For #2, I think you can get cmake to pass -D flags, perhaps you can get it to define a symbol you can check (though that is maybe not a preferable solution). David On 10/5/20, James Duong <jam...@bitquilltech.com> wrote: > An update about this issue (also on JIRA at > https://issues.apache.org/jira/browse/ARROW-10105) > Hoping for some suggestions on how to deal with C++ issues with gRPC > versioning. > > I have the Java version of this feature working and tested. > > The C++ client has some challenges: > The experimental TlsCredentials interface has an option to specify that you > want to disable server certificate checks. You also must supply > a callback that is supposed to represent how you the client should verify > the server instead. There are two problems with this interface > though: > 1. The caller needs to supply a CA roots PEM file to this. This would be > unexpected since the point of using this is to skip over > passing in certificate information. Furthermore, the SslCredentials > interface that's currently being used for SSL automatically > chooses a default CA roots PEM file if none is supplied. gRPC ships one as > part of its installation process that it uses. > There does not appear to be a programmatic way to get to the path to the > shipped with gRPC. We may need to supply > a default/dummy PEM file for the purpose of skipping server cert > verification. I've verified that if I do supply > a PEM file, everything works correctly and the server verification gets > skipped. The Python client changes associated > with this also work. > > 2. The TlsCredentials class and related Options, etc classes are in a > different namespace depending on gRPC version. > It first shows up in gRPC 1.25 in the grpc_impl::experimental namespace. It > got moved to grpc::experimental namespace > starting in gRPC 1.32. What's problematic is that we can't get the gRPC > version at compile-time, so we can't choose > the appropriate namespace for looking these up. My preference is to just > switch entirely to 1.32, however I'm > seeing issues in some CI jobs that don't seem to correctly use 1.32 such > as: > https://ci.ursalabs.org/#/builders/66/builds/11440 > The CentOS 5.11 build also can't seem to build gRPC 1.32. It fails > compiling some code in linux/tcp.h, which > may be related to using an older Linux kernel: > https://github.com/apache/arrow/pull/8325/checks?check_run_id=1208358439): > > CMakeFiles/grpc_unsecure.dir/src/core/lib/iomgr/socket_utils_common_posix.cc.o > -c src/core/lib/iomgr/socket_utils_common_posix.cc > In file included from /usr/include/asm-x86_64/byteorder.h:30:0, > from /usr/include/asm/byteorder.h:5, > from /usr/include/linux/tcp.h:21, > from src/core/lib/iomgr/socket_utils_common_posix.cc:34: > /usr/include/linux/byteorder/little_endian.h:43:19: error: ‘__le64’ > does not name a type > static __inline__ __le64 __cpu_to_le64p(const __u64 *p) > ^ > /usr/include/linux/byteorder/little_endian.h:47:46: error: ‘__le64’ > does not name a type > static __inline__ __u64 __le64_to_cpup(const __le64 *p) > ^ > /usr/include/linux/byteorder/little_endian.h:67:19: error: ‘__be64’ > does not name a type > static __inline__ __be64 __cpu_to_be64p(const __u64 *p) > ^ > /usr/include/linux/byteorder/little_endian.h:71:46: error: ‘__be64’ > does not name a type > static __inline__ __u64 __be64_to_cpup(const __be64 *p) > > > > On Fri, Oct 2, 2020 at 5:36 AM David Li <li.david...@gmail.com> wrote: > >> Hey James, >> >> Where gRPC comes from depends on your setup; the C++ development >> guide[1] describes it all, but I personally use Conda with conda-forge >> which gets me gRPC 1.32. >> >> The dependency version again depends on where you're looking - for >> bundled builds it's set in thirdparty/versions.txt[2], for the other >> CI builds it's set in various config files. >> >> Best, >> David >> >> [1]: https://arrow.apache.org/docs/developers/cpp/building.html >> [2]: >> https://arrow.apache.org/docs/developers/cpp/building.html#bundled-dependency-versions >> >> On 10/2/20, James Duong <jam...@bitquilltech.com> wrote: >> > Hi David, >> > >> > I've created a WIP PR illustrating what I'm planning to do. However I'm >> > still trying to figure out a few things: >> > >> > - In my environment, gRPC is coming from miniconda3 for C++, and >> > seems >> > to be an older version. In more recent versions, gRPC has an >> > experimental >> > API TlsCredentials, which has the option to disable cert validation. >> Is >> > it >> > normal that this comes from miniconda, or is it just my setup? >> > - See: >> > >> > >> https://github.com/grpc/grpc/blob/a6a5cbfee2ff2709a8bc66c3d4b428db48ae46af/src/cpp/common/tls_credentials_options.cc#L284 >> > . This shows up starting with 1.30.x >> > - Where do we specify which version of gRPC we depend on? In java >> > it's in maven and looks to be 1.30.2. >> > - The Java implementation looks fairly straightforward. >> > - Python implementation is pending. >> > - Tests are pending. >> > >> > >> > On Wed, Sep 30, 2020 at 8:12 AM James Duong <jam...@bitquilltech.com> >> > wrote: >> > >> >> Will do David. >> >> >> >> I've created a JIRA for the TLS_SNI extension as well: >> >> https://issues.apache.org/jira/browse/ARROW-10144 >> >> >> >> On Wed, Sep 30, 2020 at 7:52 AM David Li <li.david...@gmail.com> >> >> wrote: >> >> >> >>> Hi James, >> >>> >> >>> Please feel free to tag me as a reviewer if needed. >> >>> >> >>> Best, >> >>> David >> >>> >> >>> On 9/28/20, James Duong <jam...@bitquilltech.com> wrote: >> >>> > Hi Arrow-Dev, >> >>> > >> >>> > I've created a JIRA for supporting using Flight clients to connect >> >>> > to >> >>> > Flight servers without verifying the server certificate. >> >>> > >> >>> > This is something I've seen done frequently for other connectivity >> >>> > tools >> >>> > such as JDBC drivers. >> >>> > >> >>> > I'll be looking to implement this one for the and hopefully get it >> >>> > in >> >>> for >> >>> > the next release: >> >>> > >> https://issues.apache.org/jira/plugins/servlet/mobile#issue/ARROW-10105 >> >>> > >> >>> > Related, I'm also planning to add support for using the TLS_SNI >> >>> extension. >> >>> > >> >>> >> >> >> >> >> >> -- >> >> >> >> *James Duong* >> >> Lead Software Developer >> >> Bit Quill Technologies Inc. >> >> Direct: +1.604.562.6082 | jam...@bitquilltech.com >> >> https://www.bitquilltech.com >> >> >> >> This email message is for the sole use of the intended recipient(s) >> >> and >> >> may contain confidential and privileged information. Any unauthorized >> >> review, use, disclosure, or distribution is prohibited. If you are >> >> not >> >> the >> >> intended recipient, please contact the sender by reply email and >> >> destroy >> >> all copies of the original message. Thank you. >> >> >> > >> > >> > -- >> > >> > *James Duong* >> > Lead Software Developer >> > Bit Quill Technologies Inc. >> > Direct: +1.604.562.6082 | jam...@bitquilltech.com >> > https://www.bitquilltech.com >> > >> > This email message is for the sole use of the intended recipient(s) and >> may >> > contain confidential and privileged information. Any unauthorized >> review, >> > use, disclosure, or distribution is prohibited. If you are not the >> > intended recipient, please contact the sender by reply email and >> > destroy >> > all copies of the original message. Thank you. >> > >> > > > -- > > *James Duong* > Lead Software Developer > Bit Quill Technologies Inc. > Direct: +1.604.562.6082 | jam...@bitquilltech.com > https://www.bitquilltech.com > > This email message is for the sole use of the intended recipient(s) and may > contain confidential and privileged information. Any unauthorized review, > use, disclosure, or distribution is prohibited. If you are not the > intended recipient, please contact the sender by reply email and destroy > all copies of the original message. Thank you. >
