Hi Jean-Baptiste, Thank you. I am currently prototyping this. I'd like to ask for advice on how to properly test that feature. Essentially, I need to set up a server, install a self-signed certificate there, and test the behavior. Does a test similar to TestRESTCatalog with a custom TLS-enabled Jetty server seem like the way to go? Or are there some existing HTTPS tests?
Regards, Vladimir On Wed, Nov 13, 2024 at 4:11 PM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Hi Vladimir, > > Personally, even testing "local" REST catalogs, I'm setting up SSL > certificates with a local CA, etc. It's not very painful. > > That said, I got your point, and I think we can update > > https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/HTTPClient.java > to add a flag to disable SSL certificate checks. > > I would propose to add: > > private static final String REST_SSL_DISABLE_CERTIFICATE_CHECK = > "rest.ssl.disable.cert.check"; > > and used this for HTTP5 client setup. > > Regards > JB > > On Wed, Nov 13, 2024 at 1:53 PM Vladimir Ozerov <voze...@querifylabs.com> > wrote: > > > > Hi, > > > > Currently, RESTCatalog can work either over HTTP or HTTPS. In the latter > case, Iceberg always performs a fully-fledged check of server certificate. > While it is expected in production environments, this is not necessarily ok > for nob-prod on-premises REST catalog deployments (DEV, QA, etc). > > > > Consider a data platform team that would like to evaluate a migration to > a REST catalog from, say, HMS. It might be very convenient to issue a > self-signed certificate and use it when communicating with the catalog. > However, the current implementation of RESTCatalog client doesn't allow > this, as there is no hook to override the behavior of instantiated HTTP5 > client. Note that the advice to switch to HTTP for non-prod deployment is > not valid in this case, because during testing teams would like to have > their deployments as close as possible to production, looking for some > unexpected issues (e.g., unexpected performance drop when enabling SSL, > etc). > > > > Many vendors allow to disable SSL certificate checks in some cases. > E.g., AWS S3 APIs allows this, Trino allows, etc. It might be convenient to > add a similar capability to Iceberg's HTTPClient as well. E.g., this might > be a property "rest.client.insecure-ssl" passed to the client. > > > > What do you think about this? Apologize if it was already discussed > elsewhere, I couldn't find any relevant discussions. > > > > Regards, > > -- > > Vladimir Ozerov > > Founder > > querifylabs.com > -- *Vladimir Ozerov* Founder querifylabs.com
