Hello, Apache Beam community! Hope everything goes ok at this time.
I write to you asking for your guide and help. I have been facing some problems accessing HTTPS resources from a pipeline deployed in Dataflow. The problem occurs only when I run with DataflowRunner, the DirectRunner is working ok. The associated exception is: handshake_failure. Stack [sun.security.ssl.Alerts.getSSLException(Alerts.java:192), sun.security.ssl.Alerts.getSSLException(Alerts.java:154), sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2033), sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1135), sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385), sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) I found this post at Stackoverflow which describes my problem perfectly: https://stackoverflow.com/questions/59128640/sslhandshakeexception-when-running-apache-beam-pipeline-in-dataflow, but the proposed solution doesn't work for me. Maybe I'm missing something. Basically I changed the Apache HttpClient to the Google HttpClient, but it still isn't working. Here is the code I've tried. *private* *void* outputProcess(ProcessContext pc, T img) *throws* Exception { StomachStreamerOptions opts = pc .getPipelineOptions().as(StomachStreamerOptions.*class*); NetHttpTransport.Builder builder = *new* NetHttpTransport.Builder(); builder.setSslSocketFactory(*createSslSocketFactory*()); JsonFactory JSON_FACTORY = *new* JacksonFactory(); String urlImage = img.getURLImage(); HttpRequestFactory requestFactory = builder.build().createRequestFactory( *new* HttpRequestInitializer() { @Override *public* *void* initialize(HttpRequest request) { request.setParser(*new* JsonObjectParser(JSON_FACTORY)); } }); GenericUrl url = *new* GenericUrl(urlImage); HttpRequest request = requestFactory.buildGetRequest(url); HttpHeaders headers = *new* HttpHeaders(); headers.setUserAgent(opts.getUserAgentImages()); request.setHeaders(headers); *try* { HttpResponse res = request.execute(); *this*.outputProcess(pc, img, res); } *catch* (Exception e) { reportError(e); } } *private* *static* SSLSocketFactory createSslSocketFactory() *throws* Exception { TrustManager[] byPassTrustManagers = *new* TrustManager[]{*new* X509TrustManager() { *public* X509Certificate[] getAcceptedIssuers() { *return* *new* X509Certificate[0]; } *public* *void* checkClientTrusted(X509Certificate[] chain, String authType) { } *public* *void* checkServerTrusted(X509Certificate[] chain, String authType) { } }}; SSLContext sslContext = SSLContext.*getInstance*("TLSv1.2"); sslContext.init(*null*, byPassTrustManagers, *new* SecureRandom()); *return* sslContext.getSocketFactory(); } Things I've tried: - HttpClient Apache 4.5.1 - HttpClient Google 1.11 - Skip the certificate validation (permit all, just for testing. Not working.) I'm starting to think that maybe the problem is not on my code. Maybe is beyond; the infrastructure, the CIPHER_SUITE is not allowed by Dataflow... I don't know, just guess. What I'm using: - Java 8 (1.8.0_144) - Apache Beam 2.9 (Jeje, I'm planning to upgrade it soon) Thanks for reading my question. I will appreciate any idea that can put me in the correct direction. Andy