Hi Vijay, I’m pretty sure that this should work with the properties that you provided, unless the AWS Kinesis SDK isn’t working as expected.
What I’ve tested is that with those properties, the ClientConfiguration used to build the Kinesis client has the proxy domain / host / ports etc. properly set. And according to [1], this should be enough to configure the constructed Kinesis client to connect via the proxy. Cheers, Gordon [1] https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/section-client-configuration.html On 7 November 2018 at 1:19:02 AM, Vijay Balakrishnan (bvija...@gmail.com) wrote: Hi Gordon, This still didn't work :( Tried a few combinations with: kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + "proxyDomain", "..."); inesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + "proxyHost", "http://.....com";); kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + "proxyPort", "911"); kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + "proxyUsername", "..."); kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + "proxyPassword", ".."); kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + "nonProxyHosts", " How does the FlinkKinesisProducer work so seamlessly through a proxy ? TIA, Vijay On Thu, Oct 4, 2018 at 6:41 AM Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote: Hi, Since Flink 1.5, you should be able to set all available configurations on the ClientConfiguration through the consumer Properties (see FLINK-9188 [1]). The way to do that would be to prefix the configuration you want to set with "aws.clientconfig" and add that to the properties, as such: ``` Properties kinesisConsumerProps = new Properties(); kinesisConsumerProps.setProperty("aws.clientconfig.proxyHost", ...); kinesisConsumerProps.setProperty("aws.clientconfig.proxyPort", ...); kinesisConsumerProps.setProperty("aws.clientconfig.proxyUsert", ...); ... ``` Could you try that out and see if it works for you? I've also realized that this feature isn't documented very well, and have opened a ticket for that [2]. Cheers, Gordon [1] https://issues.apache.org/jira/browse/FLINK-9188 [2] https://issues.apache.org/jira/browse/FLINK-10492 On Thu, Oct 4, 2018, 7:57 PM Aljoscha Krettek <aljos...@apache.org> wrote: Hi, I'm looping in Gordon and Thomas, they might have some idea about how to resolve this. Best, Aljoscha On 3. Oct 2018, at 17:29, Vijay Balakrishnan <bvija...@gmail.com> wrote: I have been trying with all variations to no avail of java -Dhttp.nonProxyHosts=.. -Dhttps.proxyHost=http://... -Dhttps.proxyPort=911 -Dhttps.proxyUser= -Dhttps.proxyPassword=.. -Dhttp.proxyHost=http://.. -Dhttp.proxyPort=911 -Dhttp.proxyUser=... -Dhttp.proxyPassword=... -jar .. after looking at the code in com.amazonaws.ClientConfiguration On Tue, Oct 2, 2018 at 3:49 PM Vijay Balakrishnan <bvija...@gmail.com> wrote: HI, How do I use FlinkKinesisConsumer using the Properties through a proxy ? Getting a Connection issue through the proxy. Works outside the proxy. Properties kinesisConsumerConfig = new Properties(); kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_REGION, region); if (local) { kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_ACCESS_KEY_ID, accessKey); kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_SECRET_ACCESS_KEY, secretKey); } else { kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_CREDENTIALS_PROVIDER, "AUTO"); } //only for Consumer kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_MAX, "10000"); kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS, "2000"); FlinkKinesisConsumer<Tuple2<InputMetadata, CameraWithCube>> kinesisConsumer = new FlinkKinesisConsumer<>( "kinesisTopicRead", new Tuple2KinesisSchema(), kinesisConsumerConfig); TIA
