Hi Gordon, Finally figured out my issue.Do not need to add http:// in proxyHost name. String proxyHost= "proxy-chain....com";//not http://proxy-chain...com kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + "proxyHost", proxyHost);//<== mo http:// in proxyHost name
TIA, Vijay On Wed, Nov 14, 2018 at 12:50 AM Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote: > Hi Vijay, > > I’m pretty sure that this should work with the properties that you > provided, unless the AWS Kinesis SDK isn’t working as expected. > > What I’ve tested is that with those properties, the ClientConfiguration > used to build the Kinesis client has the proxy domain / host / ports etc. > properly set. > And according to [1], this should be enough to configure the constructed > Kinesis client to connect via the proxy. > > Cheers, > Gordon > > [1] > https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/section-client-configuration.html > > > On 7 November 2018 at 1:19:02 AM, Vijay Balakrishnan (bvija...@gmail.com) > wrote: > > Hi Gordon, > This still didn't work :( > > Tried a few combinations with: > kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + > "proxyDomain", "..."); > > inesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + > "proxyHost", "http://.....com";); > > kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + > "proxyPort", "911"); > > kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + > "proxyUsername", "..."); > > kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + > "proxyPassword", ".."); > > kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + > "nonProxyHosts", " > > > How does the FlinkKinesisProducer work so seamlessly through a proxy ? > TIA, > Vijay > > On Thu, Oct 4, 2018 at 6:41 AM Tzu-Li (Gordon) Tai <tzuli...@apache.org> > wrote: > >> Hi, >> >> Since Flink 1.5, you should be able to set all available configurations >> on the ClientConfiguration through the consumer Properties (see FLINK-9188 >> [1]). >> >> The way to do that would be to prefix the configuration you want to set >> with "aws.clientconfig" and add that to the properties, as such: >> >> ``` >> Properties kinesisConsumerProps = new Properties(); >> kinesisConsumerProps.setProperty("aws.clientconfig.proxyHost", ...); >> kinesisConsumerProps.setProperty("aws.clientconfig.proxyPort", ...); >> kinesisConsumerProps.setProperty("aws.clientconfig.proxyUsert", ...); >> ... >> ``` >> >> Could you try that out and see if it works for you? >> >> I've also realized that this feature isn't documented very well, and have >> opened a ticket for that [2]. >> >> Cheers, >> Gordon >> >> [1] https://issues.apache.org/jira/browse/FLINK-9188 >> [2] https://issues.apache.org/jira/browse/FLINK-10492 >> >> On Thu, Oct 4, 2018, 7:57 PM Aljoscha Krettek <aljos...@apache.org> >> wrote: >> >>> Hi, >>> >>> I'm looping in Gordon and Thomas, they might have some idea about how to >>> resolve this. >>> >>> Best, >>> Aljoscha >>> >>> On 3. Oct 2018, at 17:29, Vijay Balakrishnan <bvija...@gmail.com> wrote: >>> >>> I have been trying with all variations to no avail of java >>> -Dhttp.nonProxyHosts=.. -Dhttps.proxyHost=http://... >>> -Dhttps.proxyPort=911 -Dhttps.proxyUser= -Dhttps.proxyPassword=.. >>> -Dhttp.proxyHost=http://.. -Dhttp.proxyPort=911 -Dhttp.proxyUser=... >>> -Dhttp.proxyPassword=... -jar .. after looking at the code in >>> com.amazonaws.ClientConfiguration >>> >>> On Tue, Oct 2, 2018 at 3:49 PM Vijay Balakrishnan <bvija...@gmail.com> >>> wrote: >>> >>>> HI, >>>> How do I use FlinkKinesisConsumer using the Properties through a proxy >>>> ? Getting a Connection issue through the proxy. >>>> Works outside the proxy. >>>> >>>> Properties kinesisConsumerConfig = new Properties(); >>>> >>>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_REGION, region); >>>> >>>> if (local) { >>>> >>>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_ACCESS_KEY_ID, >>>> accessKey); >>>> >>>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_SECRET_ACCESS_KEY, >>>> secretKey); >>>> } else { >>>> >>>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_CREDENTIALS_PROVIDER, >>>> "AUTO"); >>>> } >>>> >>>> //only for Consumer >>>> >>>> kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_MAX, >>>> "10000"); >>>> >>>> kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS, >>>> "2000"); >>>> FlinkKinesisConsumer<Tuple2<InputMetadata, CameraWithCube>> >>>> kinesisConsumer = new FlinkKinesisConsumer<>( >>>> "kinesisTopicRead", new Tuple2KinesisSchema(), >>>> kinesisConsumerConfig); >>>> TIA >>>> >>> >>>
