I finally find the solution after some debugging hours.
Istio provides by default a RSA key, when the grpc-java lib is waiting for
a PKCS#8 key.
Add environment variable `PKCS8_KEY=true` to the istio proxy solved the
issue.
Hope this help.
Le jeudi 20 mars 2025 à 16:01:19 UTC+1, Xavier XYZ a écrit :
> Hi, I have exactly the same issue, did you find any solution ?
>
> Thank you
>
> Le vendredi 26 mai 2023 à 21:17:54 UTC+2, sanjay...@google.com a écrit :
>
>> I looked at the logs and I can confirm that the client is not using mTLS
>> because Istio didn't provide the right configuration. Let me explain:
>>
>> From your server-mtls-strict.log I see this
>>
>> "transportSocket": {
>> "name": "envoy.transport_sockets.tls",
>> "typedConfig": {
>> "@type": "
>> type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
>> ",
>> "commonTlsContext": {
>> "combinedValidationContext": {
>> "defaultValidationContext": {
>> },
>> "validationContextCertificateProviderInstance": {
>> "instanceName": "default",
>> "certificateName": "ROOTCA"
>> }
>> },
>> "tlsCertificateCertificateProviderInstance": {
>> "instanceName": "default",
>> "certificateName": "default"
>> }
>> },
>> "requireClientCertificate": true
>> }
>> },
>> "name": "inbound-mtls"
>>
>> In the Listener resource received (see timestamp 16:47:54.070)
>>
>> However I don't see a similar security configuration on the client side -
>> which is a bit complex I admin. In the client-mtls-strict.log I see a
>> Cluster resource response:
>>
>> {
>> "versionInfo": "2023-05-26T16:47:41Z/749",
>> "resources": [{
>> "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
>> "name": "outbound|8443||server.kotlin-grpc-xds.svc.cluster.local",
>> "type": "EDS",
>> "edsClusterConfig": {
>> "edsConfig": {
>> "ads": {
>> }
>> },
>> "serviceName":
>> "outbound|8443||server.kotlin-grpc-xds.svc.cluster.local"
>> }
>> }],
>> "typeUrl": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
>> "nonce": "bd0fe5d1-9cb8-4342-a15f-d5d834dd3255",
>> "controlPlane": {
>> "identifier":
>> "{\"Component\":\"istiod\",\"ID\":\"istiod-7fd9d6dd48-nf42k\",\"Info\":{\"version\":\"1.17.2\",\"revision\":\"3e857775086a061d12ee445f32a0b35ea17c8488\",\"golang_version\":\"go1.20.2\",\"status\":\"Clean\",\"tag\":\"1.17.2\"}}"
>> }
>> }
>>
>> which has no security/mTLS configuration. I would expect a config similar
>> to the transportSocket snippet I described above.
>>
>> So the error is on the Istio side: whether a bug in Istio or a user-error
>> I can't tell. I hope you are able to figure it out?
>>
>>
>>
>> On Friday, May 26, 2023 at 10:53:34 AM UTC-7 Wesley Hartford wrote:
>>
>>> I've attached a copy of the log files with xds logging set to trace for
>>> an execution of the client and server with istio's mtls mode set to STRICT
>>> and PERMISSIVE. My interpretation of these logs is:
>>>
>>> In PERMISSIVE mode, neither client nor server is trying to use any type
>>> of TLS; they're both using plain text and can interact just fine. I was
>>> under the impression that PERMISSIVE mode would use mTLS, but reading the
>>> istio docs, it's a little ambiguous, so I guess this is a correct behaviour.
>>>
>>> In STRICT mode, it looks like the server is using the supplied TLS key
>>> material to accept mTLS connections, but it doesn't look like the client is
>>> making any attempt to use TLS.
>>>
>>> I'm working on your other suggestions, but thought I should update you
>>> with the logs before spending too much time on them.
>>>
>>> On Wed, May 24, 2023 at 11:22 PM Sanjay Pujare <sanjay...@google.com>
>>> wrote:
>>>
>>>> (adding grpc.io group back)
>>>>
>>>> On Wed, May 24, 2023 at 2:57 PM Wesley Hartford <wes...@cutterslade.ca>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> My suggestion that the connection was falling back to insecure was not
>>>>> evidence based, I'm still trying to wrap my head around how all this is
>>>>> working.
>>>>>
>>>>
>>>> Okay.
>>>>
>>>>
>>>>>
>>>>> The target address on the client side is using the xds:/// prefix.
>>>>>
>>>>> I've enabled trace level logging on the io.grpc.xds logger but I'm not
>>>>> seeing any additional log messages, have I missed something? I'm using
>>>>> slf4j and logback and have the SLF4JBridgeHandler installed.
>>>>>
>>>>
>>>> The code uses Java util logging so you can just something like
>>>> -Djava.util.logging.config.file=/logging.properties in your Java
>>>> invocation command line and enable the logger for io.grpc.xds .
>>>>
>>>>
>>>>>
>>>>> The grpc-bootstrap.json file seems reasonable, though I don't know
>>>>> just what it all means (I've attached the content). The three pem files
>>>>> referenced in certificate_providers point to real files containing
>>>>> apparently valid PEM content.
>>>>>
>>>>> You've mentioned a couple times enabling vs. disabling mTLS, are you
>>>>> referring to some specific setting on the client and/or server, or in
>>>>> istio
>>>>> somewhere? My understanding has been that with the Xds server and
>>>>> channel,
>>>>> both will use mTLS unless I specifically set the mtls mode to DISABLE in
>>>>> a
>>>>> PeerAuthentication resource, which I haven't done. I've experimented with
>>>>> mtls mode set to PERMISSIVE and STRICT. Is the problem something as
>>>>> simple
>>>>> as not enabling mTLS somewhere?
>>>>>
>>>>
>>>> In your first post you said "I got everything working without too much
>>>> trouble ... ... When I configure Istio to enforce STRICT mTLS..." I
>>>> got the impression that you got everything working in plaintext (without
>>>> enabling mTLS) and then you enabled mTLS via Istio which is when you saw
>>>> connection problems. I was just referring to the same - you enable mTLS in
>>>> Istio security policy.
>>>>
>>>> I suggest couple of additional tests to troubleshoot this:
>>>>
>>>> - instead of using Xds Channel and Server credentials use the Tls
>>>> Channel and Server credentials with the same files provided by Istio
>>>> (under
>>>> /var/lib/istio/data). You can check out the doc at
>>>> https://grpc.github.io/grpc-java/javadoc/io/grpc/TlsChannelCredentials.html
>>>>
>>>> and
>>>> https://grpc.github.io/grpc-java/javadoc/io/grpc/TlsServerCredentials.html
>>>> . Note that even if you are using XDS for load balancing, service
>>>> discovery
>>>> you can use Tls credentials for security.
>>>>
>>>> - if you can try a Golang (or C++) example with your Istio setup and
>>>> verify mTLS is working with those examples.
>>>>
>>>> Hope that helps.
>>>>
>>>>
>>>>
>>>>>
>>>>> Thanks again,
>>>>>
>>>>> Wesley
>>>>>
>>>>> On Wed, May 24, 2023 at 10:27 AM 'sanjay...@google.com' via grpc.io <
>>>>> grp...@googlegroups.com> wrote:
>>>>>
>>>>>> On Wednesday, May 24, 2023 at 8:41:20 AM UTC-7 Wesley Hartford wrote:
>>>>>>
>>>>>> Thanks for getting back to me, Sanjay. As far as I can tell, my
>>>>>> client and server are both using the appropriate Xds credentials:
>>>>>> The client code is at
>>>>>> https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/client/src/main/kotlin/ca/cutterslade/kotlingrpcxds/client/main.kt#L26
>>>>>> Grpc.newChannelBuilder(targetUrl,
>>>>>> XdsChannelCredentials.create(InsecureChannelCredentials.create())).build()
>>>>>>
>>>>>> The server code is at
>>>>>> https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/server/src/main/kotlin/ca/cutterslade/kotlingrpcxds/server/main.kt#L45
>>>>>> XdsServerBuilder.forPort(8443,
>>>>>> XdsServerCredentials.create(InsecureServerCredentials.create()))
>>>>>>
>>>>>> The insecure credentials provided to both a fallback, and it looks
>>>>>> like the sample you linked is doing the same thing.
>>>>>>
>>>>>>
>>>>>> Yes, you are right - Xds credentials are being correctly used so
>>>>>> that's not an issue.
>>>>>>
>>>>>> I'm not sure why, but I'm guessing that the secure connection is
>>>>>> failing and it is falling back to insecure.
>>>>>>
>>>>>>
>>>>>> No, that cannot be the case. As described here
>>>>>> https://github.com/grpc/proposal/blob/master/A29-xds-tls-security.md#programming-api
>>>>>>
>>>>>> fallback credentials are *not* meant to be used when secure connection
>>>>>> fails. If you suspect that's happening can you provide some evidence
>>>>>> (logs
>>>>>> etc)?
>>>>>>
>>>>>>
>>>>>> Based on the example you linked, the only other requirement is that
>>>>>> the GRPC_XDS_BOOTSTRAP environment variable is set, which is being
>>>>>> done by the istio sidecar; kubectl describe pod shows that both the
>>>>>> client and server containers have two environment variables injected:
>>>>>> GRPC_XDS_EXPERIMENTAL_SECURITY_SUPPORT: true
>>>>>> GRPC_XDS_BOOTSTRAP:
>>>>>> /etc/istio/proxy/grpc-bootstrap.json
>>>>>>
>>>>>>
>>>>>> The bootstrap file (and the GRPC_XDS_BOOTSTRAP environment variable)
>>>>>> is required not just for security but overall XDS support. Based on your
>>>>>> observations in the first email it looks like client and server are able
>>>>>> to
>>>>>> communicate based on xds configuration so that part must be working.
>>>>>> Just
>>>>>> to confirm your client did use the "xds:///" prefix to access your
>>>>>> service,
>>>>>> right?
>>>>>>
>>>>>> If you are certain that XDS is working for load balancing and
>>>>>> plaintext communication but breaks only when mTLS is enabled it might be
>>>>>> useful to enable debug logging on both client and server side (for the
>>>>>> io.grpc.xds package and below).
>>>>>>
>>>>>> Also you can check the contents of
>>>>>> /etc/istio/proxy/grpc-bootstrap.json esp. the certificate_providers
>>>>>> part and confirm that the 3 files provided in the config have valid
>>>>>> certificate material. Let me know if you need help debugging that part.
>>>>>>
>>>>>>
>>>>>>
>>>>>> There are only two warning lines being logged from both the client
>>>>>> and the server:
>>>>>>
>>>>>> 14:51:55.314 [main] WARN i.g.n.s.io.netty.bootstrap.Bootstrap -
>>>>>> Unknown channel option 'SO_KEEPALIVE' for channel '[id: 0xba433026]'
>>>>>> 14:51:55.314 [main] WARN i.g.n.s.io.netty.bootstrap.Bootstrap -
>>>>>> Unknown channel option
>>>>>> 'io.grpc.netty.shaded.io.netty.channel.epoll.EpollChannelOption#TCP_USER_TIMEOUT'
>>>>>>
>>>>>> for channel '[id: 0xba433026]'
>>>>>>
>>>>>>
>>>>>> I doubt this is an issue. Just to confirm you see these messages also
>>>>>> when mTLS is not enabled, right?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Do you know of anything else I might be missing that is required for
>>>>>> a secure connection?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Wesley
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to a topic in
>>>>>> the Google Groups "grpc.io" group.
>>>>>> To unsubscribe from this topic, visit
>>>>>> https://groups.google.com/d/topic/grpc-io/e20VVBIPd7M/unsubscribe.
>>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>>> grpc-io+u...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/grpc-io/5a062461-2c73-4475-b99c-ddfe43a569e6n%40googlegroups.com
>>>>>>
>>>>>> <https://groups.google.com/d/msgid/grpc-io/5a062461-2c73-4475-b99c-ddfe43a569e6n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>>
--
You received this message because you are subscribed to the Google Groups
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion visit
https://groups.google.com/d/msgid/grpc-io/89753236-45ff-4f36-8367-20bab2bc5463n%40googlegroups.com.
