Hi, I have exactly the same issue, did you find any solution ?
Thank you
Le vendredi 26 mai 2023 à 21:17:54 UTC+2, sanjay...@google.com a écrit :
> I looked at the logs and I can confirm that the client is not using mTLS
> because Istio didn't provide the right configuration. Let me explain:
>
> From your server-mtls-strict.log I see this
>
> "transportSocket": {
> "name": "envoy.transport_sockets.tls",
> "typedConfig": {
> "@type": "
> type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
> ",
> "commonTlsContext": {
> "combinedValidationContext": {
> "defaultValidationContext": {
> },
> "validationContextCertificateProviderInstance": {
> "instanceName": "default",
> "certificateName": "ROOTCA"
> }
> },
> "tlsCertificateCertificateProviderInstance": {
> "instanceName": "default",
> "certificateName": "default"
> }
> },
> "requireClientCertificate": true
> }
> },
> "name": "inbound-mtls"
>
> In the Listener resource received (see timestamp 16:47:54.070)
>
> However I don't see a similar security configuration on the client side -
> which is a bit complex I admin. In the client-mtls-strict.log I see a
> Cluster resource response:
>
> {
> "versionInfo": "2023-05-26T16:47:41Z/749",
> "resources": [{
> "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
> "name": "outbound|8443||server.kotlin-grpc-xds.svc.cluster.local",
> "type": "EDS",
> "edsClusterConfig": {
> "edsConfig": {
> "ads": {
> }
> },
> "serviceName":
> "outbound|8443||server.kotlin-grpc-xds.svc.cluster.local"
> }
> }],
> "typeUrl": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
> "nonce": "bd0fe5d1-9cb8-4342-a15f-d5d834dd3255",
> "controlPlane": {
> "identifier":
> "{\"Component\":\"istiod\",\"ID\":\"istiod-7fd9d6dd48-nf42k\",\"Info\":{\"version\":\"1.17.2\",\"revision\":\"3e857775086a061d12ee445f32a0b35ea17c8488\",\"golang_version\":\"go1.20.2\",\"status\":\"Clean\",\"tag\":\"1.17.2\"}}"
> }
> }
>
> which has no security/mTLS configuration. I would expect a config similar
> to the transportSocket snippet I described above.
>
> So the error is on the Istio side: whether a bug in Istio or a user-error
> I can't tell. I hope you are able to figure it out?
>
>
>
> On Friday, May 26, 2023 at 10:53:34 AM UTC-7 Wesley Hartford wrote:
>
>> I've attached a copy of the log files with xds logging set to trace for
>> an execution of the client and server with istio's mtls mode set to STRICT
>> and PERMISSIVE. My interpretation of these logs is:
>>
>> In PERMISSIVE mode, neither client nor server is trying to use any type
>> of TLS; they're both using plain text and can interact just fine. I was
>> under the impression that PERMISSIVE mode would use mTLS, but reading the
>> istio docs, it's a little ambiguous, so I guess this is a correct behaviour.
>>
>> In STRICT mode, it looks like the server is using the supplied TLS key
>> material to accept mTLS connections, but it doesn't look like the client is
>> making any attempt to use TLS.
>>
>> I'm working on your other suggestions, but thought I should update you
>> with the logs before spending too much time on them.
>>
>> On Wed, May 24, 2023 at 11:22 PM Sanjay Pujare <sanjay...@google.com>
>> wrote:
>>
>>> (adding grpc.io group back)
>>>
>>> On Wed, May 24, 2023 at 2:57 PM Wesley Hartford <wes...@cutterslade.ca>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> My suggestion that the connection was falling back to insecure was not
>>>> evidence based, I'm still trying to wrap my head around how all this is
>>>> working.
>>>>
>>>
>>> Okay.
>>>
>>>
>>>>
>>>> The target address on the client side is using the xds:/// prefix.
>>>>
>>>> I've enabled trace level logging on the io.grpc.xds logger but I'm not
>>>> seeing any additional log messages, have I missed something? I'm using
>>>> slf4j and logback and have the SLF4JBridgeHandler installed.
>>>>
>>>
>>> The code uses Java util logging so you can just something like
>>> -Djava.util.logging.config.file=/logging.properties in your Java
>>> invocation command line and enable the logger for io.grpc.xds .
>>>
>>>
>>>>
>>>> The grpc-bootstrap.json file seems reasonable, though I don't know just
>>>> what it all means (I've attached the content). The three pem files
>>>> referenced in certificate_providers point to real files containing
>>>> apparently valid PEM content.
>>>>
>>>> You've mentioned a couple times enabling vs. disabling mTLS, are you
>>>> referring to some specific setting on the client and/or server, or in
>>>> istio
>>>> somewhere? My understanding has been that with the Xds server and channel,
>>>> both will use mTLS unless I specifically set the mtls mode to DISABLE in a
>>>> PeerAuthentication resource, which I haven't done. I've experimented with
>>>> mtls mode set to PERMISSIVE and STRICT. Is the problem something as simple
>>>> as not enabling mTLS somewhere?
>>>>
>>>
>>> In your first post you said "I got everything working without too much
>>> trouble ... ... When I configure Istio to enforce STRICT mTLS..." I got
>>> the impression that you got everything working in plaintext (without
>>> enabling mTLS) and then you enabled mTLS via Istio which is when you saw
>>> connection problems. I was just referring to the same - you enable mTLS in
>>> Istio security policy.
>>>
>>> I suggest couple of additional tests to troubleshoot this:
>>>
>>> - instead of using Xds Channel and Server credentials use the Tls
>>> Channel and Server credentials with the same files provided by Istio (under
>>> /var/lib/istio/data). You can check out the doc at
>>> https://grpc.github.io/grpc-java/javadoc/io/grpc/TlsChannelCredentials.html
>>> and
>>> https://grpc.github.io/grpc-java/javadoc/io/grpc/TlsServerCredentials.html
>>> . Note that even if you are using XDS for load balancing, service discovery
>>> you can use Tls credentials for security.
>>>
>>> - if you can try a Golang (or C++) example with your Istio setup and
>>> verify mTLS is working with those examples.
>>>
>>> Hope that helps.
>>>
>>>
>>>
>>>>
>>>> Thanks again,
>>>>
>>>> Wesley
>>>>
>>>> On Wed, May 24, 2023 at 10:27 AM 'sanjay...@google.com' via grpc.io <
>>>> grp...@googlegroups.com> wrote:
>>>>
>>>>> On Wednesday, May 24, 2023 at 8:41:20 AM UTC-7 Wesley Hartford wrote:
>>>>>
>>>>> Thanks for getting back to me, Sanjay. As far as I can tell, my client
>>>>> and server are both using the appropriate Xds credentials:
>>>>> The client code is at
>>>>> https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/client/src/main/kotlin/ca/cutterslade/kotlingrpcxds/client/main.kt#L26
>>>>> Grpc.newChannelBuilder(targetUrl,
>>>>> XdsChannelCredentials.create(InsecureChannelCredentials.create())).build()
>>>>>
>>>>> The server code is at
>>>>> https://github.com/wfhartford/kotlin-grpc-xds/blob/18598a7e9210be7265bc753b136cb424d087ab77/server/src/main/kotlin/ca/cutterslade/kotlingrpcxds/server/main.kt#L45
>>>>> XdsServerBuilder.forPort(8443,
>>>>> XdsServerCredentials.create(InsecureServerCredentials.create()))
>>>>>
>>>>> The insecure credentials provided to both a fallback, and it looks
>>>>> like the sample you linked is doing the same thing.
>>>>>
>>>>>
>>>>> Yes, you are right - Xds credentials are being correctly used so
>>>>> that's not an issue.
>>>>>
>>>>> I'm not sure why, but I'm guessing that the secure connection is
>>>>> failing and it is falling back to insecure.
>>>>>
>>>>>
>>>>> No, that cannot be the case. As described here
>>>>> https://github.com/grpc/proposal/blob/master/A29-xds-tls-security.md#programming-api
>>>>>
>>>>> fallback credentials are *not* meant to be used when secure connection
>>>>> fails. If you suspect that's happening can you provide some evidence
>>>>> (logs
>>>>> etc)?
>>>>>
>>>>>
>>>>> Based on the example you linked, the only other requirement is that
>>>>> the GRPC_XDS_BOOTSTRAP environment variable is set, which is being
>>>>> done by the istio sidecar; kubectl describe pod shows that both the
>>>>> client and server containers have two environment variables injected:
>>>>> GRPC_XDS_EXPERIMENTAL_SECURITY_SUPPORT: true
>>>>> GRPC_XDS_BOOTSTRAP:
>>>>> /etc/istio/proxy/grpc-bootstrap.json
>>>>>
>>>>>
>>>>> The bootstrap file (and the GRPC_XDS_BOOTSTRAP environment variable)
>>>>> is required not just for security but overall XDS support. Based on your
>>>>> observations in the first email it looks like client and server are able
>>>>> to
>>>>> communicate based on xds configuration so that part must be working. Just
>>>>> to confirm your client did use the "xds:///" prefix to access your
>>>>> service,
>>>>> right?
>>>>>
>>>>> If you are certain that XDS is working for load balancing and
>>>>> plaintext communication but breaks only when mTLS is enabled it might be
>>>>> useful to enable debug logging on both client and server side (for the
>>>>> io.grpc.xds package and below).
>>>>>
>>>>> Also you can check the contents of
>>>>> /etc/istio/proxy/grpc-bootstrap.json esp. the certificate_providers
>>>>> part and confirm that the 3 files provided in the config have valid
>>>>> certificate material. Let me know if you need help debugging that part.
>>>>>
>>>>>
>>>>>
>>>>> There are only two warning lines being logged from both the client and
>>>>> the server:
>>>>>
>>>>> 14:51:55.314 [main] WARN i.g.n.s.io.netty.bootstrap.Bootstrap -
>>>>> Unknown channel option 'SO_KEEPALIVE' for channel '[id: 0xba433026]'
>>>>> 14:51:55.314 [main] WARN i.g.n.s.io.netty.bootstrap.Bootstrap -
>>>>> Unknown channel option
>>>>> 'io.grpc.netty.shaded.io.netty.channel.epoll.EpollChannelOption#TCP_USER_TIMEOUT'
>>>>>
>>>>> for channel '[id: 0xba433026]'
>>>>>
>>>>>
>>>>> I doubt this is an issue. Just to confirm you see these messages also
>>>>> when mTLS is not enabled, right?
>>>>>
>>>>>
>>>>>
>>>>> Do you know of anything else I might be missing that is required for a
>>>>> secure connection?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Wesley
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to a topic in the
>>>>> Google Groups "grpc.io" group.
>>>>> To unsubscribe from this topic, visit
>>>>> https://groups.google.com/d/topic/grpc-io/e20VVBIPd7M/unsubscribe.
>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>> grpc-io+u...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/grpc-io/5a062461-2c73-4475-b99c-ddfe43a569e6n%40googlegroups.com
>>>>>
>>>>> <https://groups.google.com/d/msgid/grpc-io/5a062461-2c73-4475-b99c-ddfe43a569e6n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>>
--
You received this message because you are subscribed to the Google Groups
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion visit
https://groups.google.com/d/msgid/grpc-io/1812e95f-ab1c-4f21-a17b-b42e458bd3d3n%40googlegroups.com.
