Hi Aihua, I was indeed suspecting that you had a custom RESTClient :-) Thanks for digging further and clarifying! Let me know if you need my help in adapting your code.
And with that: +1 (non-binding) from me as well. Thanks, Alex On Thu, Feb 27, 2025 at 6:45 PM Aihua Xu <aihu...@gmail.com> wrote: > Hi Alex, > > I checked our code further. We have an internal implementation > for RESTClient which needs an update after your change to place the token > in DefaultAuthSession and not pass through initHeaders anymore. The > existing code assumes the token coming from initHeaders and we need to make > the change to get the token from the headers of DefaultAuthSession. > > With that, +1 (non-binding) > > Thanks, > Aihua > > > > On Thu, Feb 27, 2025 at 9:26 AM Alex Dutra <alex.du...@dremio.com.invalid> > wrote: > >> Hi Aihua Xu, >> >> I reviewed your PR but without further details I do not agree with your >> change, and I am unable to reproduce the issue. Besides, we have unit >> tests that cover this extensively. >> >> Could you please provide a simple reproducer or a test case? >> >> Thanks! >> >> Alex >> >> On Thu, Feb 27, 2025 at 5:06 PM Aihua Xu <aihu...@gmail.com> wrote: >> >>> I'm running within our integration tests. I'm able to trace and find the >>> breaking change: https://github.com/apache/iceberg/pull/11992. >>> >>> Seems we need to make the following changes to pass the token: >>> https://github.com/apache/iceberg/pull/12415/files. Alex, can you help >>> take a look at this? >>> >>> if (hasCredential) { >>> authResponse = >>> OAuth2Util.fetchToken( >>> initClient, initHeaders, credential, scope, oauth2ServerUri, >>> optionalOAuthParams); >>> Map<String, String> authHeaders = >>> RESTUtil.merge(initHeaders, >>> OAuth2Util.authHeaders(authResponse.token())); >>> config = fetchConfig(initClient, authHeaders, props); >>> } else { >>> authResponse = null; >>> Map<String, String> authHeaders = >>> RESTUtil.merge(initHeaders, OAuth2Util.authHeaders(initToken)); >>> config = fetchConfig(initClient, authHeaders, props); >>> } >>> >>> >>> On Thu, Feb 27, 2025 at 12:46 AM Alex Dutra >>> <alex.du...@dremio.com.invalid> wrote: >>> >>>> Hi Aihua, >>>> >>>> I just tested 1.8.1 with Polaris OSS and I am not seeing anything >>>> different. Can you share your setup? >>>> >>>> Below is my Spark setup. >>>> >>>> Thanks, >>>> >>>> Alex >>>> >>>> ./gradlew run >>>> >>>> token=$(curl -s http://localhost:8181/api/catalog/v1/oauth/tokens \ >>>> --user root:<secret> \ >>>> -d grant_type=client_credentials \ >>>> -d scope=PRINCIPAL_ROLE:ALL | sed -n >>>> 's/.*"access_token":"\([^"]*\)".*/\1/p') >>>> >>>> curl -s -H "Authorization: Bearer ${token}" \ >>>> -H 'Accept: application/json' \ >>>> -H 'Content-Type: application/json' \ >>>> http://localhost:8181/api/management/v1/catalogs \ >>>> -d '{ >>>> "catalog": { >>>> "name": "polaris_demo", >>>> "type": "INTERNAL", >>>> "readOnly": false, >>>> "properties": { >>>> "default-base-location": "file:///tmp/polaris/" >>>> }, >>>> "storageConfigInfo": { >>>> "storageType": "FILE", >>>> "allowedLocations": [ >>>> "file:///tmp" >>>> ] >>>> } >>>> } >>>> }' >>>> >>>> >>>> spark-sql \ >>>> --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1 >>>> \ >>>> --conf >>>> spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions >>>> \ >>>> --conf >>>> spark.sql.catalog.polaris=org.apache.iceberg.spark.SparkCatalog \ >>>> --conf >>>> spark.sql.catalog.polaris.catalog-impl=org.apache.iceberg.rest.RESTCatalog >>>> \ >>>> --conf spark.sql.catalog.polaris.uri= >>>> http://127.0.0.1:8181/api/catalog \ >>>> --conf spark.sql.catalog.polaris.credential=root:<secret> \ >>>> --conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \ >>>> --conf spark.sql.catalog.polaris.warehouse=polaris_demo >>>> >>>> spark-sql (default)> create namespace polaris.foo; >>>> 25/02/27 09:43:24 WARN RESTSessionCatalog: Iceberg REST client is >>>> missing the OAuth2 server URI configuration and defaults to >>>> http://127.0.0.1:8181/api/catalog/v1/oauth/tokens. This automatic >>>> fallback will be removed in a future Iceberg release.It is recommended to >>>> configure the OAuth2 endpoint using the 'oauth2-server-uri' property to be >>>> prepared. This warning will disappear if the OAuth2 endpoint is explicitly >>>> configured. See https://github.com/apache/iceberg/issues/10537 >>>> Time taken: 0.688 seconds >>>> spark-sql (default)> >>>> >>>> >>>> On Thu, Feb 27, 2025 at 12:22 AM Aihua Xu <aihu...@gmail.com> wrote: >>>> >>>>> I tested 1.8.1 RC with Snowflake build. I'm seeing the following (I'm >>>>> not seeing that in 1.7.x). >>>>> >>>>> "exception": "java.io.IOException: *Authorization header is missing*\n\tat >>>>> org.apache.polaris.service.dropwizard.auth.PolarisPrincipalAuthenticator.filter(PolarisPrincipalAuthenticator.java:43)\n\tat >>>>> org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:108)\n\tat >>>>> org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:44)\n\tat >>>>> org.glassfish.jersey.process.internal.Stages.process(Stages.java:173)\n\tat >>>>> org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:248). >>>>> >>>>> Does anyone know which change could introduce this? >>>>> >>>>> On Wed, Feb 26, 2025 at 5:16 AM Péter Váry < >>>>> peter.vary.apa...@gmail.com> wrote: >>>>> >>>>>> +1 >>>>>> >>>>>> checked the signatures, checksums >>>>>> build and run some tests >>>>>> >>>>>> Amogh Jahagirdar <2am...@gmail.com> ezt írta (időpont: 2025. febr. >>>>>> 26., Sze, 6:11): >>>>>> >>>>>>> +1 (binding) >>>>>>> >>>>>>> Verified signatures, checksum, RAT checks. >>>>>>> Ran build and test with JDK17 >>>>>>> >>>>>>> Thanks, >>>>>>> Amogh Jahagirdar >>>>>>> >>>>>>> On Wed, Feb 26, 2025 at 2:30 AM Honah J. <hon...@apache.org> wrote: >>>>>>> >>>>>>>> +1 (binding) >>>>>>>> >>>>>>>> - Checked signatures and checksum >>>>>>>> - Checked license >>>>>>>> - Full Build and Test >>>>>>>> >>>>>>>> Best regards, >>>>>>>> Honah >>>>>>>> >>>>>>>> On Tue, Feb 25, 2025 at 10:52 AM Russell Spitzer < >>>>>>>> russell.spit...@gmail.com> wrote: >>>>>>>> >>>>>>>>> +1 >>>>>>>>> Checked Sigs and Checksum >>>>>>>>> Ran Rat >>>>>>>>> Ran full build/test >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Feb 25, 2025 at 11:30 AM Driesprong, Fokko >>>>>>>>> <fo...@driesprong.frl> wrote: >>>>>>>>> >>>>>>>>>> +1 (binding) >>>>>>>>>> >>>>>>>>>> - Checked signatures and checksum >>>>>>>>>> - Checked licenses >>>>>>>>>> - Spotchecked NOTICE/LICENSE >>>>>>>>>> >>>>>>>>>> Kind regards, >>>>>>>>>> Fokko >>>>>>>>>> >>>>>>>>>> Op di 25 feb 2025 om 16:56 schreef Kevin Liu < >>>>>>>>>> kevinjq...@apache.org>: >>>>>>>>>> >>>>>>>>>>> +1 (non-binding) >>>>>>>>>>> >>>>>>>>>>> I followed "How to Verify a Release" >>>>>>>>>>> <https://iceberg.apache.org/how-to-release/#how-to-verify-a-release> >>>>>>>>>>> . >>>>>>>>>>> Checked out artifact from SVN, >>>>>>>>>>> ``` >>>>>>>>>>> svn checkout >>>>>>>>>>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-1.8.1-rc1/ >>>>>>>>>>> . >>>>>>>>>>> ``` >>>>>>>>>>> >>>>>>>>>>> Verified >>>>>>>>>>> * Signature Good >>>>>>>>>>> * Checksum Ok >>>>>>>>>>> * RAT check passed. 1 unrelated error message >>>>>>>>>>> ``` >>>>>>>>>>> ERROR: Ignored 0 lines in your exclusion files as comments or >>>>>>>>>>> empty lines. >>>>>>>>>>> ``` >>>>>>>>>>> * Build + test passed, running on Java 17.0.6 (openjdk 17.0.6 >>>>>>>>>>> 2023-01-17 LTS) on M1 >>>>>>>>>>> * Ran a few examples on Spark >>>>>>>>>>> * Ran pyiceberg integration tests, >>>>>>>>>>> https://github.com/kevinjqliu/iceberg-python/pull/11 >>>>>>>>>>> >>>>>>>>>>> I ran the tests both with and without the docker daemon. Without >>>>>>>>>>> docker, a few tests failed in `iceberg-aws`, `iceberg-azure`, and >>>>>>>>>>> `iceberg-kafka-connect`. There's already an issue to track this at >>>>>>>>>>> https://github.com/apache/iceberg/issues/12236. >>>>>>>>>>> I'm also continuing to see the flakey test for `iceberg-core`'s >>>>>>>>>>> `testConcurrentFastAppends` test. I believe this is a local issue >>>>>>>>>>> with my >>>>>>>>>>> machine. >>>>>>>>>>> >>>>>>>>>>> Thanks for running the release, Eduard! >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> Kevin Liu >>>>>>>>>>> >>>>>>>>>>> On Tue, Feb 25, 2025 at 4:23 AM Jean-Baptiste Onofré < >>>>>>>>>>> j...@nanthrax.net> wrote: >>>>>>>>>>> >>>>>>>>>>>> +1 (non binding) >>>>>>>>>>>> >>>>>>>>>>>> - Hash and checksum are good >>>>>>>>>>>> - LICENSE and NOTICE are OK in different distributed artifacts >>>>>>>>>>>> (source >>>>>>>>>>>> distribution, aws bundle, etc) >>>>>>>>>>>> - ASF header present in all expected files >>>>>>>>>>>> - No binary file found in the source distribution >>>>>>>>>>>> - Did quick smoke tests >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Regards >>>>>>>>>>>> JB >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Feb 24, 2025 at 1:46 PM Eduard Tudenhoefner >>>>>>>>>>>> <etudenhoef...@gmail.com> wrote: >>>>>>>>>>>> > >>>>>>>>>>>> > Hi Everyone, >>>>>>>>>>>> > >>>>>>>>>>>> > I propose that we release the following RC as the official >>>>>>>>>>>> Apache Iceberg 1.8.1 release. >>>>>>>>>>>> > >>>>>>>>>>>> > The commit ID is 9ce0fcf0af7becf25ad9fc996c3bad2afdcfd33d >>>>>>>>>>>> > * This corresponds to the tag: apache-iceberg-1.8.1-rc1 >>>>>>>>>>>> > * >>>>>>>>>>>> https://github.com/apache/iceberg/commits/apache-iceberg-1.8.1-rc1 >>>>>>>>>>>> > * >>>>>>>>>>>> https://github.com/apache/iceberg/tree/9ce0fcf0af7becf25ad9fc996c3bad2afdcfd33d >>>>>>>>>>>> > >>>>>>>>>>>> > The release tarball, signature, and checksums are here: >>>>>>>>>>>> > * >>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-1.8.1-rc1 >>>>>>>>>>>> > >>>>>>>>>>>> > You can find the KEYS file here: >>>>>>>>>>>> > * https://downloads.apache.org/iceberg/KEYS >>>>>>>>>>>> > >>>>>>>>>>>> > Convenience binary artifacts are staged on Nexus. The Maven >>>>>>>>>>>> repository URL is: >>>>>>>>>>>> > * >>>>>>>>>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1184/ >>>>>>>>>>>> > >>>>>>>>>>>> > Please download, verify, and test. >>>>>>>>>>>> > >>>>>>>>>>>> > Please vote in the next 72 hours. >>>>>>>>>>>> > >>>>>>>>>>>> > [ ] +1 Release this as Apache Iceberg 1.8.1 >>>>>>>>>>>> > [ ] +0 >>>>>>>>>>>> > [ ] -1 Do not release this because... >>>>>>>>>>>> > >>>>>>>>>>>> > Only PMC members have binding votes, but other community >>>>>>>>>>>> members are encouraged to cast >>>>>>>>>>>> > non-binding votes. This vote will pass if there are 3 binding >>>>>>>>>>>> +1 votes and more binding >>>>>>>>>>>> > +1 votes than -1 votes. >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>
