Thanks Alex. On Thu, Feb 27, 2025 at 10:31 AM Alex Dutra <alex.du...@dremio.com.invalid> wrote:
> Hi Aihua, > > I was indeed suspecting that you had a custom RESTClient :-) Thanks for > digging further and clarifying! Let me know if you need my help in adapting > your code. > > And with that: +1 (non-binding) from me as well. > > Thanks, > > Alex > > On Thu, Feb 27, 2025 at 6:45 PM Aihua Xu <aihu...@gmail.com> wrote: > >> Hi Alex, >> >> I checked our code further. We have an internal implementation >> for RESTClient which needs an update after your change to place the token >> in DefaultAuthSession and not pass through initHeaders anymore. The >> existing code assumes the token coming from initHeaders and we need to make >> the change to get the token from the headers of DefaultAuthSession. >> >> With that, +1 (non-binding) >> >> Thanks, >> Aihua >> >> >> >> On Thu, Feb 27, 2025 at 9:26 AM Alex Dutra <alex.du...@dremio.com.invalid> >> wrote: >> >>> Hi Aihua Xu, >>> >>> I reviewed your PR but without further details I do not agree with your >>> change, and I am unable to reproduce the issue. Besides, we have unit >>> tests that cover this extensively. >>> >>> Could you please provide a simple reproducer or a test case? >>> >>> Thanks! >>> >>> Alex >>> >>> On Thu, Feb 27, 2025 at 5:06 PM Aihua Xu <aihu...@gmail.com> wrote: >>> >>>> I'm running within our integration tests. I'm able to trace and find >>>> the breaking change: https://github.com/apache/iceberg/pull/11992. >>>> >>>> Seems we need to make the following changes to pass the token: >>>> https://github.com/apache/iceberg/pull/12415/files. Alex, can you help >>>> take a look at this? >>>> >>>> if (hasCredential) { >>>> authResponse = >>>> OAuth2Util.fetchToken( >>>> initClient, initHeaders, credential, scope, oauth2ServerUri, >>>> optionalOAuthParams); >>>> Map<String, String> authHeaders = >>>> RESTUtil.merge(initHeaders, >>>> OAuth2Util.authHeaders(authResponse.token())); >>>> config = fetchConfig(initClient, authHeaders, props); >>>> } else { >>>> authResponse = null; >>>> Map<String, String> authHeaders = >>>> RESTUtil.merge(initHeaders, OAuth2Util.authHeaders(initToken)); >>>> config = fetchConfig(initClient, authHeaders, props); >>>> } >>>> >>>> >>>> On Thu, Feb 27, 2025 at 12:46 AM Alex Dutra >>>> <alex.du...@dremio.com.invalid> wrote: >>>> >>>>> Hi Aihua, >>>>> >>>>> I just tested 1.8.1 with Polaris OSS and I am not seeing anything >>>>> different. Can you share your setup? >>>>> >>>>> Below is my Spark setup. >>>>> >>>>> Thanks, >>>>> >>>>> Alex >>>>> >>>>> ./gradlew run >>>>> >>>>> token=$(curl -s http://localhost:8181/api/catalog/v1/oauth/tokens \ >>>>> --user root:<secret> \ >>>>> -d grant_type=client_credentials \ >>>>> -d scope=PRINCIPAL_ROLE:ALL | sed -n >>>>> 's/.*"access_token":"\([^"]*\)".*/\1/p') >>>>> >>>>> curl -s -H "Authorization: Bearer ${token}" \ >>>>> -H 'Accept: application/json' \ >>>>> -H 'Content-Type: application/json' \ >>>>> http://localhost:8181/api/management/v1/catalogs \ >>>>> -d '{ >>>>> "catalog": { >>>>> "name": "polaris_demo", >>>>> "type": "INTERNAL", >>>>> "readOnly": false, >>>>> "properties": { >>>>> "default-base-location": "file:///tmp/polaris/" >>>>> }, >>>>> "storageConfigInfo": { >>>>> "storageType": "FILE", >>>>> "allowedLocations": [ >>>>> "file:///tmp" >>>>> ] >>>>> } >>>>> } >>>>> }' >>>>> >>>>> >>>>> spark-sql \ >>>>> --packages >>>>> org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1 \ >>>>> --conf >>>>> spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions >>>>> \ >>>>> --conf >>>>> spark.sql.catalog.polaris=org.apache.iceberg.spark.SparkCatalog \ >>>>> --conf >>>>> spark.sql.catalog.polaris.catalog-impl=org.apache.iceberg.rest.RESTCatalog >>>>> \ >>>>> --conf spark.sql.catalog.polaris.uri= >>>>> http://127.0.0.1:8181/api/catalog \ >>>>> --conf spark.sql.catalog.polaris.credential=root:<secret> \ >>>>> --conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \ >>>>> --conf spark.sql.catalog.polaris.warehouse=polaris_demo >>>>> >>>>> spark-sql (default)> create namespace polaris.foo; >>>>> 25/02/27 09:43:24 WARN RESTSessionCatalog: Iceberg REST client is >>>>> missing the OAuth2 server URI configuration and defaults to >>>>> http://127.0.0.1:8181/api/catalog/v1/oauth/tokens. This automatic >>>>> fallback will be removed in a future Iceberg release.It is recommended to >>>>> configure the OAuth2 endpoint using the 'oauth2-server-uri' property to be >>>>> prepared. This warning will disappear if the OAuth2 endpoint is explicitly >>>>> configured. See https://github.com/apache/iceberg/issues/10537 >>>>> Time taken: 0.688 seconds >>>>> spark-sql (default)> >>>>> >>>>> >>>>> On Thu, Feb 27, 2025 at 12:22 AM Aihua Xu <aihu...@gmail.com> wrote: >>>>> >>>>>> I tested 1.8.1 RC with Snowflake build. I'm seeing the following (I'm >>>>>> not seeing that in 1.7.x). >>>>>> >>>>>> "exception": "java.io.IOException: *Authorization header is >>>>>> missing*\n\tat >>>>>> org.apache.polaris.service.dropwizard.auth.PolarisPrincipalAuthenticator.filter(PolarisPrincipalAuthenticator.java:43)\n\tat >>>>>> org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:108)\n\tat >>>>>> org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:44)\n\tat >>>>>> org.glassfish.jersey.process.internal.Stages.process(Stages.java:173)\n\tat >>>>>> org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:248). >>>>>> >>>>>> Does anyone know which change could introduce this? >>>>>> >>>>>> On Wed, Feb 26, 2025 at 5:16 AM Péter Váry < >>>>>> peter.vary.apa...@gmail.com> wrote: >>>>>> >>>>>>> +1 >>>>>>> >>>>>>> checked the signatures, checksums >>>>>>> build and run some tests >>>>>>> >>>>>>> Amogh Jahagirdar <2am...@gmail.com> ezt írta (időpont: 2025. febr. >>>>>>> 26., Sze, 6:11): >>>>>>> >>>>>>>> +1 (binding) >>>>>>>> >>>>>>>> Verified signatures, checksum, RAT checks. >>>>>>>> Ran build and test with JDK17 >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Amogh Jahagirdar >>>>>>>> >>>>>>>> On Wed, Feb 26, 2025 at 2:30 AM Honah J. <hon...@apache.org> wrote: >>>>>>>> >>>>>>>>> +1 (binding) >>>>>>>>> >>>>>>>>> - Checked signatures and checksum >>>>>>>>> - Checked license >>>>>>>>> - Full Build and Test >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> Honah >>>>>>>>> >>>>>>>>> On Tue, Feb 25, 2025 at 10:52 AM Russell Spitzer < >>>>>>>>> russell.spit...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> +1 >>>>>>>>>> Checked Sigs and Checksum >>>>>>>>>> Ran Rat >>>>>>>>>> Ran full build/test >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Feb 25, 2025 at 11:30 AM Driesprong, Fokko >>>>>>>>>> <fo...@driesprong.frl> wrote: >>>>>>>>>> >>>>>>>>>>> +1 (binding) >>>>>>>>>>> >>>>>>>>>>> - Checked signatures and checksum >>>>>>>>>>> - Checked licenses >>>>>>>>>>> - Spotchecked NOTICE/LICENSE >>>>>>>>>>> >>>>>>>>>>> Kind regards, >>>>>>>>>>> Fokko >>>>>>>>>>> >>>>>>>>>>> Op di 25 feb 2025 om 16:56 schreef Kevin Liu < >>>>>>>>>>> kevinjq...@apache.org>: >>>>>>>>>>> >>>>>>>>>>>> +1 (non-binding) >>>>>>>>>>>> >>>>>>>>>>>> I followed "How to Verify a Release" >>>>>>>>>>>> <https://iceberg.apache.org/how-to-release/#how-to-verify-a-release> >>>>>>>>>>>> . >>>>>>>>>>>> Checked out artifact from SVN, >>>>>>>>>>>> ``` >>>>>>>>>>>> svn checkout >>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-1.8.1-rc1/ >>>>>>>>>>>> . >>>>>>>>>>>> ``` >>>>>>>>>>>> >>>>>>>>>>>> Verified >>>>>>>>>>>> * Signature Good >>>>>>>>>>>> * Checksum Ok >>>>>>>>>>>> * RAT check passed. 1 unrelated error message >>>>>>>>>>>> ``` >>>>>>>>>>>> ERROR: Ignored 0 lines in your exclusion files as comments or >>>>>>>>>>>> empty lines. >>>>>>>>>>>> ``` >>>>>>>>>>>> * Build + test passed, running on Java 17.0.6 (openjdk 17.0.6 >>>>>>>>>>>> 2023-01-17 LTS) on M1 >>>>>>>>>>>> * Ran a few examples on Spark >>>>>>>>>>>> * Ran pyiceberg integration tests, >>>>>>>>>>>> https://github.com/kevinjqliu/iceberg-python/pull/11 >>>>>>>>>>>> >>>>>>>>>>>> I ran the tests both with and without the docker daemon. >>>>>>>>>>>> Without docker, a few tests failed in `iceberg-aws`, >>>>>>>>>>>> `iceberg-azure`, and >>>>>>>>>>>> `iceberg-kafka-connect`. There's already an issue to track this at >>>>>>>>>>>> https://github.com/apache/iceberg/issues/12236. >>>>>>>>>>>> I'm also continuing to see the flakey test for `iceberg-core`'s >>>>>>>>>>>> `testConcurrentFastAppends` test. I believe this is a local issue >>>>>>>>>>>> with my >>>>>>>>>>>> machine. >>>>>>>>>>>> >>>>>>>>>>>> Thanks for running the release, Eduard! >>>>>>>>>>>> >>>>>>>>>>>> Best, >>>>>>>>>>>> Kevin Liu >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Feb 25, 2025 at 4:23 AM Jean-Baptiste Onofré < >>>>>>>>>>>> j...@nanthrax.net> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> +1 (non binding) >>>>>>>>>>>>> >>>>>>>>>>>>> - Hash and checksum are good >>>>>>>>>>>>> - LICENSE and NOTICE are OK in different distributed artifacts >>>>>>>>>>>>> (source >>>>>>>>>>>>> distribution, aws bundle, etc) >>>>>>>>>>>>> - ASF header present in all expected files >>>>>>>>>>>>> - No binary file found in the source distribution >>>>>>>>>>>>> - Did quick smoke tests >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Regards >>>>>>>>>>>>> JB >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Feb 24, 2025 at 1:46 PM Eduard Tudenhoefner >>>>>>>>>>>>> <etudenhoef...@gmail.com> wrote: >>>>>>>>>>>>> > >>>>>>>>>>>>> > Hi Everyone, >>>>>>>>>>>>> > >>>>>>>>>>>>> > I propose that we release the following RC as the official >>>>>>>>>>>>> Apache Iceberg 1.8.1 release. >>>>>>>>>>>>> > >>>>>>>>>>>>> > The commit ID is 9ce0fcf0af7becf25ad9fc996c3bad2afdcfd33d >>>>>>>>>>>>> > * This corresponds to the tag: apache-iceberg-1.8.1-rc1 >>>>>>>>>>>>> > * >>>>>>>>>>>>> https://github.com/apache/iceberg/commits/apache-iceberg-1.8.1-rc1 >>>>>>>>>>>>> > * >>>>>>>>>>>>> https://github.com/apache/iceberg/tree/9ce0fcf0af7becf25ad9fc996c3bad2afdcfd33d >>>>>>>>>>>>> > >>>>>>>>>>>>> > The release tarball, signature, and checksums are here: >>>>>>>>>>>>> > * >>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-1.8.1-rc1 >>>>>>>>>>>>> > >>>>>>>>>>>>> > You can find the KEYS file here: >>>>>>>>>>>>> > * https://downloads.apache.org/iceberg/KEYS >>>>>>>>>>>>> > >>>>>>>>>>>>> > Convenience binary artifacts are staged on Nexus. The Maven >>>>>>>>>>>>> repository URL is: >>>>>>>>>>>>> > * >>>>>>>>>>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1184/ >>>>>>>>>>>>> > >>>>>>>>>>>>> > Please download, verify, and test. >>>>>>>>>>>>> > >>>>>>>>>>>>> > Please vote in the next 72 hours. >>>>>>>>>>>>> > >>>>>>>>>>>>> > [ ] +1 Release this as Apache Iceberg 1.8.1 >>>>>>>>>>>>> > [ ] +0 >>>>>>>>>>>>> > [ ] -1 Do not release this because... >>>>>>>>>>>>> > >>>>>>>>>>>>> > Only PMC members have binding votes, but other community >>>>>>>>>>>>> members are encouraged to cast >>>>>>>>>>>>> > non-binding votes. This vote will pass if there are 3 >>>>>>>>>>>>> binding +1 votes and more binding >>>>>>>>>>>>> > +1 votes than -1 votes. >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>
