The MSGraphAsyncOperator already handles multi paged responses internally using the defer mechanism, but the result of this are indexed XComs, which means you have to expand over them to process it. We sometimes also use jq in combination with BashOperator to merge those multiple JSON responses so we can handle it in one task, but in some situations, you just can't do that as explained in my previous example.
-----Original Message----- From: Daniel Standish <daniel.stand...@astronomer.io.INVALID> Sent: Wednesday, September 18, 2024 6:41 PM To: dev@airflow.apache.org Subject: Re: [PROPOSAL] Add streaming support to PartialOperator EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, stuur deze e-mail als bijlage naar ab...@infrabel.be<mailto:ab...@infrabel.be>. Curious why you want to model this as many tasks, e.g. one page == one task. Another option would be to handle many pages in one task. And I'm curious what were the factors that led you to split it out more granularly.
