Here are some open tickets I created to support expressions as json compatible strings for my flight ticket.
https://github.com/apache/arrow/issues/43858 schema support https://github.com/apache/arrow/issues/39128 Expression support I have a ticket for sqlglot to support sql operations using json compatible strings as well. ________________________________ From: Lee, David (ITE) <david....@blackrock.com.INVALID> Sent: Saturday, March 29, 2025 7:12:47 AM To: dev@arrow.apache.org <dev@arrow.apache.org> Subject: Re: [DISCUSS] Arrow Flight Predicate Pushdown This is hacky but I just use json as my flight ticket like a body for a http request post call. The json includes the data request and optional compute/filtering that needs to be performed. The server applies the compute/filtering using sql or pyarrow compute depending on the data source. ________________________________ From: David Li <lidav...@apache.org> Sent: Friday, March 28, 2025 5:07:24 AM To: dev@arrow.apache.org <dev@arrow.apache.org> Subject: Re: [DISCUSS] Arrow Flight Predicate Pushdown External Email: Use caution with links and attachments There is no real interoperability between Flight services, precisely because CMD is entirely user-defined. And unfortunately Flight forces everyone into sharing one service definition in order to accomplish it. (I mean, you could establish some way to define and distribute the structure of the requests/responses, but then you're just reinventing Protobuf+gRPC/OpenAPI, but badly and with no community.) > There may not be a secret sauce but there is the ease of > implementation/deployment of having everything in a single service. Being > able to write a Flight server following the cookbooks in different > languages is very beneficial. *It is great actually. * The problem is that it puts us as Arrow developers in a weird spot of effectively being level 1 tech support for Google (gRPC/Protobuf), which is also an unpaid position for a multi billion dollar company. Also I think it "scales up" badly: once you do want to use any gRPC feature that we didn't have the foresight to expose, you're stuck. (Not entirely stuck, if in Go or Java, but it certainly borks Flight SQL's design enough to be painful.) > Operating multiple gRPC services in the same processes in Python doesn't > seem to be possible due to how gRPC is wrapped by Arrow Flight. Yes, that's another implementation mistake unfortunately. I don't consider it as a reason to ditch Flight/gRPC entirely, though, just that the implementation in C++ specifically has a lot of downsides. Python service developers are certainly stuck in a rough spot, though. On Fri, Mar 28, 2025, at 18:28, Rusty Conover wrote: > On Thu, Mar 27, 2025 at 12:01 PM David Li <lidav...@apache.org> wrote: > >> It's not clear to me why (4) doesn't work. Can you speak more about the >> flow of requests here? Putting a ticket inside a FlightDescriptor makes me >> think something more complicated is going on. >> > > I made a mistake in my description of method 4, I meant FlightDescriptor > rather than "ticket". I apologize. > > Method 4 would be the creation of a new FlightDescriptor that is of the CMD > type, with the original FlightDescriptor along with the predicate > information serialized in some format understandable to the server. > > While technically any of these methods could work, I'm trying to achieve a > solution that could yield the most interoperability with other Flight > servers. It seems that this may not be important as the existing ecosystem > of Flight servers don't have consistency beyond the gRPC spec. Or there > really isn't a public ecosystem to speak of? > > If there isn't really an ecosystem or a desire to foster one I'm happy to > just make a decision, send some type of "Flight client protocol" version > header so clients/servers can determine the choices made, and move forward > with implementation. > > >> Alternatively...just create your own gRPC service, and return whatever >> payload you need. There's no need to bend Flight to do something it wasn't >> designed to do, and apart from the Arrow/Protobuf/gRPC integrations in the >> data-carrying methods, there's no secret sauce in Flight anyways. >> > > > There may not be a secret sauce but there is the ease of > implementation/deployment of having everything in a single service. Being > able to write a Flight server following the cookbooks in different > languages is very beneficial. *It is great actually. * > > Operating multiple gRPC services in the same processes in Python doesn't > seem to be possible due to how gRPC is wrapped by Arrow Flight. > > As an aside, could "Arrow Flight: The Good Parts" be summarized as: > > * DoGet, DoExchange, DoPut, and DoAction. > * FlightDescriptor/FlightEndpoint are both reasonable types to use, but > FlightInfo should be avoided. > * Authentication is a call to DoAction and gRPC headers. > > Best, > > Rusty This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.blackrock.com%2Fcorporate%2Fcompliance%2Femail-disclaimers&data=05%7C02%7CDavid.Lee%40blackrock.com%7C6e8f03aa288142ed679508dd6ecbfba9%7C282a32955c424d939ec16631001cc5f7%7C0%7C0%7C638788544557665790%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=IhWzG9jPPhpOw42dfFk7at1bOoXeWWifDdoL6J6R2e0%3D&reserved=0<http://www.blackrock.com/corporate/compliance/email-disclaimers> for further information. Please refer to https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.blackrock.com%2Fcorporate%2Fcompliance%2Fprivacy-policy&data=05%7C02%7CDavid.Lee%40blackrock.com%7C6e8f03aa288142ed679508dd6ecbfba9%7C282a32955c424d939ec16631001cc5f7%7C0%7C0%7C638788544557683288%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=d4nQV3jn%2Fb5cZXc99Gw34DpDIWkkXbNIB22FJzdlpHI%3D&reserved=0<http://www.blackrock.com/corporate/compliance/privacy-policy> for more information about BlackRock’s Privacy Policy. For a list of BlackRock's office addresses worldwide, see https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.blackrock.com%2Fcorporate%2Fabout-us%2Fcontacts-locations&data=05%7C02%7CDavid.Lee%40blackrock.com%7C6e8f03aa288142ed679508dd6ecbfba9%7C282a32955c424d939ec16631001cc5f7%7C0%7C0%7C638788544557697394%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=sWxRV9JF3bJt2fWLt%2ByDg0I0%2Fjk6NCjw2rVCTUBR70g%3D&reserved=0<http://www.blackrock.com/corporate/about-us/contacts-locations> . © 2025 BlackRock, Inc. All rights reserved. This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See http://www.blackrock.com/corporate/compliance/email-disclaimers for further information. Please refer to http://www.blackrock.com/corporate/compliance/privacy-policy for more information about BlackRock’s Privacy Policy. For a list of BlackRock's office addresses worldwide, see http://www.blackrock.com/corporate/about-us/contacts-locations. © 2025 BlackRock, Inc. All rights reserved.
