kz930 opened a new pull request, #8341: URL: https://github.com/apache/texera/pull/8341
### What changes were proposed in this PR? Ten source operators implement `StandaloneCodeGenerator`, so an exported script starts from the same data the workflow did rather than from a variable the reader has to fill in: the CSV family, JSON Lines, Arrow, plain text, and the two that read a file named at run time. A source is the one place where the script cannot simply repeat what the operator does. The engine resolves a dataset through Texera's storage and hands the operator a URI; a script has no such resolver, so it reads the file from its own directory under the name that URI ended with. That is what makes an exported script portable, and it is equally its one precondition: the data has to sit beside the script. The name is taken from the last path segment rather than by parsing the whole string as a URI, because the resolver percent-encodes the file-relative parts but leaves the repository and version names as the user typed them, and a dataset version with a space in its name makes `new URI` throw before any code is generated. Two sources are reported as unverifiable rather than exported blind. File Scan takes its filenames from an input port at run time, which a source harness has nothing to feed. URL Fetcher reads a live URL, so two runs are not required to agree and a comparison would only measure the network. ### Any related issues, documentation, discussions? Part of #8325. Builds on #8327, which adds the trait and the verification. ### How was this PR tested? `SourceCategoryRunner` reads each source twice — once through the engine, once through its generated script — and compares the tables, across the encodings and options each source offers. Twelve runs pass; the two named above are reported with their reason instead. Arrow is the one worth calling out. Its column of timestamps only matches once #7672 is in, which landed while this was being written: before it the engine read the file's numbers through the JVM's zone while `pd.read_feather` read the wall clock the file states, and the two differed by whatever the machine was set to. This branch is up to date with main, so the comparison passes; on a checkout without #7672 it does not, which is the clearest statement of what that change fixed. `WorkflowOperator` passes in full at 2546 tests. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
