Could you share the code that creates the pipeline. Especially the part about pipeline options. Seems you might reference there files with s3:// protocol. Try to find such files and replace this with file:// to point to local file. Personally I do not have much exp with java sdk, but this is I would take a look after seeing stack trace. Best
pt., 22 lis 2024, 02:20 użytkownik XQ Hu via user <user@beam.apache.org> napisał: > Hard to tell what went wrong. Have you tried our Java starter example ( > https://github.com/apache/beam-starter-java)? This should work just fine > with TestPipeline. > > On Thu, Nov 21, 2024 at 5:56 PM Ramya Prasad via user < > user@beam.apache.org> wrote: > >> Hello, >> I am a developer trying to use Apache Beam in Java, and I am having an >> issue with my TestPipeline taking a long time to get created for my unit >> tests. When I create a brand new TestPipeline() and call my pipeline.run() >> method, there are a lot of calls to get AWS information, even though I'm >> just creating a bare TestPipeline by calling TestPipeline.create(). Here >> are some stack traces I see in my code: >> poll:-1, Net (sun.nio.ch) park:191, NioSocketImpl (sun.nio.ch) >> timedRead:280, NioSocketImpl (sun.nio.ch) implRead:304, NioSocketImpl ( >> sun.nio.ch) read:346, NioSocketImpl (sun.nio.ch) read:796, >> NioSocketImpl$1 (sun.nio.ch) read:1099, Socket$SocketInputStream ( >> java.net) fill:291, BufferedInputStream (java.io) read1:347, >> BufferedInputStream (java.io) implRead:420, BufferedInputStream (java.io) >> read:399, BufferedInputStream (java.io) parseHTTPHeader:827, HttpClient >> (sun.net.www.http) parseHTTP:759, HttpClient (sun.net.www.http) >> getInputStream0:1705, HttpURLConnection (sun.net.www.protocol.http) >> getInputStream:1614, HttpURLConnection (sun.net.www.protocol.http) >> getResponseCode:531, HttpURLConnection (java.net) readResource:112, >> HttpResourcesUtils (software.amazon.awssdk.regions.util) getToken:436, >> EC2MetadataUtils (software.amazon.awssdk.regions.internal.util) >> getItems:426, EC2MetadataUtils >> (software.amazon.awssdk.regions.internal.util) getData:362, >> EC2MetadataUtils (software.amazon.awssdk.regions.internal.util) >> getData:358, EC2MetadataUtils >> (software.amazon.awssdk.regions.internal.util) getEC2InstanceRegion:242, >> EC2MetadataUtils (software.amazon.awssdk.regions.internal.util) >> tryDetectRegion:72, InstanceProfileRegionProvider >> (software.amazon.awssdk.regions.providers) getRegion:56, >> InstanceProfileRegionProvider (software.amazon.awssdk.regions.providers) >> getRegion:51, AwsRegionProviderChain >> (software.amazon.awssdk.regions.providers) create:60, >> AwsOptions$AwsRegionFactory (org.apache.beam.sdk.io.aws2.options) >> create:55, AwsOptions$AwsRegionFactory >> (org.apache.beam.sdk.io.aws2.options) returnDefaultHelper:689, >> ProxyInvocationHandler (org.apache.beam.sdk.options) getDefault:630, >> ProxyInvocationHandler (org.apache.beam.sdk.options) invoke:227, >> ProxyInvocationHandler (org.apache.beam.sdk.options) valueOrElse:235, >> ClientBuilderFactory$DefaultClientBuilder >> (org.apache.beam.sdk.io.aws2.common) create:142, >> ClientBuilderFactory$DefaultClientBuilder >> (org.apache.beam.sdk.io.aws2.common) create:56, ClientBuilderFactory >> (org.apache.beam.sdk.io.aws2.common) createBuilder:40, >> DefaultS3ClientBuilderFactory (org.apache.beam.sdk.io.aws2.s3) >> createBuilder:35, DefaultS3ClientBuilderFactory >> (org.apache.beam.sdk.io.aws2.s3) getBuilder:100, S3FileSystemConfiguration >> (org.apache.beam.sdk.io.aws2.s3) builderFrom:88, S3FileSystemConfiguration >> (org.apache.beam.sdk.io.aws2.s3) fromS3Options:92, >> S3FileSystemConfiguration (org.apache.beam.sdk.io.aws2.s3) fromOptions:35, >> DefaultS3FileSystemSchemeRegistrar (org.apache.beam.sdk.io.aws2.s3) >> lambda$fromOptions$0:49, S3FileSystemRegistrar >> (org.apache.beam.sdk.io.aws2.s3) accept:273, ReferencePipeline$7$1 >> (java.util.stream) forEachRemaining:133, Iterator (java.util) >> forEachRemaining:1939, Spliterators$IteratorSpliterator (java.util) >> copyInto:509, AbstractPipeline (java.util.stream) wrapAndCopyInto:499, >> AbstractPipeline (java.util.stream) evaluateSequential:921, >> ReduceOps$ReduceOp (java.util.stream) evaluate:234, AbstractPipeline >> (java.util.stream) collect:682, ReferencePipeline (java.util.stream) >> fromOptions:50, S3FileSystemRegistrar (org.apache.beam.sdk.io.aws2.s3) >> verifySchemesAreUnique:567, FileSystems (org.apache.beam.sdk.io) >> setDefaultPipelineOptions:554, FileSystems (org.apache.beam.sdk.io) >> fromOptions:47, PipelineRunner (org.apache.beam.sdk) run:317, Pipeline >> (org.apache.beam.sdk) run:398, TestPipeline (org.apache.beam.sdk.testing) >> run:335, TestPipeline (org.apache.beam.sdk.testing) Regardless of how I >> create my TestPipeline(), will it always make calls to S3 File System >> Configuration, or is there a way to bypass this behavior since I don't need >> to use S3 in my pipeline for my unit tests? >> >> Any help would be appreciated! >> Thanks and sincerely, >> Ramya >> ------------------------------ >> >> The information contained in this e-mail may be confidential and/or >> proprietary to Capital One and/or its affiliates and may only be used >> solely in performance of work or services for Capital One. The information >> transmitted herewith is intended only for use by the individual or entity >> to which it is addressed. If the reader of this message is not the intended >> recipient, you are hereby notified that any review, retransmission, >> dissemination, distribution, copying or other use of, or taking of any >> action in reliance upon this information is strictly prohibited. If you >> have received this communication in error, please contact the sender and >> delete the material from your computer. >> >> >> >> >>
