Hard to tell what went wrong. Have you tried our Java starter example ( https://github.com/apache/beam-starter-java)? This should work just fine with TestPipeline.
On Thu, Nov 21, 2024 at 5:56 PM Ramya Prasad via user <[email protected]> wrote: > Hello, > I am a developer trying to use Apache Beam in Java, and I am having an > issue with my TestPipeline taking a long time to get created for my unit > tests. When I create a brand new TestPipeline() and call my pipeline.run() > method, there are a lot of calls to get AWS information, even though I'm > just creating a bare TestPipeline by calling TestPipeline.create(). Here > are some stack traces I see in my code: > poll:-1, Net (sun.nio.ch) park:191, NioSocketImpl (sun.nio.ch) > timedRead:280, NioSocketImpl (sun.nio.ch) implRead:304, NioSocketImpl ( > sun.nio.ch) read:346, NioSocketImpl (sun.nio.ch) read:796, > NioSocketImpl$1 (sun.nio.ch) read:1099, Socket$SocketInputStream (java.net) > fill:291, BufferedInputStream (java.io) read1:347, BufferedInputStream ( > java.io) implRead:420, BufferedInputStream (java.io) read:399, > BufferedInputStream (java.io) parseHTTPHeader:827, HttpClient > (sun.net.www.http) parseHTTP:759, HttpClient (sun.net.www.http) > getInputStream0:1705, HttpURLConnection (sun.net.www.protocol.http) > getInputStream:1614, HttpURLConnection (sun.net.www.protocol.http) > getResponseCode:531, HttpURLConnection (java.net) readResource:112, > HttpResourcesUtils (software.amazon.awssdk.regions.util) getToken:436, > EC2MetadataUtils (software.amazon.awssdk.regions.internal.util) > getItems:426, EC2MetadataUtils > (software.amazon.awssdk.regions.internal.util) getData:362, > EC2MetadataUtils (software.amazon.awssdk.regions.internal.util) > getData:358, EC2MetadataUtils > (software.amazon.awssdk.regions.internal.util) getEC2InstanceRegion:242, > EC2MetadataUtils (software.amazon.awssdk.regions.internal.util) > tryDetectRegion:72, InstanceProfileRegionProvider > (software.amazon.awssdk.regions.providers) getRegion:56, > InstanceProfileRegionProvider (software.amazon.awssdk.regions.providers) > getRegion:51, AwsRegionProviderChain > (software.amazon.awssdk.regions.providers) create:60, > AwsOptions$AwsRegionFactory (org.apache.beam.sdk.io.aws2.options) > create:55, AwsOptions$AwsRegionFactory > (org.apache.beam.sdk.io.aws2.options) returnDefaultHelper:689, > ProxyInvocationHandler (org.apache.beam.sdk.options) getDefault:630, > ProxyInvocationHandler (org.apache.beam.sdk.options) invoke:227, > ProxyInvocationHandler (org.apache.beam.sdk.options) valueOrElse:235, > ClientBuilderFactory$DefaultClientBuilder > (org.apache.beam.sdk.io.aws2.common) create:142, > ClientBuilderFactory$DefaultClientBuilder > (org.apache.beam.sdk.io.aws2.common) create:56, ClientBuilderFactory > (org.apache.beam.sdk.io.aws2.common) createBuilder:40, > DefaultS3ClientBuilderFactory (org.apache.beam.sdk.io.aws2.s3) > createBuilder:35, DefaultS3ClientBuilderFactory > (org.apache.beam.sdk.io.aws2.s3) getBuilder:100, S3FileSystemConfiguration > (org.apache.beam.sdk.io.aws2.s3) builderFrom:88, S3FileSystemConfiguration > (org.apache.beam.sdk.io.aws2.s3) fromS3Options:92, > S3FileSystemConfiguration (org.apache.beam.sdk.io.aws2.s3) fromOptions:35, > DefaultS3FileSystemSchemeRegistrar (org.apache.beam.sdk.io.aws2.s3) > lambda$fromOptions$0:49, S3FileSystemRegistrar > (org.apache.beam.sdk.io.aws2.s3) accept:273, ReferencePipeline$7$1 > (java.util.stream) forEachRemaining:133, Iterator (java.util) > forEachRemaining:1939, Spliterators$IteratorSpliterator (java.util) > copyInto:509, AbstractPipeline (java.util.stream) wrapAndCopyInto:499, > AbstractPipeline (java.util.stream) evaluateSequential:921, > ReduceOps$ReduceOp (java.util.stream) evaluate:234, AbstractPipeline > (java.util.stream) collect:682, ReferencePipeline (java.util.stream) > fromOptions:50, S3FileSystemRegistrar (org.apache.beam.sdk.io.aws2.s3) > verifySchemesAreUnique:567, FileSystems (org.apache.beam.sdk.io) > setDefaultPipelineOptions:554, FileSystems (org.apache.beam.sdk.io) > fromOptions:47, PipelineRunner (org.apache.beam.sdk) run:317, Pipeline > (org.apache.beam.sdk) run:398, TestPipeline (org.apache.beam.sdk.testing) > run:335, TestPipeline (org.apache.beam.sdk.testing) Regardless of how I > create my TestPipeline(), will it always make calls to S3 File System > Configuration, or is there a way to bypass this behavior since I don't need > to use S3 in my pipeline for my unit tests? > > Any help would be appreciated! > Thanks and sincerely, > Ramya > ------------------------------ > > The information contained in this e-mail may be confidential and/or > proprietary to Capital One and/or its affiliates and may only be used > solely in performance of work or services for Capital One. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the intended > recipient, you are hereby notified that any review, retransmission, > dissemination, distribution, copying or other use of, or taking of any > action in reliance upon this information is strictly prohibited. If you > have received this communication in error, please contact the sender and > delete the material from your computer. > > > > >
