Vinodraj Konda created BEAM-13706:
-------------------------------------
Summary: Beam IO WriteToText - FileNotFoundError
Key: BEAM-13706
URL: https://issues.apache.org/jira/browse/BEAM-13706
Project: Beam
Issue Type: Bug
Components: io-py-files
Affects Versions: 2.26.0
Environment: RHEL 7 & python 3.6
Reporter: Vinodraj Konda
Hi Team,
We are running a beam pipeline & writing the final output to a file as below.
This runs fine most times but fails sometimes with error "FileNotFoundError:
[Errno 2] No such file or directory". This has been intermittent and there is
no issue with the file location, other jobs using the same location run fine at
the same time & this too works fine if we immediately re-try. Please check and
advise, I couldn't find any solution/details for this online. Thanks in advance.
| 'write to file' >> beam.io.WriteToText(self.output_file,
header=','.join(output_cols_header),
shard_name_template=''
)
+*ERROR:*+
[2022-01-17 15:20:43,512] \{bash_operator.py:126} INFO - Traceback (most recent
call last):
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"apache_beam/runners/common.py", line 1213, in
apache_beam.runners.common.DoFnRunner.process
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"apache_beam/runners/common.py", line 742, in
apache_beam.runners.common.PerWindowInvoker.invoke_process
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"apache_beam/runners/common.py", line 867, in
apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/iobase.py",
line 1148, in process
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - writer =
self.sink.open_writer(init_result, str(uuid.uuid4()))
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/options/value_provider.py",
line 135, in _f
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return fnc(self,
*args, **kwargs)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filebasedsink.py",
line 196, in open_writer
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return
FileBasedSinkWriter(self, writer_path)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filebasedsink.py",
line 417, in __init__
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - self.temp_handle =
self.sink.open(temp_shard_path)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/textio.py",
line 405, in open
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - file_handle =
super(_TextSink, self).open(temp_path)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/options/value_provider.py",
line 135, in _f
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return fnc(self,
*args, **kwargs)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filebasedsink.py",
line 138, in open
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return
FileSystems.create(temp_path, self.mime_type, self.compression_type)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/filesystems.py",
line 229, in create
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return
filesystem.create(path, mime_type, compression_type)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/localfilesystem.py",
line 168, in create
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - return
self._path_open(path, 'wb', mime_type, compression_type)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - File
"/local/home/airflow/kumulus_venv/lib64/python3.6/site-packages/apache_beam/io/localfilesystem.py",
line 143, in _path_open
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - raw_file =
io.open(path, mode)
[2022-01-17 15:20:43,513] \{bash_operator.py:126} INFO - FileNotFoundError:
[Errno 2] No such file or directory:
'/dropbox/weather/prd/kumulus/data/earthsat/daily/2022/01/v2/beam-temp-earthsat_daily_4cast_20220117.csv-0163f65477a911ecb8d80aec9c027dc6/da21a024-9253-4500-9d10-576bdb6a4711.earthsat_daily_4cast_20220117.csv'
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
