Spark job failing and not giving error to do diagnosis

2022-04-23 Thread rajat kumar
Hello All I am not getting anything in the logs and also history url is not opening. Has someone faced this issue? Application failed 1 times (global limit =5; local limit is =1) due to ApplicationMaster for attempt timed out. Failing the application. Thanks Rajat

Re: Streaming write to orc problem

2022-04-23 Thread Gourav Sengupta
Hi, can you please take a screen shot and show us the number of records that the streaming programme is reading from the source? If I am not mistaken it should be able to write out records to the output location every 5 mins. Also, it may be of help to check whether you have permissions to write

Re: Streaming write to orc problem

2022-04-23 Thread Mich Talebzadeh
It writes data every 5 seconds. under checkpoint directory gs://testdata/raw_chk what do you see? You should have four entries under chkpt directory example /mnt/gs/prices/chkpt> ltr total 1 -rw-r--rwx. 1 hduser hadoop 45 May 4 07:38 metadata drwxr-xrwx. 3 hduser hadoop 4096 May 4 07:38 sourc