[GitHub] [doris] voyagertanyao opened a new issue, #16945: [spark-doris-connector] spark application can't stop when there is a "insert into" sql with a net proxy

[via GitHub]

voyagertanyao opened a new issue, #16945:
URL: https://github.com/apache/doris/issues/16945

   ### Discussed in https://github.com/apache/doris/discussions/16933
   
   <div type='discussions-op-text'>
   
   <sup>Originally posted by **voyagertanyao** February 20, 2023</sup>
   Hi, I am using spark-dorms-connector-3.1_2.12_1.1.0 release and Doris 
1.2.0-rc04
   here is the problem.
   Here is my application code :
   
   ```
   package cn.vediot.bigdata
   
   import org.apache.spark.sql.SparkSession
   
   object S3ToDoris {
     def main(args: Array[String]): Unit = {
       val spark = SparkSession.builder()
         .master("local[4]")
         .getOrCreate()
   
       val sql =
         """
           |CREATE TEMPORARY VIEW doris_bigdata_user_info
           |USING doris
           |OPTIONS(
           |  "table.identifier"="bigdata.user_info",
           |  "fenodes"="10.12.21.58:8030",
           |  "user"="root",
           |  "password"="",
           |);""".stripMargin
   
       spark.sql(sql)
   
       val sql0 =
         """
           |insert into doris_bigdata_user_info select
           |2003,'sunliu','18874218716',18,'SH-pudong','2023-02-03'
           |""".stripMargin
   
       spark.sql(sql0)
   
       spark.sql("select * from doris_bigdata_user_info").show(false)
     }
   
   }
   ```
   every thing works fine when I built my spark application into a jar ,and 
submit this spark application in linux env.
   
   **However**, **When I debug my code in IDEA**, things were different, I can 
query the newly inserted data in Doris, it means my `sql0` executed ,but my 
application in idea cannot execute the next query sql `spark.sql("select * from 
doris_bigdata_user_info").show(false)`. 
   
   This picture shows that spark application hang up there for too long, but 
data has already into doris (**duplicate data was produced because of retrying 
for many times by myself** ).
   <img width="1519" alt="image" 
src="https://user-images.githubusercontent.com/20474171/220047360-0abe53f9-a81b-4189-8434-33b8dde9c185.png";>
   <img width="842" alt="image" 
src="https://user-images.githubusercontent.com/20474171/220047643-2e5f46da-40c8-475d-8009-960883dfb31d.png";>
   
   I debug this issue for a while, I found that sql0 executed as a Doris stream 
load http request to BE, and got the response from BE . The code `while ((line 
= br.readLine()) != null)` worked well, but after that,  it can not reach the 
code `return new LoadResponse(status, respMsg, response.toString());` in 
org.apache.doris.spark.DorisStreamLoad in function `private LoadResponse 
loadBatch(String value)` , even can not reach `catch` block or `finally block` .
   
   And Here is spark-doris code 
   <img width="1174" alt="image" 
src="https://user-images.githubusercontent.com/20474171/220044284-f61da907-4408-4aa7-9237-18505fd431a2.png";>
   <img width="1294" alt="image" 
src="https://user-images.githubusercontent.com/20474171/220044497-3651994b-962a-4bc8-aeb2-cc4c5664e52c.png";>
   <img width="1380" alt="image" 
src="https://user-images.githubusercontent.com/20474171/220046176-b9d29146-acb6-45d1-8e52-1638f11e2825.png";>
   
   
   I think that's the reason why my application hang up  somewhere, and spark 
con not stop. Or any other reason for my application can not stop in IDEA env?
   
   </div>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org