Hello, When enabling speculation, my first job is to launch a partitioned JDBC dataframe query, in which some partitions take longer than others to respond.
This causes speculation and creates new nodes to launch the query. When one of those nodes finish the query, the speculative one remains forever connected to the Database and never ends. I have to go to the Database management tools and interrupt the query. This does not affect the spark program since the JDBC task have already ended. I only get the log messages saying that the task execution has been ignored since the task in speculation already finished. Is there any way to enable/disable speculation on a specific task to avoid this? do you have any suggestions ? or how can I report this issue? Saif
