Hi,
Worth checking this link
https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation
On Saturday, 28 January 2023 at 06:18:28 GMT, Lingzhe Sun
<[email protected]> wrote:
#yiv9684413148 body {line-height:1.5;}#yiv9684413148 ol, #yiv9684413148 ul
{margin-top:0px;margin-bottom:0px;list-style-position:inside;}#yiv9684413148
body {font-size:10.5pt;font-family:'Microsoft YaHei UI';color:rgb(0, 0,
0);line-height:1.5;}#yiv9684413148 body
{font-size:10.5pt;font-family:'Microsoft YaHei UI';color:rgb(0, 0,
0);line-height:1.5;}Hi all,
I'm wondering if dynamic resource allocation works in spark+kafka streaming
applications. Here're some questions:
- Will structured streaming be supported?
- Is the number of consumers always equal to the number of the partitions of
subscribed topic (let's say there's only one topic)?
- If consumers is evenly distributed across executors, will newly added
executor(through dynamic resource allocation) trigger a consumer reassignment?
- Would it be simply a bad idea to use dynamic resource allocation in
streaming app, because there's no way to scale down number of executors unless
no data is coming in?
Any thoughts are welcomed.
Lingzhe SunHirain Technology
