I don't think adding this to the streaming flow (at micro level) will be
that useful
However, this can be added to Spark UI as an enhancement to the Streaming
Query Statistics page.
HTH
Mich Talebzadeh,
Dad | Technologist | Solutions Architect | Engineer
London
United Kingdom
view my Linked
Agree, the default behavior does not need to change.
Neil, how about separating it into two sections:
- Actual rows in the sink (same as current output)
- Followed by metadata data
Maybe we could keep the default as it is, and explicitly turn on
verboseMode to enable auxiliary information. I'm not a believer that anyone
will parse the output of console sink (which means this could be a breaking
change), but changing the default behavior should be taken conservatively.
We can
Thanks all for the positive feedback! Will figure out time to go through
the RC process. Stay tuned!
On Mon, Feb 5, 2024 at 7:46 AM Gengliang Wang wrote:
> +1
>
> On Sun, Feb 4, 2024 at 1:57 PM Hussein Awala wrote:
>
>> +1
>>
>> On Sun, Feb 4, 2024 at 10:13 PM John Zhuge wrote:
>>
>>> +1
>>>
>
Hey,
Disk space not enough is also a reliability concern, but might need a diff
strategy to handle it.
As suggested by Mridul, I am working on making things more configurable in
another(new) module… with that, we can plug in new rules for each type of
error.
Regards
Kalyan.
On Mon, 5 Feb 2024 at
