but worth investigating it.
cheers
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Sat, 7 Jun 2025 at 08:08, Ángel Álvarez Pascua <
angel.al
tunately something is missing somewhere
They have seen this error with postgres Hive metastore DB as well. I need
to work on it when I have a chance
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com
And great effort by you Jerry to drive this proposal through.
Let us see how it progresses.Will be interesting
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
(e.g., transactional
sinks) or careful custom
implementation for both stateless and stateful operations
etc
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-p
Are you running in YARN mode and you want to put these jar files into HDFS
in a distributed cluster?
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
"near real-time streaming" or "interactive streaming" to accurately
describe the system's capabilities and bridge the gap between academic
rigor and practical industry usage. This IMO is a good suggestion to reduce
ambiguity.
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Fina
" are typically operating on
the softer end of this spectrum, providing performance crucial for
applications under considerations (for example within SLAs) where delays
are undesirable but not show stopper.
I therefore suggest the SPIP should mention this explicitly, so we can
move on
ot; Principle
In summary, "Real-time Mode" seems to describe an approach that delivers
low-latency processing with high reliability and ease of use, leveraging
established, battle-tested components.I invite the audience to have a
discussion on this.
HTH
Dr Mich Talebzadeh,
Architect | Dat
uot; answer is simply not good enough. As a colliery it is a
fundamental concept, so it has to be treated as such not as a comment.in
SPIP
Hope this clarifies the connection in practical terms
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linked
of the application.if I get the right answer too slowly
it becomes useless or wrong
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Wed, 28 May 2025 at
tra
low-latency execution mode. A time interval can also be specified, e.g.
“300 Seconds”, to indicate how long each micro-batch should run for.
"
will inevitably depend on many factors. Not that simple
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analys
Maybe you should emphasize Sparc 4 (RC5) as the current state of
sparc 4, undergoing
extensive testing.
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
+1
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Wed, 9 Apr 2025 at 20:05, Gengliang Wang wrote:
> +1
>
> On Wed, Apr 9, 2025 at 11:57
+1
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Wed, 9 Apr 2025 at 08:07, Peter Toth wrote:
> +1
>
> On Wed, Apr 9, 2025 at 8:51 AM C
Because of dependencies we need to ensure that the underlying artifacts
(Hive 4.0.1) is also stable enough. We should aim to establish that first
and look for release timelines and where it fits
cheers
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
major headache.
Now I just need to customise various files under $HIVE_HOME/conf and then I
will have some testing underway.
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-p
+1 Sounds like a plan
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Sun, 16 Mar 2025 at 21:10, Martin Grund
wrote:
> So I was just playing with
open forum, then the person is
expected to back it up. *I cannot see how anyone could object to the
statement: if you make a claim or have a strong opinion, be prepared to
prove it or debate it.* Regardless, as stated mistakes can and do happen.
HTH
Dr Mich Talebzadeh,
Architect | Data Science
Hi Jungtaek.
With regard to your point below
"...Hi dev, I'm really tired of the discussion which does not move forward
because the argument is not backed by strict ASF policy"
Regardless, we all appreciate your efforts and your tenacity.
cheers
Dr Mich Talebzadeh,
A
of Compound Sentiment Scores) / (Total
Messages Sent)
[image: sentiment_score.png]
Dongjoon sentiment seems to be pretty neutral and the rest mildly positive
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://ww
This is my gist
Mark from your passionate language I gather you see this as a "Code Change"
veto. Your reasoning seems to be straightforward, i.e. the vote's purpose
is to decide whether to add code (migration logic) to the Spark 4.0 branch.
In your view, the outcome of the vote directly alters th
Agreed. Hive upgrade is more time consuming as it involves backing up Hive
schema on your metastore and then running Hive provided upgrade schema
scripts against Hive schema that could be problematic,but needs to be done
one way or another.
HTH
Dr Mich Talebzadeh,
Architect | Data Science
s, and bug fixes. Compiling against it would allow Spark to take
advantage of these. Plus using the latest versions of both Spark and Hive
is important for maintaining a secure data platform.
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedi
The first link seems to be still invalid, although the proposal itself is
sound
https://github.com/apache/spark-connect-swift
Can someone else please confirm it?
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<ht
Glad to see that eventually this repository is created now
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Mon, 10 Mar 2025 at 23:37, Dongjoon Hyun
Can you please double check the first link, I am getting 404!
thanks
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Sun, 9 Mar 2025 at 22:31, Dongjoo
Sure we leave it as it is. No big deal
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Tue, 4 Mar 2025 at 23:29, Jungtaek Lim
wrote:
> Thanks for
ately that Spark Connect is an interface for interacting
with Spark, not a replacement for the entire system.
HTH
..
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
Thanks.
Can you point to a link or any further documentation please?
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Tue, 4 Mar 2025 at 13:22, Herm
more informed knowledge.
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Mon, 24 Feb 2025 at 19:13, D. Mohith Akshay
wrote:
> Hello Everyone,
&g
+1 on the basis of Dongjoon statement which I trust
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Mon, 24 Feb 2025 at 00:47, Dongjoon Hyun
+1 for me following my recent comments on the discussion thread on this
topic as well
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Sun, 23 Feb 2025 at
thread.
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Mon, 17 Feb 2025 at 16:07, Max Gekk wrote:
> Hello Mich,
>
> Thank you for the pro
.
-
RC1 is typically followed by a sequence of additional RCs (e.g., RC2,
RC3) as needed, until all blockers are resolved and the final release is
ready.
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<ht
+1
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Wed, 19 Feb 2025 at 09:31, Wenchen Fan wrote:
> Please vote on releasing the following candidate
+1
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Wed, 19 Feb 2025 at 06:51, Ángel wrote:
> +1 (non-binding)
>
> El mié, 19 feb 2025, 7:
through intermediate versions to avoid breakage.
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Wed, 19 Feb 2025 at 00:41, Jungtaek Lim
program and make it work
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Wed, 12 Feb 2025 at 19:53, Max Gekk wrote:
> Hello Mich,
>
> >
✅ *"Thanks, Matei. ✅ Looks like a plan!*
*📌 We resurrected the old thread! *
*https://lists.apache.org/thread/wwjyp1bhryvx7ytooj1lqtd8kgzxb6vq
<https://lists.apache.org/thread/wwjyp1bhryvx7ytooj1lqtd8kgzxb6vq>*
🔗 Hopefully, there will be more traction this round.
HTH
Dr Mich
it to a default value.
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Wed, 12 Feb 2025 at 18:56, Sakthi wrote:
> Thanks for the proposal, Max. T
Let us carry on on that thread.
Need to catch-up
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Tue, 11 Feb 2025 at 06:01, Pavan Kotikalapudi
can
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Mon, 10 Feb 2025 at 23:05, Jungtaek Lim
wrote:
> Let's move the discussion to the other t
Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Mon, 10 Feb 2025 at 12:39, José Müller wrote:
> Hi Mitch,
>
> All you said is well understood, but I believe you
cluster, Have you looked at Koalas which I believe is
currently integrated as pyspark.pandas?
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Mon, 10 Fe
Well, everything is possible. Please initiate a discussion on the matter of
a proposal to "Create a pluggable cluster manager" and put it to the
community.
See some examples here
https://lists.apache.org/list.html?dev@spark.apache.org
HTH
Dr Mich Talebzadeh,
Architect | Data Science |
YARN).
2. Implementing *a full pluggability for spark-submit *would require
redesign and implementation to handle the diverse requirements of different
cluster managers which I think will be a major project for itself
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial
mode cluster \
--name sparkArmada
then modify or copy Spark-Submit code to Spark-Submit-Armanda to handle
this custom URL for now for test/debugging purposes
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.
Kubernetes cluster
*as a separate container.
which provides better resource isolation and is more suitable for this type
of cluster you are using Armada
Anyway you can see how it progresses in debugging mode.
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
I am familiar with some of your work in G-Research
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Thu, 6 Feb 2025 at 23:40, Dejan Pejchev wrote:
&
I don't see its relevance to ASF board report? It is a minor technicality
and probably tangential. It is not a show stopper and the Board does it
need to worry about it.
Best to take this discussion on its own thread
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | For
+1
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Wed, 5 Feb 2025 at 08:26, Yuming Wang wrote:
> +1
>
> On Wed, Feb 5, 2025 at 4:15 PM
Hi Frank,
I think this would be for the Spark dev team. I have added to the email.
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Fri, 31 Jan 2025
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
Hi Rob,
As a matter of interest, have you got an indication of a ballpark figure
for percentage of queries that end up with skewed distribution?
Thanks
Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.
. 1 hduser hadoop44704 Oct 21 03:29 hive-cli-2.3.9.jar
-rw-r--r--. 1 hduser hadoop 183633 Oct 21 03:29 hive-beeline-2.3.9.jar
I have all these jars there but are you implying that the potential
vulnerability will
be from hive-metastore-2.3.9.jar alone or all of hive jars?
Cheers
Mich Talebza
To answer your question, I did not read this CVE, but I am responding
solely from my previous experiences with vulennabiries and the thread owner
implications, having used spark in conjunction with Spark for many years.
Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic
store, as they can indirectly impact the security and stability of
Spark applications among other things
HTH
Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On
mv hive-exec-1.2.1.jar hive-exec-1.2.1.spark2.jar
HTH
Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Sat, 25 Jan 2025 at 08:44, 王则杰 wrote:
> rename
Ok so the catalyst optimizer will use this method of inline key counting to
provide spark optimizer with prior notification, so it identifies the hot
keys? What is this inline key counting based? Likely Count-Min Sketch
algorithm!
HTH
Mich Talebzadeh,
Architect | Data Science | Financial Crime
e
the challenges in a nutshell that you referred to?
HTH,
Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Wed, 22 Jan 2025 at 20:47, David Milicevic
wrote
Sorry I forgot to mention once you extract the JAR file, copy or symlink
it to $SPARK_HOME/jars directory
HTH
Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
CI broken is really an operational aspect albeit in this case was quote
temporary. We should put that aside and move on as 1) product is sound and
2) spark connect is strategic for the future of Spark.
HTH
Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
Spark's internals as opposed to RDDs. *Moreover, **maintaining
backward compatibility fo*r the existing *RDD-based applications and
libraries* is crucial during this transition window so the timeframe is
another factor for consideration.
HTH
Mich Talebzadeh,
Architect | Data Science | Fina
mp/apache-hive-1.2.1-src/ql/target/"
HTH
Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Tue, 21 Jan 2025 at 02:42, 王则杰 wrote:
> I need to mo
Given our recent discussion on using spark connect as a stable API, this
will be another positive step.
HTH
Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
this evolution of Spark.
HTH,
Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
*Disclaimer:* The information provided is correct to the best of my
knowledge but of course cannot be guaranteed .
gt;>>>> At a high level, some notable shaded prefixes included org.json,
>>>>> com.google.common / protobuf, org.apache.commons, and org.antlr. Key
>>>>> dependencies *not* shaded were avro, jackson, datanucleus, logging /
>>>>> JRE / sc
ewing past discussions and votes
on the dev list will be very helpful and informative.
HTH
Architect | Data Science | Financial Crime | GDPR & Compliance Specialist
PhD Imperial College London London, United Kingdom
view my Linkedin profile
<https://www.linkedin.com/in/mich-tal
On your point
...I believe there are better ways to improve the pythonic surface of
PySpark. ..
Can you please elaborate?
HTH
Mich Talebzadeh,
Architect | Data Science | Financial Crime | GDPR & Compliance Specialist
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperi
ations, aligning with
Python's emphasis on clarity and expressiveness (as the above link).
HTH
Mich Talebzadeh,
Architect | Data Science | Financial Crime | GDPR & Compliance Specialist
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
London <https
shading will introduce more debugging and testing as packages will be
renamed impacting flexibility. Case in point, things like unit and
integration tests may need adjustments to account for the renamed packages.
HTH
Mich Talebzadeh,
Architect | Data Science | Financial Crime | GDPR & Compli
+ 1
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
London <https://en.wikipedia.org/wiki/Imperial_College_London>
London, United Kingdom
view my Linkedin profile
<https://w
Hm. Since it sounds like a plan why Russell you go ahead and create a SPIP
for it, then, this discussion takes a formal approach and is documented.
Otherwise we are just flogging a dead horse so to speak.
HTH
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <ht
OK I added a comment to PR
HTH,
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
London <https://en.wikipedia.org/wiki/Imperial_College_London>
London, United Kingdom
view
and actively contribute. If no substantial engagement
occurs within this timeframe, we may need to consider closing the project.
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
London
Upgraded from Spark 3.4.0 to 3.4.4
Looks good with the following versions I have tested
- openjdk 11.0.8
- hadoop-3.1.0
- hive-3.1.1
- hbase-1.2.6
- GoogleBigQuery with spark-3.4-bigquery-0.41.0.jar
HTH
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
+1
It will be a desirable feature
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
London <https://en.wikipedia.org/wiki/Imperial_College_London>
London, United Kingdom
view
Hi Jay
As far as I am aware in Spark 2.4.4, there is no feature to enable executor
decommissioning with graceful shutdown, nor is there a way to specify a
timeout for forcefully killing executors. These were introduced in Spark
3.0.
HTH
Mich Talebzadeh,
Architect | Data Engineer | Data
to be clear are you referring to these
spark.executor.decommission.enabled=true
spark.executor.decommission.gracefulShutdown=true
thanks
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
nfig("spark.executor.decommission.forceKillTimeout", "100s") \
.getOrCreate()
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
London <https://en.wikipedia.org/wiki/Imper
}")
The output
Spark version: 3.4.0
spark.executor.decommission.enabled: true
spark.executor.decommission.forceKillTimeout: 100s
By creating a simple Spark application and verifying the configuration
values, I trust it is shown that these two parameters are valid and are
appl
Do you have a better recommendation?
Or trying to waste time as usual.
It is far easier to throw than catch.
Do your homework and stop throwing spanners at work.
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philoso
Before responding, what configuration parameters are you using to make this
work?
spark.executor.decommission.enabled=true
spark.executor.decommission.gracefulShutdown=true
spark.executor.decommission.forceKillTimeout=100s
HTH
Mich Talebzadeh,
Architect | Data Engineer | Data Science
business and technical realities.
HTH
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
London <https://en.wikipedia.org/wiki/Imperial_College_London>
London, United Kingdom
view
graph processing in Spark. I saw someone
created some documents
HTH
Mich Talebzadeh,
*Disclaimer:* The information provided is correct to the best of my
knowledge but of course cannot be guaranteed . It is essential to note
that, as with any advice, quote "one test result is worth one-tho
+1 on the assumption that we should phase this release on an incremental
basis. Probably will take us to end of release 5.
HTH
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
London
ffs of complexity, resource
availability and long-term gains.
HTH
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
London <https://en.wikipedia.org/wiki/Imperial_College_London>
London, United
+1
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
London <https://en.wikipedia.org/wiki/Imperial_College_London>
London, United Kingdom
view my Linkedin profile
<https://w
should prioritize the health of the Spark
ecosystem and ensure that we are investing resources into actively
maintained components.
HTH
Mich Talebzadeh
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
London
+1
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
London <https://en.wikipedia.org/wiki/Imperial_College_London>
London, United Kingdom
view my Linkedin profile
<https://w
ement
declaring Spark 2.4.0 as the final minor release, the fact that 2.4.8 is
still being maintained suggests it might be an LTS release. This is likely
due to its continued usage?
HTH
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.
ards,
> Mark Andreev
>
>
> On Wed, 21 Aug 2024 at 23:08, Mich Talebzadeh
> wrote:
>
>> Hi Mark,
>>
>> You have already done that and have made the request for review.
>>
>> +1 for me
>>
>> Mich Talebzadeh,
>>
>> Architect |
Hi Mark,
You have already done that and have made the request for review.
+1 for me
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
London <https://en.wikipedia
ted}."
By providing this additional context, developers can more efficiently
pinpoint and resolve schema mismatches.
HTH
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
L
k -f convert_sum.awk size.txt
11.88 GB
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
London <https://en.wikipedia.org/wiki/Imperial_College_London>
London, United Kingdom
Hi Kent,
Can you if possible provide a heuristic estimate of space reduction your
proposal is going to achieve?
Thanks
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
London
Hi Kent,
Can you if possible please provide a heuristic estimate of storage
reduction that will be achieved through this approach?
Thanks
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial C
achieved through this
approach.
Overall, the proposal offers a viable solution for managing Spark
documentation while reducing storage concerns. However, addressing the
potential complexity of managing older documentation versions is crucial.
+1 for me
Mich Talebzadeh,
Architect | Data Engineer | Data
+1 for me
Mich Talebzadeh,
Architect | Data Engineer | Data Science | Financial Crime
PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
London <https://en.wikipedia.org/wiki/Imperial_College_London>
London, United Kingdom
view my Linkedin pr
1 - 100 of 468 matches
Mail list logo
