Hi folks,
Just an FYI -- I've found a potential race condition in this RC with block
manager decommissioning and the torrent broadcast factory (
https://issues.apache.org/jira/browse/SPARK-34193 ).
I don't think this should block the release (it's not a regression), so
my +1 stands as is and I haven't triggered it more than once.
Cheers,
Holden
On Wed, Jan 20, 2021 at 9:05 PM Mridul Muralidharan <mri...@gmail.com>
wrote:
> +1
>
> Signatures, digests, etc check out fine.
> Checked out tag and build/tested with -Pyarn -Phadoop-2.7 -Phive
> -Phive-thriftserver -Pmesos -Pkubernetes
>
> The sha512 signature for spark-3.1.1.tgz tripped up my scripts :-)
>
>
> Regards,
> Mridul
>
>
> On Wed, Jan 20, 2021 at 8:17 PM 郑瑞峰 <ruife...@foxmail.com> wrote:
>
>> +1 (non-binding)
>>
>> Thank you, Hyukjin
>>
>> Bests,
>> Ruifeng
>>
>> ------------------ 原始邮件 ------------------
>> *发件人:* "Dongjoon Hyun" <dongjoon.h...@gmail.com>;
>> *发送时间:* 2021年1月20日(星期三) 中午1:57
>> *收件人:* "Holden Karau"<hol...@pigscanfly.ca>;
>> *抄送:* "Sean Owen"<sro...@gmail.com>;"Hyukjin Kwon"<gurwls...@gmail.com
>> >;"dev"<dev@spark.apache.org>;
>> *主题:* Re: [VOTE] Release Spark 3.1.1 (RC1)
>>
>> +1
>>
>> I additionally
>> - Ran JDBC integration test
>> - Ran with AWS EKS 1.16
>> - Ran unit tests with Python 3.9.1 combination (numpy 1.19.5, pandas
>> 1.2.0, scipy 1.6.0)
>> (PyArrow is not tested because it's not supported in Python 3.9.x. This
>> is documented via SPARK-34162)
>>
>> There exists some on-going work in the umbrella JIRA (SPARK-33507:
>> Improve and fix cache behavior in v1 and v2).
>> I believe it can be achieved at 3.2.0 and we can add some comments on the
>> release note at 3.1.0.
>>
>> Thank you, Hyukjin and all.
>>
>> Bests,
>> Dongjoon.
>>
>> On Tue, Jan 19, 2021 at 10:49 AM Holden Karau <hol...@pigscanfly.ca>
>> wrote:
>>
>>> +1, pip installs on Python 3.8
>>>
>>> One potential thing we might want to consider if there ends up being
>>> another RC is that the error message for installing with Python2 could be
>>> clearer.
>>>
>>> Processing ./pyspark-3.1.1.tar.gz
>>> ERROR: Command errored out with exit status 1:
>>> command: /tmp/py3.1/bin/python2 -c 'import sys, setuptools,
>>> tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-lmlitE/setup.py'"'"';
>>> __file__='"'"'/tmp/pip-req-build-lmlitE/setup.py'"'"';f=getattr(tokenize,
>>> '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
>>> '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))'
>>> egg_info --egg-base /tmp/pip-pip-egg-info-W1BsIL
>>> cwd: /tmp/pip-req-build-lmlitE/
>>> Complete output (6 lines):
>>> Traceback (most recent call last):
>>> File "<string>", line 1, in <module>
>>> File "/tmp/pip-req-build-lmlitE/setup.py", line 31
>>> file=sys.stderr)
>>> ^
>>> SyntaxError: invalid syntax
>>> ----------------------------------------
>>> ERROR: Command errored out with exit status 1: python setup.py egg_info
>>> Check the logs for full command output.
>>>
>>>
>>>
>>> On Tue, Jan 19, 2021 at 10:26 AM Sean Owen <sro...@gmail.com> wrote:
>>>
>>>> +1 from me. Same results as in 3.1.0 testing.
>>>>
>>>> On Mon, Jan 18, 2021 at 6:06 AM Hyukjin Kwon <gurwls...@gmail.com>
>>>> wrote:
>>>>
>>>>> Please vote on releasing the following candidate as Apache Spark
>>>>> version 3.1.1.
>>>>>
>>>>> The vote is open until January 22nd 4PM PST and passes if a majority
>>>>> +1 PMC votes are cast, with a minimum of 3 +1 votes.
>>>>>
>>>>> [ ] +1 Release this package as Apache Spark 3.1.0
>>>>> [ ] -1 Do not release this package because ...
>>>>>
>>>>> To learn more about Apache Spark, please see http://spark.apache.org/
>>>>>
>>>>> The tag to be voted on is v3.1.1-rc1 (commit
>>>>> 53fe365edb948d0e05a5ccb62f349cd9fcb4bb5d):
>>>>> https://github.com/apache/spark/tree/v3.1.1-rc1
>>>>>
>>>>> The release files, including signatures, digests, etc. can be found at:
>>>>> https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-bin/
>>>>>
>>>>> Signatures used for Spark RCs can be found in this file:
>>>>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>
>>>>> The staging repository for this release can be found at:
>>>>> https://repository.apache.org/content/repositories/orgapachespark-1364
>>>>>
>>>>> The documentation corresponding to this release can be found at:
>>>>> https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-docs/
>>>>>
>>>>> The list of bug fixes going into 3.1.1 can be found at the following
>>>>> URL:
>>>>> https://s.apache.org/41kf2
>>>>>
>>>>> This release is using the release script of the tag v3.1.1-rc1.
>>>>>
>>>>> FAQ
>>>>>
>>>>> ===================
>>>>> What happened to 3.1.0?
>>>>> ===================
>>>>>
>>>>> There was a technical issue during Apache Spark 3.1.0 preparation, and
>>>>> it was discussed and decided to skip 3.1.0.
>>>>> Please see
>>>>> https://spark.apache.org/news/next-official-release-spark-3.1.1.html
>>>>> for more details.
>>>>>
>>>>> =========================
>>>>> How can I help test this release?
>>>>> =========================
>>>>>
>>>>> If you are a Spark user, you can help us test this release by taking
>>>>> an existing Spark workload and running on this release candidate, then
>>>>> reporting any regressions.
>>>>>
>>>>> If you're working in PySpark you can set up a virtual env and install
>>>>> the current RC via "pip install
>>>>> https://dist.apache.org/repos/dist/dev/spark/v3.1.1-rc1-bin/pyspark-3.1.1.tar.gz
>>>>> "
>>>>> and see if anything important breaks.
>>>>> In the Java/Scala, you can add the staging repository to your projects
>>>>> resolvers and test
>>>>> with the RC (make sure to clean up the artifact cache before/after so
>>>>> you don't end up building with an out of date RC going forward).
>>>>>
>>>>> ===========================================
>>>>> What should happen to JIRA tickets still targeting 3.1.1?
>>>>> ===========================================
>>>>>
>>>>> The current list of open tickets targeted at 3.1.1 can be found at:
>>>>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>>>>> Version/s" = 3.1.1
>>>>>
>>>>> Committers should look at those and triage. Extremely important bug
>>>>> fixes, documentation, and API tweaks that impact compatibility should
>>>>> be worked on immediately. Everything else please retarget to an
>>>>> appropriate release.
>>>>>
>>>>> ==================
>>>>> But my bug isn't fixed?
>>>>> ==================
>>>>>
>>>>> In order to make timely releases, we will typically not hold the
>>>>> release unless the bug in question is a regression from the previous
>>>>> release. That being said, if there is something which is a regression
>>>>> that has not been correctly targeted please ping me or a committer to
>>>>> help target the issue.
>>>>>
>>>>>
>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>
--
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
