That sounds great, please include me so I can get involved. On Fri, Dec 9, 2016 at 7:39 AM, Marco Mistroni <mmistr...@gmail.com> wrote:
> Me too as I spent most of my time writing unit/integ tests.... pls advise > on where I can start > Kr > > On 9 Dec 2016 12:15 am, "Miguel Morales" <therevolti...@gmail.com> wrote: > >> I would be interested in contributing. Ive created my own library for >> this as well. In my blog post I talk about testing with Spark in RSpec >> style: >> https://medium.com/@therevoltingx/test-driven-development-w- >> apache-spark-746082b44941 >> >> Sent from my iPhone >> >> On Dec 8, 2016, at 4:09 PM, Holden Karau <hol...@pigscanfly.ca> wrote: >> >> There are also libraries designed to simplify testing Spark in the >> various platforms, spark-testing-base >> <http://github.com/holdenk/spark-testing-base> for Scala/Java/Python (& >> video https://www.youtube.com/watch?v=f69gSGSLGrY), sscheck >> <https://github.com/juanrh/sscheck> (scala focused property based), >> pyspark.test (python focused with py.test instead of unittest2) (& blog >> post from nextdoor https://engblog.nextdoor.com/unit-testing-apache- >> spark-with-py-test-3b8970dc013b#.jw3bdcej9 ) >> >> Good luck on your Spark Adventures :) >> >> P.S. >> >> If anyone is interested in helping improve spark testing libraries I'm >> always looking for more people to be involved with spark-testing-base >> because I'm lazy :p >> >> On Thu, Dec 8, 2016 at 2:05 PM, Lars Albertsson <la...@mapflat.com> >> wrote: >> >>> I wrote some advice in a previous post on the list: >>> http://markmail.org/message/bbs5acrnksjxsrrs >>> >>> It does not mention python, but the strategy advice is the same. Just >>> replace JUnit/Scalatest with pytest, unittest, or your favourite >>> python test framework. >>> >>> >>> I recently held a presentation on the subject. There is a video >>> recording at https://vimeo.com/192429554 and slides at >>> http://www.slideshare.net/lallea/test-strategies-for-data-pr >>> ocessing-pipelines-67244458 >>> >>> You can find more material on test strategies at >>> http://www.mapflat.com/lands/resources/reading-list/index.html >>> >>> >>> >>> >>> Lars Albertsson >>> Data engineering consultant >>> www.mapflat.com >>> https://twitter.com/lalleal >>> +46 70 7687109 >>> Calendar: https://goo.gl/6FBtlS, https://freebusy.io/la...@mapflat.com >>> >>> >>> On Thu, Dec 8, 2016 at 4:14 PM, pseudo oduesp <pseudo20...@gmail.com> >>> wrote: >>> > somone can tell me how i can make unit test on pyspark ? >>> > (book, tutorial ...) >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>> >>> >> >> >> -- >> Cell : 425-233-8271 <(425)%20233-8271> >> Twitter: https://twitter.com/holdenkarau >> >>
