You should use df.where(conditionExpr) which is more convenient to express some
simple term in SQL.
/**
* Filters rows using the given SQL expression.
* {{{
* peopleDf.where("age > 15")
* }}}
* @group dfops
* @since 1.5.0
*/
def where(conditionExpr: String): DataFrame = {
filter(Column(SqlParser.parseExpression(conditionExpr)))
}
> On Aug 7, 2016, at 10:58 PM, Mich Talebzadeh <[email protected]>
> wrote:
>
> although the logic should be col1 <> a && col(1) <> b
>
> to exclude both
>
> Like
>
> df.filter('transactiontype > " ").filter(not('transactiontype ==="DEB") &&
> not('transactiontype
> ==="BGC")).select('transactiontype).distinct.collect.foreach(println)
>
> HTH
>
> Dr Mich Talebzadeh
>
> LinkedIn
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>
>
> http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
>
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss,
> damage or destruction of data or any other property which may arise from
> relying on this email's technical content is explicitly disclaimed. The
> author will in no case be liable for any monetary damages arising from such
> loss, damage or destruction.
>
>
> On 7 August 2016 at 16:53, Mich Talebzadeh <[email protected]
> <mailto:[email protected]>> wrote:
> try similar to this
>
> df.filter(not('transactiontype ==="DEB") || not('transactiontype ==="CRE"))
>
> HTH
>
>
> Dr Mich Talebzadeh
>
> LinkedIn
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>
>
> http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
>
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss,
> damage or destruction of data or any other property which may arise from
> relying on this email's technical content is explicitly disclaimed. The
> author will in no case be liable for any monetary damages arising from such
> loss, damage or destruction.
>
>
> On 7 August 2016 at 15:43, Divya Gehlot <[email protected]
> <mailto:[email protected]>> wrote:
> Hi,
> I have use case where I need to use or[||] operator in filter condition.
> It seems its not working its taking the condition before the operator and
> ignoring the other filter condition after or operator.
> As any body faced similar issue .
>
> Psuedo code :
> df.filter(col("colName").notEqual("no_value") || col("colName").notEqual(""))
>
> Am I missing something.
> Would really appreciate the help.
>
>
> Thanks,
> Divya
>
>
