[
https://issues.apache.org/jira/browse/SPARK-8480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827932#comment-15827932
]
Emlyn Corrin commented on SPARK-8480:
-------------------------------------
[~skp33] with this change, you can do similar:
{code}
scala> val df = sc.range(1,1000).toDF
df: org.apache.spark.sql.DataFrame = [value: bigint]
scala> df.setName("MyDataset")
res0: df.type = MyDataset
scala> df.cache
res1: df.type = MyDataset
scala> df.count
res2: Long = 999
scala> sc.getPersistentRDDs.foreach(println)
(4,In-memory table MyDataset MapPartitionsRDD[4] at cache at <console>:27)
scala> sc.getPersistentRDDs.filter(_._2.name == "In-memory table
MyDataset").foreach(_._2.unpersist())
scala> sc.getPersistentRDDs.foreach(println)
scala>
{code}
Although the name is not identical (it has some string prefixed). Is this good
enough, or did you have something else in mind?
> Add setName for Dataframe
> -------------------------
>
> Key: SPARK-8480
> URL: https://issues.apache.org/jira/browse/SPARK-8480
> Project: Spark
> Issue Type: Wish
> Components: SQL
> Affects Versions: 1.4.0
> Reporter: Peter Rudenko
> Priority: Minor
>
> Rdd has a method setName, so in spark UI, it's more easily to understand
> what's this cache for. E.g. ("data for LogisticRegression model", etc.).
> Would be nice to have the same method for Dataframe, since it displays a
> logical schema, in cache page, which could be quite big.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
