PS from an offline exchange -- yes more is being called here, the rest
is the standard WordCount example.
The trick was to make sure the task executes locally, and calling
setMaster("local") on SparkConf in the example code does that. That
seems to work fine in IntelliJ for debugging this.
On Mon, Aug 25, 2014 at 6:41 PM, Steve Lewis <lordjoe2...@gmail.com> wrote:
> ????
> That was not quite in English
>
>
> My Flatmap code is shown below
>
> I know the code is called since the answers are correct but would like to
> put a break point in dropNonLetters to make sure that code works properly
>
> I am running in the IntelliJ debugger but believe the code is executing on a
> Spark Worker.
> I am not sure what magic Intellij uses to hook up a debugger to a worker but
> hope it is possib;e
>
> public class WordsMapFunction implements FlatMapFunction<String, String> {
>
> private static final Pattern SPACE = Pattern.compile(" ");
>
> public Iterable<String> call(String s) {
> String[] split = SPACE.split(s);
> for (int i = 0; i < split.length; i++) {
> split[i] = regularizeString(split[i]);
> }
> return Arrays.asList(split);
> }
>
> public static String dropNonLetters(String s) {
> StringBuilder sb = new StringBuilder();
> for (int i = 0; i < s.length(); i++) {
> char c = s.charAt(i);
> if (Character.isLetter(c))
> sb.append(c);
> }
>
> return sb.toString();
> }
>
>
> public static String regularizeString(String inp) {
> inp = inp.trim();
> inp = inp.toUpperCase();
> return dropNonLetters(inp);
> }
>
> }
>
>
> On Mon, Aug 25, 2014 at 10:35 AM, Sean Owen <so...@cloudera.com> wrote:
>>
>> flatMap() is a transformation only. Calling it by itself does nothing,
>> and it just describes the relationship between one RDD and another.
>> You should see it swing into action if you invoke an action, like
>> count(), on the words RDD.
>>
>> On Mon, Aug 25, 2014 at 6:32 PM, Steve Lewis <lordjoe2...@gmail.com>
>> wrote:
>> > I was able to get JavaWordCount running with a local instance under
>> > IntelliJ.
>> >
>> > In order to do so I needed to use maven to package my code and
>> > call
>> > String[] jars = {
>> > "/SparkExamples/target/word-count-examples_2.10-1.0.0.jar" };
>> > sparkConf.setJars(jars);
>> >
>> > After that the sample ran properly and in the debugger I could set break
>> > points in the main.
>> >
>> > However when I do
>> > something like
>> > JavaRDD<String> words = lines.flatMap( new WordsMapFunction());
>> >
>> > where WordsMapFunction is a separate class like
>> >
>> > public static class WordsMapFunction implements FlatMapFunction<String,
>> > String> {
>> > private static final Pattern SPACE = Pattern.compile(" ");
>> > public Iterable<String> call(String s) {
>> > String[] split = SPACE.split(s);
>> > for (int i = 0; i < split.length; i++) {
>> > split[i] = toUpperCase(split[i]);
>> > }
>> > return Arrays.asList(split);
>> > }
>> > }
>> >
>> > Breakpoints set in WordsMapFunction are never hit.
>> >
>> > Most interesting functionality in the problems I am trying to solve if
>> > in
>> > the FlatMapFunction and the Function2 code and this is the functionality
>> > I
>> > will need to examine in more detail.
>> >
>> > Has anyone figured out how to configure a project to hit breakpoints in
>> > these functions??
>
>
>
>
> --
> Steven M. Lewis PhD
> 4221 105th Ave NE
> Kirkland, WA 98033
> 206-384-1340 (cell)
> Skype lordjoe_com
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
