Hello Weng, This definitely helps a lot, however I know my initial DS has a single row content then I would in theory just create a DS which is what I need. That is why I need to know how to create a new environment DS within a map function.
thanks so much On Tue, Jul 23, 2019 at 11:41 PM Caizhi Weng <[email protected]> wrote: > Hi Andres, > > Thanks for the detailed explanation. > > but apparently I can't create a new DS within a map function > > > If you create a new DS within the map function, then you'll create as many > DSs as the number of elements in the old DS which... doesn't seem to be > your desired situation? I suppose you want to create a DS<Tuple> from > DS<String>. If that is the case you can write something like this: > > public static void main(String[] args) throws Exception { > StreamExecutionEnvironment env = > StreamExecutionEnvironment.getExecutionEnvironment(); > > DataStream<String> dsString = env.fromElements("1,a,1.1", "2,b,2.2,-2", > "3,c", "4,d,4.4"); > DataStream<Object> dsTuple = dsString.map(s -> { > String[] split = s.split(","); > if (split.length == 2) { > return new Tuple2<>(Integer.valueOf(split[0]), split[1]); > } else if (split.length == 3) { > return new Tuple3<>(Integer.valueOf(split[0]), split[1], > Double.valueOf(split[2])); > } else { > return new Tuple4<>(Integer.valueOf(split[0]), split[1], > Double.valueOf(split[2]), Long.valueOf(split[3])); > } > }); > > dsTuple.print(); > env.execute(); > } > > > How dynamically create the DS<Tuple> > > > As you can see in the above code, I did not create a DS<Tuple> but a > DS<Object>, because Tuple can't be directly used. It seems that you want to > turn this new DS into a table, but if different records have different > number of columns this is not a good practice as the schema of each record > is not the same (but as a workaround, you can fill the columns with null if > some record doesn't have this column). > > Hope this solves your problem. If you have any other problems feel free to > write back. > > Andres Angel <[email protected]> 于2019年7月24日周三 上午10:50写道: > >> Hello, >> >> Let me list properly the questions I have: >> >> * How to catch into a string the content of a DataStream? about this >> point basically I have a DS<String> , the only way how I can use the >> content is within a map function , print , store the content somewhere or >> SQL queries. The point is that I need the content because depending on that >> content I need to create another DS and later register it as a Table >> environment, which means I need the value content but likewise the headers >> content and the whole info is within the DS<String>. The first option I had >> was play with the map function but apparently I can't create a new DS >> within a map function and less register it as a new table environment. >> >> My second option in this point could be create a sort of public variable >> to store the DS<String> content and then create my UDF, but sadly this is >> neither allowed. My options in this case would be either somehow store >> public the content of the DS<String> into a new variable, turn the >> DS<String> as String or store the content in a file and read the file and >> start over to parse the content to serve the header and content for the new >> DS. >> >> * How dynamically create the DS<Tuple>: well basically after parse the >> point above I might end up with an array of fields sometimes 4,3,2 doesnt >> matter then I might need to swing between different tuples or turn my >> content into Row to create a DS<Row>. >> >> I'm looking forward to reading your comments. >> >> thanks so much >> >> On Tue, Jul 23, 2019 at 10:34 PM Caizhi Weng <[email protected]> >> wrote: >> >>> Hi Andres, >>> >>> Sorry I can't quite get your question... Do you mean that how to spilt >>> the string into fields? >>> >>> There is a `split` method in java. You can give it a regexp and it will >>> return an array containing all the split fields. >>> >>> Andres Angel <[email protected]> 于2019年7月24日周三 上午10:28写道: >>> >>>> Hello Weng, >>>> >>>> thanks for your reply, however I'm struggling to somehow read the >>>> content of my DS with the payload that defines how many fields the message >>>> contains into a String. That is the reason why I thought into a map >>>> function for that DS. >>>> >>>> The Tuple part can change overtime can even pass from 3 or 4 to 2 then >>>> it can change the whole time. How could I approach this challenge? >>>> >>>> thanks so much >>>> >>>> On Tue, Jul 23, 2019 at 10:23 PM Caizhi Weng <[email protected]> >>>> wrote: >>>> >>>>> Hi Andres, >>>>> >>>>> Are the payloads strings? If yes, one method is that you can store >>>>> them as strings and process it further with user defined functions when >>>>> you >>>>> need to use them. >>>>> >>>>> Another method is that you can store them into arrays. >>>>> >>>>> Also, if the type of the first 3 fields are the same for the first and >>>>> second payload, you can use a Tuple4<> and set the last element as null >>>>> for >>>>> the first payload. >>>>> >>>>> Andres Angel <[email protected]> 于2019年7月24日周三 上午10:09写道: >>>>> >>>>>> Hello everyone, >>>>>> >>>>>> I need to create dynamically the size of my Tuple that feeds a DS, >>>>>> let me explain it better. Let's assume the first payload I read has this >>>>>> format "filed1,field2,field3", then this might require a Tuple3<> but my >>>>>> payload later can be "field1,field2,field3,field4" then my Tuple might >>>>>> need >>>>>> to be refine it on the flight and now be Tuple4<>. >>>>>> >>>>>> How could I create this dynamically, any idea? >>>>>> >>>>>> Thanks so much >>>>>> >>>>>
