Hi there, 

I built myself a small example in order to test MATCH_RECOGNIZE in flink. 

I have a pattern of format "A B* C", where B is a `1=1` event matching 
anything. Additionally, I have a time interval of '2' DAY. Now I create 
testdata matching 
event A, send some random events (B), followed by a C matching event and add 
another event > 2 days later. 

In my expectation, the pipeline should output a match as the pipeline advanced 
more than 2 days so that even the greedy variable B, which would have consumed 
event C, now "steps back" in favor for C and produces a complete match. In my 
test, I sadly didn't get any match. 

Some additional information: 
-I tested this with Flink 1.8.1 
-Making B reluctant (B*?) produces a result, but that's not what I want for the 
usecase. 
-I am not entirely sure whether this is a bug or an expected behavior, that's 
why I ask here. If that's expected could you explain me why? It seems to 
contradict the doc in section "time constraint" where another more greedy match 
is not considered due to "out of time interval" 

And another question: In my usecase, I have a lot of events per user but want 
to apply the pattern only to a small subset of events. That's why in the 
example pattern, I have the `MISC*` in between two events. In my program, as is 
right now, I don't use Flink SQL but Flink CEP. I observed that I had a lot of 
events to be stored in state which drastically reduced performane (Basically 
all for the pattern time period). So I did a "predicate pushdown" in my 
pipeline and first filtered out the stream to only interesting events, then 
applied the pattern which works fine. I was hoping that with MATCH_RECOGNIZE, 
something similar smart could be optimized. Do you know if some optimizations 
are planned which go into this direction? I plan to let the user specify 
queries by himself and that's why I am testing CEP in SQL instead of 
programming "hardcoded" in Java as I currently do. It will be tough to reason 
to the advanced user "Your CEP query causes a lot of performance issues due to 
state, so let me rewrite it for you to e.g. something making a nested query 
with a pre select and then apply the pattern". 

Bug for the bug, here finally is a MVP: 

public class FlinkSQLtest { 

private static final String QUERY_MATCH_REOGNIZE = 
"SELECT * FROM SAMPLE MATCH_RECOGNIZE (" + 
"   PARTITION BY user " + 
"   ORDER BY ts " + 
"   MEASURES FIRSTEVENT.id AS FIRSTID, SECONDEVENT.id AS SECONDID " + 
"   ONE ROW PER MATCH " + 
"   AFTER MATCH SKIP PAST LAST ROW " + 
"   PATTERN (FIRSTEVENT MISC*? SECONDEVENT) " + 
"   WITHIN INTERVAL '2' DAY " + 
"   DEFINE " + 
"      FIRSTEVENT AS payload = 'google.com', " + 
"      SECONDEVENT AS payload = 'evil.com'" + 
")" ; 

private static List<Tuple5<Integer, Long, Integer, String, Long>> 
createTestdata() { 
List<Tuple5<Integer, Long, Integer, String, Long>> data = new ArrayList<>(); 
data.add( new Tuple5<>( 6 , 1531981987000L , 15 , "google.com" , 2L )); // 
2018-07-19T06:33:07Z 
data.add( new Tuple5<>( 6 , 1531981988000L , 16 , "blabla.com" , 2L )); // 
2018-07-19T06:33:08Z 
data.add( new Tuple5<>( 6 , 1532031988000L , 17 , "blabla.com" , 2L )); // 
2018-07-19T20:26:28Z 
data.add( new Tuple5<>( 6 , 1532069836000L , 20 , "evil.com" , 2L )); // 
2018-07-20T06:57:16Z 
data.add( new Tuple5<>( 6 , 1532415436000L , 21 , "heartbeat.com" , 1L )); // 
2018-07-24T06:57:16Z advance a few days. 
return data; 
} 

@Test 
public void testRowPatternMatching() throws Exception { 
StreamExecutionEnvironment env = StreamExecutionEnvironment. 
createLocalEnvironment (); 
env.setParallelism( 1 ); 
env.setStreamTimeCharacteristic(TimeCharacteristic. EventTime ); 
StreamTableEnvironment tEnv = StreamTableEnvironment. create (env); 

List<Tuple5<Integer, Long, Integer, String, Long>> data = createTestdata (); 

DataStream<Tuple5<Integer, Long, Integer, String, Long>> stream = 
env.fromCollection(data); 
stream = stream.assignTimestampsAndWatermarks( new 
AscendingTimestampExtractor<Tuple5<Integer, Long, Integer, String, Long>>() { 
@Override 
public long extractAscendingTimestamp(Tuple5<Integer, Long, Integer, String, 
Long> element) { 
return element. f1 ; 
} 
}); 
tEnv.registerDataStream( "SAMPLE" , stream, "user, ts.rowtime, id, payload, 
group" ); 

Table tbl = tEnv.sqlQuery( QUERY_MATCH_REOGNIZE ); 
DataStream<Row> backToStream = tEnv.toAppendStream(tbl, Row. class ); 
List<Row> list = IteratorUtils. toList (DataStreamUtils. collect 
(backToStream)); 
list.stream().forEach(System. err ::println); 
assertThat (list, hasSize ( 1 )); 

System. out .println(tEnv.explain(tbl)); 
} 
} 

Best regards 
Theo Diefenthal 

Reply via email to