> On April 14, 2015, 10:14 p.m., Yi Pan (Data Infrastructure) wrote: > > samza-sql/src/main/java/org/apache/samza/sql/metadata/Stream.java, line 37 > > <https://reviews.apache.org/r/33142/diff/1/?file=926293#file926293line37> > > > > What's the meaning of this parent? > > Milinda Pathirage wrote: > parent identifies the system. For example 'kafka'. And this is similar to > db schema.
Milinda, I think you should add doc to clarify the purpose of parent. It will probably not be a good sign if you find yourself using parent (or even name) a lot. Calcite relations are kind of like files - and the name and parent are like the inode part of the file. Just like files, they should behave the same whatever you call them. The RelOptTable that wraps a Table (or its subclass, Stream) remembers the fully-qualified name for you. > On April 14, 2015, 10:14 p.m., Yi Pan (Data Infrastructure) wrote: > > samza-sql/src/main/java/org/apache/samza/sql/metadata/Stream.java, line 76 > > <https://reviews.apache.org/r/33142/diff/1/?file=926293#file926293line76> > > > > I am not sure whether I fully understand this interface method: from > > Calcite code base, it seems that this should return an iterator for the > > stream? This seems to be a polling method to get the rows from the stream. > > However, Samza tasks are not polling the streams to get the messages. > > Instead, Samza SystemConsumers delivers (i.e. pushes) the messages from the > > physical streams to Samza tasks, and hence, the physical operators. How is > > this stream() API used in Calcite? Can we leave it un-implemented? > > Milinda Pathirage wrote: > AFAIK, this method is only useful in Calcite. We don't have to implement > this. May be I should change this to throw UnsupportedOperationException. My intention was that if you had a relation that could act as both a stream and a table (consider Orders - you can ask it for the history (Table) or future orders (Stream)) - then the catalog would give you the table, then you call its stream() method to give you the streaming version. Note how this relates to my previous comment. The Table object you get by calling stream() is anonymous. It is not stored in the catalog directly. Anyway. That method is experimental, and still evolving. Don't worry about it too much unless you see errors. We will revisit it when we want to have streams that can also operate as tables. - Julian ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33142/#review79941 ----------------------------------------------------------- On April 13, 2015, 9:04 p.m., Yi Pan (Data Infrastructure) wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/33142/ > ----------------------------------------------------------- > > (Updated April 13, 2015, 9:04 p.m.) > > > Review request for samza and Milinda Pathirage. > > > Bugs: SAMZA-561 > https://issues.apache.org/jira/browse/SAMZA-561 > > > Repository: samza > > > Description > ------- > > [SAMZA-561] Review in progress > > Post Milinda's patch for SAMZA-561 to ease the comment and discussion. > > > Diffs > ----- > > build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e > gradle/dependency-versions.gradle ee6dfc411b7ab90b187df79f109884127953862e > samza-sql/src/main/java/org/apache/samza/sql/Utils.java PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/api/operators/spec/OperatorSpec.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/data/IntermediateMessageTuple.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/data/serializers/SqlAvroSerdeFactory.java > PRE-CREATION > samza-sql/src/main/java/org/apache/samza/sql/expressions/Expression.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/expressions/RexToJavaCompiler.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/expressions/RexToJavaUtils.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/metadata/AvroSchemaConverter.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/metadata/RelDataTypeToAvroSchemaConverter.java > PRE-CREATION > samza-sql/src/main/java/org/apache/samza/sql/metadata/Stream.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorSpec.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/operators/factory/TypeAwareOperatorSpec.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/operators/insert/InsertToStreamOp.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/operators/insert/InsertToStreamSpec.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/operators/project/ProjectOp.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/operators/project/ProjectSpec.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/operators/scan/ProjectableFilterableStreamScanOp.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/operators/scan/ProjectableFilterableStreamScanSpec.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/operators/scan/StreamScanSpec.java > PRE-CREATION > samza-sql/src/main/java/org/apache/samza/sql/planner/ExecutionPlanner.java > PRE-CREATION > samza-sql/src/main/java/org/apache/samza/sql/planner/QueryPlanner.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/planner/rules/FilterableStreamScanRule.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/planner/rules/ProjectableStreamScanRule.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/planner/rules/RemoveIdentityProjectRule.java > PRE-CREATION > > samza-sql/src/main/java/org/apache/samza/sql/rel/ProjectableFilterableStreamScan.java > PRE-CREATION > samza-sql/src/main/java/org/apache/samza/sql/rel/StreamScan.java > PRE-CREATION > samza-sql/src/main/java/org/apache/samza/task/sql/StreamSqlTask.java > PRE-CREATION > > samza-sql/src/test/java/org/apache/samza/sql/data/serializers/SqlAvroSerdeTest.java > PRE-CREATION > samza-sql/src/test/java/org/apache/samza/sql/planner/QueryPlannerTest.java > PRE-CREATION > > samza-sql/src/test/java/org/apache/samza/sql/planner/SamzaStreamTableFactory.java > PRE-CREATION > > samza-sql/src/test/java/org/apache/samza/sql/planner/TestExecutionPlanner.java > PRE-CREATION > samza-sql/src/test/java/org/apache/samza/sql/planner/TestQueryPlanner.java > PRE-CREATION > > samza-sql/src/test/java/org/apache/samza/sql/planner/TestRexToJavaCompiler.java > PRE-CREATION > samza-sql/src/test/java/org/apache/samza/sql/test/Constants.java > PRE-CREATION > samza-sql/src/test/java/org/apache/samza/sql/test/Utils.java PRE-CREATION > > samza-sql/src/test/java/org/apache/samza/sql/test/metadata/TestAvroSchemaConverter.java > PRE-CREATION > samza-sql/src/test/java/org/apache/samza/task/sql/RandomOperatorTask.java > PRE-CREATION > samza-sql/src/test/java/org/apache/samza/task/sql/StreamSqlTask.java > PRE-CREATION > samza-sql/src/test/resources/orders.avsc PRE-CREATION > samza-sql/src/test/resources/orders.json PRE-CREATION > samza-test/src/main/config/sql-filter.properties PRE-CREATION > > samza-test/src/main/java/org/apache/samza/test/integration/sql/OrdersStreamFactory.java > PRE-CREATION > samza-test/src/main/java/org/apache/samza/test/integration/sql/SqlTask.java > PRE-CREATION > samza-test/src/main/python/integration_tests.py > df64e239a2e467c8e4429dbeb7039f1aa9965ecc > samza-test/src/main/python/requirements.txt > 2ae95908248516b5b26e671f24fa680f7b801675 > samza-test/src/main/python/samza_job_yarn_deployer.py > 38635ca5899c43fb61d6b4042e8543f0508fd41b > samza-test/src/main/python/tests/sql_tests.py PRE-CREATION > samza-test/src/main/resources/orders.avsc PRE-CREATION > samza-test/src/main/resources/orders.json PRE-CREATION > > Diff: https://reviews.apache.org/r/33142/diff/ > > > Testing > ------- > > > Thanks, > > Yi Pan (Data Infrastructure) > >
