Hi Eric,
Answers to your questions are as follows
>
>
>
> *Can I, or is it recommended to, package multiple jobs as 1 deploymentwith
> 1 properties file or keep each app separated? Based on thedocumentation,
> it appears to support 1 app/job within a singleconfiguration as there is no
> mechanism to assign multiple app classes andgiven each a name unless I am
> mistaken*
>
*app.class* is a single valued configuration and your understanding about
it based on the documentation is correct.
>
>
> *If only 1 app per config+deployment, what is the best way to
> handlerequirement #3 - common/shared system services as there is no app or
> jobper say, I just need to specify the streams and output system
> (ieorg.apache.samza.system.hdfs.writer*
>
There are couple of options to achieve your #3 requirement.
1. If there is enough commonality between your jobs, you could have one
application class that describes the logic and have the different
configurations to modify the behavior of the application logic. This does
come with some of the following considerations
1. Your deployment system needs to support deploying the same
application with different configs.
2. Potential duplication of configuration if you configuration system
doesn't support hierarchies and overrides.
3. Potentially unmanageable for evolution, since the change in
application affects multiple jobs and requires extensive testing across
different configurations.
2. You could potentially have libraries to perform some piece of
business logic and have your different jobs leverage them using
composition. Some things to consider with this option
1. Your application and configuration stay isolated.
2. You could still leverage some of the common configurations if your
configuration system supports hierarchies and overrides
3. Alleviates concerns over evolution and testing as long as the
changes are application specific.
I am still unclear about the second part of your 2nd question.
Do you mean to say all your jobs consume from same sources and write to
sources and only your processing logic is different?
> *common/shared system services as there is no app or jobper say, I just
> need to specify the streams and output system*
Also, I am not sure I follow what do you mean by "*there is no app or job"*.
You still have 1 app per config + deployment, right?
Thanks,
Bharath
On Thu, Oct 10, 2019 at 9:46 AM Eric Shieh <datosl...@gmail.com> wrote:
> Hi,
>
> I am new to Samza, I am evaluating Samza as the backbone for my streaming
> CEP requirement. I have:
>
> 1. Multiple data enrichment and ETL jobs
> 2. Multiple domain specific CEP rulesets
> 3. Common/shared system services like consuming topics/streams and
> persisting the messages in ElasticSearch and HDFS.
>
> My questions are:
>
> 1. Can I, or is it recommended to, package multiple jobs as 1 deployment
> with 1 properties file or keep each app separated? Based on the
> documentation, it appears to support 1 app/job within a single
> configuration as there is no mechanism to assign multiple app classes and
> given each a name unless I am mistaken.
> 2. If only 1 app per config+deployment, what is the best way to handle
> requirement #3 - common/shared system services as there is no app or job
> per say, I just need to specify the streams and output system (ie
> org.apache.samza.system.hdfs.writer.
> BinarySequenceFileHdfsWriter or
>
> org.apache.samza.system.elasticsearch.indexrequest.DefaultIndexRequestFactory).
> Given it's a common shared system service not tied to specific jobs, can it
> be deployed without an app?
>
> Thank you in advance for your help, looking forward to learning more about
> Samza and developing this critical feature using Samza!
>
> Regards,
>
> Eric
>
