On Mon, Jan 8, 2018 at 11:19 AM, Lukasz Cwik <lc...@google.com> wrote:
> Ken, this is specifically about running integration tests and not a users
> main().
>
Yea, I was just addressing the bit about advanced users having a headache.
Hopefully no end users are really writing options as JSON except in special
circumstances. That's why I asked why someone would be writing it by hand
in that sort of situation.
For embedding a full set of pipeline options in another format, like a pom
or IntelliJ config, JSON seems better to me since it is more explicit and
not fragile. Difficulty isn't a big issue since manual edits should be
pretty rare and smoke tests catch any errors.
Note that I'm deliberately not addressing the security question. I have no
special insights there.
Kenn
> Note, that PipelineOptions JSON format was used because it was a
> convenient serialized form that is easy to explain to people what is
> required.
> Using a different string tokenizer and calling
> PipelineOptionsFactory.fromArgs()
> with the parsed strings seems like it would be better.
>
> These are the supported formats for fromArgs():
> * --project=MyProject (simple property, will set the "project"
> property to "MyProject")
> * --readOnly=true (for boolean properties, will set the "readOnly"
> property to "true")
> * --readOnly (shorthand for boolean properties, will set the
> "readOnly" property to "true")
> * --x=1 --x=2 --x=3 (list style simple property, will set the "x"
> property to [1, 2, 3])
> * --x=1,2,3 (shorthand list style simple property, will set the "x"
> property to [1, 2, 3])
> * --complexObject='{"key1":"value1",...} (JSON format for all other
> complex types)
>
> Using a string tokenizer that minimizes the number of required escape
> characters would be good so we could use newline characters as our only
> token. I would avoid ',\t ' as tokens since they are more likely to appear.
>
> On Mon, Jan 8, 2018 at 10:33 AM, Kenneth Knowles <k...@google.com> wrote:
>
>> We do have a plain command line syntax, and whoever writes the
>> main(String[]) function is responsible for invoking the parser. It isn't
>> quite as nice as standard arg parse libraries, but it isn't too bad. It
>> would be great to improve, though.
>>
>> Jackson is for machine-to-machine communication or other situations where
>> command line parsing doesn't work so well.
>>
>> Are we using these some other way?
>>
>> Kenn
>>
>> On Sun, Jan 7, 2018 at 7:21 AM, Romain Manni-Bucau <rmannibu...@gmail.com
>> > wrote:
>>
>>>
>>>
>>> Le 7 janv. 2018 12:53, "Jean-Baptiste Onofré" <j...@nanthrax.net> a
>>> écrit :
>>>
>>> Hi Romain,
>>>
>>> I guess you are assuming that pipeline options are flat command line
>>> like argument, right ?
>>>
>>> Actually, theoretically, pipeline options can be represented as json,
>>> that's why we use jackson.
>>> The pipeline options can be serialized/deserialized as json.
>>>
>>>
>>> Yes but if users (or dev ;)) start to use that then it is trivial to
>>> break the cli handling and fromArgs parsing or, if not, break the user
>>> experience. So at the end it is a kind of "yes but no", right?
>>>
>>> PS: already see some advanced users having a headache trying to pass
>>> pipeline options in json so using the plain command line syntax can be more
>>> friendly too.
>>>
>>>
>>> The purpose is to remove the jackson dependencies ?
>>>
>>>
>>> Later yes, seeing core dep tree I identify a lot of dep which can
>>> conflict in some env and are not really needed or bring much being in the
>>> core - like avro as mentionned in another thread. Can need a sanitizing
>>> round. Short term it was really a "why is it that complicated" ;).
>>>
>>>
>>>
>>> Regards
>>> JB
>>>
>>>
>>> On 01/07/2018 11:38 AM, Romain Manni-Bucau wrote:
>>>
>>>> Hi guys,
>>>>
>>>> not sure i fully get why jackson is used to parse pipeline options in
>>>> the testing integration
>>>>
>>>> why not using a token parser like [1] which allows to map 1-1 with the
>>>> user interface (command line) the options?
>>>>
>>>> [1] https://github.com/Talend/component-runtime/blob/master/comp
>>>> onent-server/src/main/java/org/talend/sdk/component/server/l
>>>> ang/StringPropertiesTokenizer.java
>>>>
>>>> Romain Manni-Bucau
>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog <
>>>> https://rmannibucau.metawerx.net/> | Old Blog <
>>>> http://rmannibucau.wordpress.com> | Github <
>>>> https://github.com/rmannibucau> | LinkedIn <
>>>> https://www.linkedin.com/in/rmannibucau>
>>>>
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbono...@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>>>
>>>
>>
>
