On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote:
> Thanks Colin and Rajini.
>
> I've updated the KIP to say that KIP-421 functionality is available to
> encrypt sensitive configs like the ZK key store and trust store
> passwords. (I've also made it clear that the configs are not
> dynamically reconfigurable since dynamic values are stored in ZK and
> the data is needed to get to ZK in the first place). Colin, let me
> know if you think anything else needs to be said/done about it -- I
> wasn't sure if your comment above implies that there are additional
> direct actions that we need to take aside from this.
>
> I agree that making the brokers' ZooKeeper clients dynamic with
> respect to key and trust stores (e.g. via
> zookeeper.ssl.context.supplier.class) may increase the risk that this
> KIP may not land in time for 2.5.
>
> Regarding the config names for ZK being different than the ZooKeeper
> configs (e.g. camel-case keyStore/trustStore instead of
> keystore/truststore, ciphersuites instead of cipher.suites,
> enabledProtocols instead of enabled.protocols), I agree it is an
> issue. I tried to mitigate it by putting warning text in the config
> docs for these. Regarding configuration inheritance, I think what you
> are saying is that for any configs where the broker has a clear
> semantic equivalent, the broker could fall back to the broker value if
> the ZK value is not given. The potential list is:
>
> zookeeper.ssl.keyStore.location => ssl.keystore.location
> zookeeper.ssl.keyStore.password => ssl.keystore.password
> zookeeper.ssl.keyStore.type => ssl.keystore.type
> zookeeper.ssl.trustStore.location => ssl.truststore.location
> zookeeper.ssl.trustStore.password => ssl.truststore.password
> zookeeper.ssl.trustStore.type => ssl.truststore.type
> zookeeper.ssl.ciphersuites => ssl.cipher.suites
>
> Note that there are two configs that look the same based on their key
> names but whose allowable values differ:
>
> zookeeper.ssl.protocol(Default: TLSv1.2) => ssl.protocol(Default: TLS)
> zookeeper.ssl.enabledProtocols(Default: value of protocol property) =
> ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)
>
> Not sure what the right answer is, but hopefully this detail will help
> get us there.
>
> Ron
I think, on balance, I agree with Rajini that it would be nice to make the
configs look more like Kafka configs. We don't have camel-case in
configuration keys, so we should avoid it here.
best,
Colin
>
> On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe <cmcc...@apache.org> wrote:
> >
> > On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
> > > Hi Ron,
> > >
> > > Thanks for the detailed explanation, sounds good to me.
> > >
> > > A few more questions:
> > >
> > > 1) At the moment, all sensitive broker configs including
> > > keystore/truststore passwords can be stored encrypted in ZooKeeper prior
> > > to
> > > starting up brokers. We are now adding SSL keystore/truststore passwords
> > > for ZooKeeper client that cannot be stored in ZooKeeper since you need
> > > these to connect to ZK. We should perhaps document that these configs can
> > > be encrypted using KIP-421.
> >
> > That's a good point. Previously, we didn't have ZK on-the-wire security
> > support. However, now that we do, sending sensitive keystore and
> > truststore passwords to ZK in cleartext should be deprecated in favor of
> > using KIP-421.
> >
> > >
> > > 2) We can dynamically update keystores and trust stores used by brokers
> > > without restarting the broker. Can we support this easily for ZK clients
> > > created by the broker, for example by adding our own
> > > `zookeeper.ssl.context.supplier.class`?
> > >
> >
> > Hmm. That might be better to consider in a follow-up, since the deadline
> > for 2.5 KIPs is coming up?
> >
> > best,
> > Colin
> >
> > > 3) Looks like we are using config names that directly map to ZK configs.
> > > Have we considered using equivalent Kafka config names with prefixes,
> > > perhaps with inheritance from the non-prefixed names? Not sure if this is
> > > a
> > > good idea, but perhaps worth documenting in Rejected Alternatives at
> > > least.
> > >
> > >
> > > On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <cmcc...@apache.org> wrote:
> > >
> > > > Hi Ron,
> > > >
> > > > Thanks for the explanation. I guess thinking about it a little bit
> > > > more,
> > > > we should just add --zk-tls-config-file to all of these commands.
> > > >
> > > > We will be removing this option (plus ZK in general) from these commands
> > > > in the next major release, but ZK is still supported in 2.5, so we
> > > > should
> > > > just do the logical thing. (The exception is ZkSecurityMigrator, which
> > > > will stay).
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > > > > Hi Colin.
> > > > >
> > > > > <<< It seems like this [--zk-tls-config-file information] could just
> > > > appear
> > > > > in a configuration file, which all of these tools already accept (I
> > > > think)
> > > > >
> > > > > ZkSecurityMigrator has no such property file facility; adding a
> > > > > "--zk-tls-config-file" parameter is exactly for this purpose. If we
> > > > > add
> > > > > that to ZkSecurityMigrator then it is trivial to add it to other
> > > > > commands
> > > > > (the same code is simply reused; it ends up being just a few extra
> > > > lines).
> > > > > I do not see any parameter in the other two commands to adjust the ZK
> > > > > connection; ConfigCommand accepts a "--command-config" flag, but
> > > > according
> > > > > to the code "This is used only with --bootstrap-server option for
> > > > > describing and altering broker configs."
> > > > >
> > > > > I do agree there would be no need to add "--zk-tls-config-file" to
> > > > > ReassignPartitionsCommand if its direct ZK connectivity is replaced in
> > > > time
> > > > > for the next release.
> > > > >
> > > > > ConfigCommand supports the "--bootstrap-server" option and will have
> > > > > its
> > > > > direct ZooKeeper access formally deprecated as per KIP-555, but the
> > > > special
> > > > > use case of bootstrapping a ZooKeeper ensemble with encrypted
> > > > > credentials
> > > > > prior to starting Kafka will still exist, so it feels like while
> > > > > "--zk-tls-config-file" would never be used except for this use case it
> > > > > could probably still be added for this particular situation.
> > > > >
> > > > > Ron
> > > > >
> > > > > P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more
> > > > > I
> > > > > sent) did not go through. I am trying to get emails through now to
> > > > > move
> > > > > this discussion forward.
> > > > >
> > > > > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <cmcc...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > > > > > Hi everyone. I would like to make the following changes to the
> > > > > > > KIP.
> > > > > > >
> > > > > > > MOTIVATION:
> > > > > > > Include a statement that it will be difficult in the short term to
> > > > > > > deprecate direct Zookeeper communication in kafka-configs.{sh,
> > > > > > > bat}
> > > > > > (which
> > > > > > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka
> > > > cluster
> > > > > > > with encrypted passwords in Zookeeper is an explicitly-supported
> > > > > > > use
> > > > > > case;
> > > > > > > therefore it is in scope to be able to securely configure the CLI
> > > > tools
> > > > > > > that still leverage non-deprecated direct Zookeeper communication
> > > > for TLS
> > > > > > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > > > > > zookeeper-security-migration.sh).
> > > > > >
> > > > > > Hi Ron,
> > > > > >
> > > > > > Thanks for the KIP.
> > > > > >
> > > > > > About deprecations:
> > > > > >
> > > > > > * zookeeper-security-migration: clearly, deprecating ZK access in
> > > > > > this
> > > > one
> > > > > > would not make sense, since it would defeat the whole point of the
> > > > tool :)
> > > > > >
> > > > > > * kafka-reassign-partitions: ZK access should be deprecated here.
> > > > KIP-455
> > > > > > implementation has dragged a bit, but this should get done soon.
> > > > Certainly
> > > > > > before the next release.
> > > > > >
> > > > > > * kafka-configs: I think ZK access should be deprecated here as
> > > > > > well.
> > > > I
> > > > > > agree there is a super-special bootstrapping case here, but that
> > > > > > should
> > > > > > have its own tool, not use kafka-configs.
> > > > > >
> > > > > > I will post a separate KIP for this, though.
> > > > > >
> > > > > > >
> > > > > > > GOALS:
> > > > > > > Support the secure configuration of TLS-encrypted communication
> > > > between
> > > > > > > Zookeeper and:
> > > > > > > a) Kafka brokers
> > > > > > > b) The three CLI tools mentioned above that still support
> > > > > > > direct,
> > > > > > > non-deprecated communication to Zookeeper
> > > > > > > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > > > > > > communication in CLI tools as part of this KIP; such work will
> > > > > > > occur
> > > > in
> > > > > > > future KIPs instead.
> > > > > > >
> > > > > > > PUBLIC INTERFACES:
> > > > > > > 1) The following new broker configurations will be recognized.
> > > > > > > zookeeper.client.secure (default value = false, for backwards
> > > > > > > compatibility)
> > > > > > > zookeeper.clientCnxnSocket
> > > > > > > zookeeper.ssl.keyStore.location
> > > > > > > zookeeper.ssl.keyStore.password
> > > > > > > zookeeper.ssl.trustStore.location
> > > > > > > zookeeper.ssl.trustStore.password
> > > > > > > It will be an error for any of the last 5 values to be left
> > > > unspecified
> > > > > > if
> > > > > > > zookeeper.client.secure is explicitly set to true.
> > > > > > >
> > > > > > > 2) In addition, the kafka.security.authorizer.AclAuthorizer class
> > > > > > supports
> > > > > > > the ability to connect to a different Zookeeper instance than the
> > > > one the
> > > > > > > brokers use. We therefore also add the following optional
> > > > > > > configs,
> > > > which
> > > > > > > override the corresponding ones from above when present:
> > > > > > > authorizer.zookeeper.client.secure
> > > > > > > authorizer.zookeeper.clientCnxnSocket
> > > > > > > authorizer.zookeeper.ssl.keyStore.location
> > > > > > > authorizer.zookeeper.ssl.keyStore.password
> > > > > > > authorizer.zookeeper.ssl.trustStore.location
> > > > > > > authorizer.zookeeper.ssl.trustStore.password
> > > > > > >
> > > > > > > 3) The three CLI tools mentioned above will support a new
> > > > > > --zk-tls-config-file
> > > > > > > <String: Zookeeper TLS configuration file path>" option. The
> > > > following
> > > > > > > properties will be recognized in that file, and unrecognized
> > > > properties
> > > > > > > will be ignored to allow the possibility of pointing
> > > > zk-tls-config-file
> > > > > > at
> > > > > > > the broker's config file.
> > > > > > > zookeeper.client.secure (default value = false)
> > > > > > > zookeeper.clientCnxnSocket
> > > > > > > zookeeper.ssl.keyStore.location
> > > > > > > zookeeper.ssl.keyStore.password
> > > > > > > zookeeper.ssl.trustStore.location
> > > > > > > zookeeper.ssl.trustStore.password
> > > > > > > It will be an error for any of the last 5 values to be left
> > > > unspecified
> > > > > > if
> > > > > > > zookeeper.client.secure is explicitly set to true.
> > > > > >
> > > > > > Do we really need a --zk-tls-config-file flag? It seems like this
> > > > could
> > > > > > just appear in a configuration file, which all of these tools
> > > > > > already
> > > > > > accept (I think).
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > Ron
> > > > > > >
> > > > > > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rndg...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi everyone. Let's get this discussion going again now that
> > > > > > > > Kafka
> > > > 2.4
> > > > > > > > with Zookeeper 3.5.6 is out.
> > > > > > > >
> > > > > > > > First, regarding the KIP number, the other KIP that was using
> > > > > > > > this
> > > > > > number
> > > > > > > > moved to KIP 534, so KIP 515 remains the correct number for this
> > > > > > > > discussion. I've updated the Kafka Improvement Proposal page to
> > > > list
> > > > > > this
> > > > > > > > KIP in the 515 slot, so we're all set there.
> > > > > > > >
> > > > > > > > Regarding the actual issues under discussion, I think there are
> > > > some
> > > > > > > > things we should clarify.
> > > > > > > >
> > > > > > > > 1) It is possible to use TLS connectivity to Zookeeper from
> > > > > > > > Apache
> > > > > > Kafka
> > > > > > > > 2.4 -- the problem is that configuration information has to be
> > > > passed
> > > > > > via
> > > > > > > > system properties as "-D" command line options on the java
> > > > invocation
> > > > > > of
> > > > > > > > the broker, and those are not secure (anyone with access to the
> > > > > > > > box
> > > > > > can see
> > > > > > > > the command line used to invoke the broker); the configuration
> > > > includes
> > > > > > > > sensitive information (e.g. a keystore password), so we need a
> > > > secure
> > > > > > > > mechanism for passing the configuration values. I believe the
> > > > > > > > real
> > > > > > > > motivation for this KIP is to harden the configuration mechanism
> > > > for
> > > > > > > > Zookeeper TLS connectivity.
> > > > > > > >
> > > > > > > > 2) I believe the list of CLI tools that continue to use direct
> > > > > > Zookeeper
> > > > > > > > connectivity in a non-deprecated fashion is:
> > > > > > > > a)
> > > > zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > > > > > > > b)
> > > > > > > >
> > > > > >
> > > > kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > > > > > > > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> > > > > > > >
> > > > > > > > 3) I believe Kafka.admin.ConfigCommand presents a conundrum
> > > > because it
> > > > > > > > explicitly states in a comment that a supported use case is
> > > > > > bootstrapping a
> > > > > > > > Kafka cluster with encrypted passwords in Zookeeper (see
> > > > > > > >
> > > > > >
> > > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> > > > > > ).
> > > > > > > > This means it will be especially difficult to deprecate this
> > > > particular
> > > > > > > > direct Zookeeper connectivity without a different storage
> > > > mechanism for
> > > > > > > > dynamic configuration values being available (i.e. the
> > > > > > > > self-managed
> > > > > > quorum
> > > > > > > > referred to in KIP-500).
> > > > > > > >
> > > > > > > > I think it would be easier and simpler to harden the Zookeeper
> > > > > > > > TLS
> > > > > > > > configuration in both Kafka and in CLI tools compared to trying
> > > > > > > > to
> > > > > > > > deprecate the direct Zookeeper connectivity in the above 3 CLI
> > > > tools --
> > > > > > > > especially when ConfigCommand has no obvious short-term path to
> > > > > > deprecation.
> > > > > > > >
> > > > > > > > Regarding how to harden the TLS configuration, adding the
> > > > appropriate
> > > > > > > > configs to Kafka is an obvious choice for the brokers. Not that
> > > > these
> > > > > > > > values cannot be dynamically reconfigurable because the values
> > > > would be
> > > > > > > > required to connect to Zookeeper via TLS in the first place.
> > > > > > > >
> > > > > > > > Regarding how to harden the TLS configuration for the 3
> > > > > > > > remaining
> > > > CLI
> > > > > > > > tools, it would be relatively straightforward to add support
> > > > > > > > for a
> > > > > > > > "--zk-tls-config-file <String: Zookeeper TLS configuration
> > > > > > > > system
> > > > > > > > properties file path>" option to each one.
> > > > > > > >
> > > > > > > > Thoughts?
> > > > > > > >
> > > > > > > > Ron
> > > > > > > >
> > > > > > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <
> > > > pere.ur...@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> True, the number is duplicated.
> > > > > > > >>
> > > > > > > >> do you know how can we get the problem fix? I was not aware,
> > > > > > > >> I
> > > > > > missed,
> > > > > > > >> sorry the need to add the KIP to the table.
> > > > > > > >>
> > > > > > > >> -- Pere
> > > > > > > >>
> > > > > > > >> Missatge de Jorge Esteban Quilcate Otoya <
> > > > quilcate.jo...@gmail.com>
> > > > > > del
> > > > > > > >> dia
> > > > > > > >> dt., 3 de set. 2019 a les 13:43:
> > > > > > > >>
> > > > > > > >> > Hi Pere,
> > > > > > > >> >
> > > > > > > >> > Have you add your KIP to the list here
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > > > > >> ?
> > > > > > > >> >
> > > > > > > >> > I found the KIP number assigned to another.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <
> > > > > > pere.ur...@gmail.com>
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> >> Thanks for your time Harsha,
> > > > > > > >> >> anyone else with comments? looking forward to hearing
> > > > > > > >> >> from
> > > > you.
> > > > > > > >> >>
> > > > > > > >> >> Stupid question: when do you move from discussion to vote?
> > > > > > > >> >>
> > > > > > > >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia
> > > > dv., 30
> > > > > > > >> d’ag.
> > > > > > > >> >> 2019 a les 21:59:
> > > > > > > >> >>
> > > > > > > >> >> > Thanks Pere. KIP looks good to me.
> > > > > > > >> >> > -Harsha
> > > > > > > >> >> >
> > > > > > > >> >> >
> > > > > > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
> > > > > > > >> >> pere.ur...@gmail.com>
> > > > > > > >> >> > wrote:
> > > > > > > >> >> >
> > > > > > > >> >> >> Not really,
> > > > > > > >> >> >> my idea is to keep the JAAS parameter, so people don't
> > > > > > > >> >> >> see
> > > > > > major
> > > > > > > >> >> >> changes. But if you pass a properties file, then this
> > > > > > > >> >> >> takes
> > > > > > > >> precedence
> > > > > > > >> >> over
> > > > > > > >> >> >> the other, with the idea that you can do sasl as well
> > > > > > > >> >> >> with
> > > > the
> > > > > > > >> >> properties
> > > > > > > >> >> >> files.
> > > > > > > >> >> >>
> > > > > > > >> >> >> Makes sense?
> > > > > > > >> >> >>
> > > > > > > >> >> >> -- Pere
> > > > > > > >> >> >>
> > > > > > > >> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia
> > > > dv.,
> > > > > > 30
> > > > > > > >> >> d’ag.
> > > > > > > >> >> >> 2019 a les 19:00:
> > > > > > > >> >> >>
> > > > > > > >> >> >>> Hi Pere,
> > > > > > > >> >> >>> Thanks for the KIP. Enabling SSL for zookeeper
> > > > for
> > > > > > Kafka
> > > > > > > >> >> makes
> > > > > > > >> >> >>> sense.
> > > > > > > >> >> >>> "The changes are planned to be introduced in a
> > > > > > > >> >> >>> compatible
> > > > way,
> > > > > > by
> > > > > > > >> >> >>> keeping the current JAAS variable precedence."
> > > > > > > >> >> >>> Can you elaborate a bit here. If the user configures a
> > > > > > > >> >> >>> JAAS
> > > > > > file
> > > > > > > >> with
> > > > > > > >> >> >>> Client section it will take precedence over zookeeper
> > > > > > > >> >> >>> SSL
> > > > > > configs?
> > > > > > > >> >> >>>
> > > > > > > >> >> >>> Thanks,
> > > > > > > >> >> >>> Harsha
> > > > > > > >> >> >>>
> > > > > > > >> >> >>>
> > > > > > > >> >> >>>
> > > > > > > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
> > > > > > > >> >> pere.ur...@gmail.com>
> > > > > > > >> >> >>> wrote:
> > > > > > > >> >> >>>
> > > > > > > >> >> >>>> Hi,
> > > > > > > >> >> >>>> quick question, I saw in another mail that 2.4 release
> > > > > > > >> >> >>>> is
> > > > > > planned
> > > > > > > >> for
> > > > > > > >> >> >>>> September. I think it would be really awesome to have
> > > > this for
> > > > > > > >> this
> > > > > > > >> >> >>>> release, do you think we can make it?
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> -- Pere
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> Missatge de Pere Urbón Bayes <pere.ur...@gmail.com> del
> > > > dia
> > > > > > dj.,
> > > > > > > >> 29
> > > > > > > >> >> >>>> d’ag. 2019 a les 20:10:
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> Hi,
> > > > > > > >> >> >>>> this is my first KIP for a change in Apache Kafka, so
> > > > > > > >> >> >>>> I'm
> > > > > > really
> > > > > > > >> need
> > > > > > > >> >> >>>> to the process. Looking forward to hearing from you and
> > > > learn
> > > > > > the
> > > > > > > >> >> best
> > > > > > > >> >> >>>> ropes here.
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> I would like to propose this KIP-515 to enable the
> > > > > > > >> ZookeeperClients
> > > > > > > >> >> to
> > > > > > > >> >> >>>> take full advantage of the TLS communication in the new
> > > > > > Zookeeper
> > > > > > > >> >> 3.5.5.
> > > > > > > >> >> >>>> Specially interesting it the Zookeeper Security
> > > > > > > >> >> >>>> Migration,
> > > > > > that
> > > > > > > >> >> without
> > > > > > > >> >> >>>> this change will not work with TLS, disabling users to
> > > > > > > >> >> >>>> use
> > > > > > ACLs
> > > > > > > >> when
> > > > > > > >> >> the
> > > > > > > >> >> >>>> Zookeeper cluster use TLS.
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> link:
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>>
> > > > > > > >> >>
> > > > > > > >>
> > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> Looking forward to hearing from you on this,
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> /cheers
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> --
> > > > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > > > >> >> >>>> Software Architect
> > > > > > > >> >> >>>> http://www.purbon.com
> > > > > > > >> >> >>>> https://twitter.com/purbon
> > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> --
> > > > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > > > >> >> >>>> Software Architect
> > > > > > > >> >> >>>> http://www.purbon.com
> > > > > > > >> >> >>>> https://twitter.com/purbon
> > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>
> > > > > > > >> >> >>>
> > > > > > > >> >> >>
> > > > > > > >> >> >> --
> > > > > > > >> >> >> Pere Urbon-Bayes
> > > > > > > >> >> >> Software Architect
> > > > > > > >> >> >> http://www.purbon.com
> > > > > > > >> >> >> https://twitter.com/purbon
> > > > > > > >> >> >> https://www.linkedin.com/in/purbon/
> > > > > > > >> >> >>
> > > > > > > >> >> >
> > > > > > > >> >> >
> > > > > > > >> >>
> > > > > > > >> >> --
> > > > > > > >> >> Pere Urbon-Bayes
> > > > > > > >> >> Software Architect
> > > > > > > >> >> http://www.purbon.com
> > > > > > > >> >> https://twitter.com/purbon
> > > > > > > >> >> https://www.linkedin.com/in/purbon/
> > > > > > > >> >>
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >> Pere Urbon-Bayes
> > > > > > > >> Software Architect
> > > > > > > >> http://www.purbon.com
> > > > > > > >> https://twitter.com/purbon
> > > > > > > >> https://www.linkedin.com/in/purbon/
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
