If Dataflow changes the access pattern, it should also apply a wrapper
function around how the side input is accessed and all of this should be
effectively opaque to your execution.
Can you provide more details as to what the "mess" is?
On Thu, Feb 13, 2020 at 10:48 AM André Rocha Silva <
a.si...@portaltelemedicina.com.br> wrote:
> Hello everybody
>
> I am facing a problem with a pipeline that runs perfectly on directrunner,
> but when it comes to dataflow, it turns into a mess. It changes the element
> and the side input (access).
>
> The side input reads only a line with credentials.
>
> Any thoughts on how its done are more than welcome. How do you manage
> sensitive information in templated pipelines?
>
> It is something like this:
>
> class GetStuff(beam.DoFn):
>
> def __init__(self, input1, input2):
> self.input1 = input1
> self.input2 = input2
>
> def process(self, element, access):
> user, token = access.split('\t')
>
> thing1, thing2 = element.split('\t')
>
>
> credentials_pipe = (
> p
> | 'Get credentials' >> beam.io.ReadFromText(user_options.credentials)
> )
>
> main_pipe = (
> p
> | 'Get information' >> beam.io.ReadFromText(user_options.input_file)
> | 'Get prediction from severity' >> beam.ParDo(GetPrediction(
> user_options.input1,
> user_options.input2,
> ), beam.pvalue.AsSingleton(credentials_pipe))
> )
>
> p.run()
>
>
> --
>
> *ANDRÉ ROCHA SILVA*
> * DATA ENGINEER*
> (48) 3181-0611
>
> <https://www.linkedin.com/in/andre-rocha-silva/> /andre-rocha-silva/
> <http://portaltelemedicina.com.br/>
> <https://www.youtube.com/channel/UC0KH36-OXHFIKjlRY2GyAtQ>
> <https://pt-br.facebook.com/PortalTelemedicina/>
> <https://www.linkedin.com/company/9426084/>
>
>
