It is getting access as an element and vice versa, it switches both. So I am never sure whether I am receiving (user token) or (thing1 thing2).
On Thu, Feb 13, 2020 at 4:17 PM Luke Cwik <[email protected]> wrote: > If Dataflow changes the access pattern, it should also apply a wrapper > function around how the side input is accessed and all of this should be > effectively opaque to your execution. > > Can you provide more details as to what the "mess" is? > > On Thu, Feb 13, 2020 at 10:48 AM André Rocha Silva < > [email protected]> wrote: > >> Hello everybody >> >> I am facing a problem with a pipeline that runs perfectly on >> directrunner, but when it comes to dataflow, it turns into a mess. It >> changes the element and the side input (access). >> >> The side input reads only a line with credentials. >> >> Any thoughts on how its done are more than welcome. How do you manage >> sensitive information in templated pipelines? >> >> It is something like this: >> >> class GetStuff(beam.DoFn): >> >> def __init__(self, input1, input2): >> self.input1 = input1 >> self.input2 = input2 >> >> def process(self, element, access): >> user, token = access.split('\t') >> >> thing1, thing2 = element.split('\t') >> >> >> credentials_pipe = ( >> p >> | 'Get credentials' >> beam.io.ReadFromText(user_options.credentials) >> ) >> >> main_pipe = ( >> p >> | 'Get information' >> beam.io.ReadFromText(user_options.input_file) >> | 'Get prediction from severity' >> beam.ParDo(GetPrediction( >> user_options.input1, >> user_options.input2, >> ), beam.pvalue.AsSingleton(credentials_pipe)) >> ) >> >> p.run() >> >> >> -- >> >> *ANDRÉ ROCHA SILVA* >> * DATA ENGINEER* >> (48) 3181-0611 >> >> <https://www.linkedin.com/in/andre-rocha-silva/> /andre-rocha-silva/ >> <http://portaltelemedicina.com.br/> >> <https://www.youtube.com/channel/UC0KH36-OXHFIKjlRY2GyAtQ> >> <https://pt-br.facebook.com/PortalTelemedicina/> >> <https://www.linkedin.com/company/9426084/> >> >> -- *ANDRÉ ROCHA SILVA* * DATA ENGINEER* (48) 3181-0611 <https://www.linkedin.com/in/andre-rocha-silva/> /andre-rocha-silva/ <http://portaltelemedicina.com.br/> <https://www.youtube.com/channel/UC0KH36-OXHFIKjlRY2GyAtQ> <https://pt-br.facebook.com/PortalTelemedicina/> <https://www.linkedin.com/company/9426084/>
