Re: How to load Python Pickle File in Spark Data frame

Best Regards Roland Johann Software Developer/Data Engineer phenetic GmbH Lütticher Straße 10, 50674 Köln, Germany Mobil: +49 172 365 26 46 Mail: roland.joh...@phenetic.io Web: phenetic.io Handelsregister: Amtsgericht Köln (HRB 92595) Geschäftsführer: Roland Johann, Uwe Reimann > Am 26.08.2

Re: Please help view the problem of spark dynamic partition

ed by year/month/day INSERT OVERWRITE TABLE bsl12.email_edge_lyh_partitioned2 PARTITION (year, month, day) SELECT year(link_crtd_date, '-MM-dd') as year, month(link_crtd_date, '-MM-dd') as month, day(link_crtd_date, '-MM-dd') as day, * FROM bsl1

Re: [K8S] properties file via SPARK_CONF_DIR and --properties-file prevents definition of own properties via secrets/own mounts

I would like to discuss if we can introduce a change here - maybe it’s also possible to get that into 2.4.4? I’m keen to contribute that change. Please let me know your thoughts on this. King Regards Roland Johann Software Developer/Data Engineer phenetic GmbH Lütticher Straße 10, 50674 Köln

[K8S] properties file via SPARK_CONF_DIR and --properties-file prevents definition of own properties via secrets/own mounts

properties, such as from secrets - currently it seems to be the only possibility to define spark config properties from k8s secrets. Is the current implementation intent? If yes, what are the reasons behind that decision? Thanks and kind regards Roland Johann Software Developer/Data Engineer