Hi Navin If the MongoDatabaseManager is serialized, you should be able to dd it to the config of the topology. If not , the class has to be serialized.
Thanks Deepak On Thu, Apr 21, 2016 at 11:57 AM, Navin Ipe <[email protected] > wrote: > Ah ok that's a nice idea, but in my situation it's like this: > I want to initialize a database connection and also hold some database > rows and other values in a class. Once these are initialized in a class, I > don't want any other class to initialize it for the sake of saving memory > and for referential integrity. It is this initialized object that I want > all my Spouts to be able to access. > That's why I was hoping that at least if Config could hold the class, then > all Spouts and Bolts in the topology would be able to access the > initialized values, because Config is part of the topology and can be > accessed in open() and prepare(). > > > On Thu, Apr 21, 2016 at 11:08 AM, Abhishek Agarwal <[email protected]> > wrote: > >> In the prepare/open method, you can get the class name and initialize the >> instance from class name >> >> Excuse typos >> On Apr 21, 2016 9:17 AM, "Navin Ipe" <[email protected]> >> wrote: >> >>> Thanks Abhishek, but I don't quite understand what you meant. Do you >>> mean I pass a class name of a static class and then use it? >>> >>> Also took a look at the source code and this is what it says: >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> */** * Topology configs are specified as a plain old map. This class >>> provides a * convenient way to create a topology config map by providing >>> setter methods for * all the configs that can be set. It also makes it >>> easier to do things like add * serializations. * * This class also provides >>> constants for all the configurations possible on * a Storm cluster and >>> Storm topology. Each constant is paired with an annotation * that defines >>> the validity criterion of the corresponding field. Default * values for >>> these configs can be found in defaults.yaml. * * Note that you may put >>> other configurations in any of the configs. Storm * will ignore anything it >>> doesn't recognize, but your topologies are free to make * use of them by >>> reading them in the prepare method of Bolts or the open method of * >>> Spouts. */public class Config extends HashMap<String, Object> {* >>> >>> So if Storm is supposed to ignore anything it does not recognize, then >>> it's strange it's throwing an error for my serializable class. >>> >>> On Thu, Apr 21, 2016 at 12:01 AM, Abhishek Agarwal <[email protected] >>> > wrote: >>> >>>> Pass the class name in config instead of concrete object. >>>> >>>> Excuse typos >>>> On Apr 20, 2016 9:43 PM, "Navin Ipe" <[email protected]> >>>> wrote: >>>> >>>>> Does anyone know how to do this? If not adding a custom class, then at >>>>> least about how to maintain information retrieved from a database in a >>>>> single class which all spouts/bolts in all the workers will be able to >>>>> access? >>>>> >>>>> On Tue, Apr 19, 2016 at 2:52 PM, Navin Ipe < >>>>> [email protected]> wrote: >>>>> >>>>>> I have this >>>>>> >>>>>> Config config = new Config(); >>>>>> MongoDatabaseManager mongoManager = new MongoDatabaseManager(); >>>>>> config.put("MongoManager", mongoManager); >>>>>> >>>>>> and MongoDatabaseManager is an empty class: >>>>>> public class MongoDatabaseManager implements Serializable {} >>>>>> >>>>>> But after submitting the topology, I get this error: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> *14:45:18.320 [timer] DEBUG o.a.s.scheduler.SupervisorDetails - >>>>>> Creating a new supervisor (null-5b950364-129e-4e04-b10c-d1e496c12043) >>>>>> with >>>>>> resources: {"supervisor.cpu.capacity" 400.0, >>>>>> "supervisor.memory.capacity.mb" 3072.0}14:45:18.321 [main] ERROR >>>>>> o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Thread Thread[main,5,main] >>>>>> diedjava.lang.IllegalArgumentException: Topology conf is not >>>>>> json-serializable at >>>>>> org.apache.storm.testing$submit_local_topology.invoke(testing.clj:299) >>>>>> ~[storm-core-1.0.0.jar:1.0.0] at >>>>>> org.apache.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:49) >>>>>> ~[storm-core-1.0.0.jar:1.0.0] at >>>>>> org.apache.storm.LocalCluster.submitTopology(Unknown Source) >>>>>> ~[storm-core-1.0.0.jar:1.0.0] at >>>>>> com.slh.Basicstorm.BasicStorm.main(BasicStorm.java:44) ~[main/:na]* >>>>>> >>>>>> Why can't I add a custom class to config? The idea here is to store >>>>>> some info in the class so that all workers can access the information >>>>>> (because the class is part of the topology) even though they are part of >>>>>> different JVM's (the class can be accessed in the open() or prepare() >>>>>> functions of spouts or bolts, but first, I don't understand why this >>>>>> crash >>>>>> is happening). >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> Navin >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> Navin >>>>> >>>> >>> >>> >>> -- >>> Regards, >>> Navin >>> >> > > > -- > Regards, > Navin > -- Thanks Deepak www.bigdatabig.com www.keosha.net
