In scenarios where multiple models are used back to back, with multiple inputs and outputs, TVM doesn't produce helpful native libraries to connect them: - `get_num_inputs()` returns all tensors instead of only the inputs of the model - `get_output(id)` has no support for strings. And since output names are mangled, it's unclear what an `id` corresponds to which output. - as mentioned in the topic "multithreading and TVM runtime", there seems to be an issue with the module factory shared between threads. In a multimode scenario, each model runs under different threads and caching the module factory doesn't work, forcing each thread to recreate it, which incurs some performance hit. - while a secondary goal, the names of operators in the graph can be many characters long, where a simple integer would suffice. - also a secondary goal, parameters saved in a library are uncompressed. When saved separately and compressed with even a simple gzip, quite a lot of space can be reclaimed.
What we need - `get_num_inputs()` to return only inputs of the model, - `get_num_params()` to return only parameters/weights, - preserve output nodes names and so `get_output(name)` works, - make sure 2 models running in their own thread can cache their module factory at setup time and reuse PackedFuncs as fast as possible, - replace parameter names with integers, - provide an option to compress parameters' tensors, especially when stored in the same library, even a default gz or LZ4 saves lots of space, and more dedicated methods could be provided by users. These would be extensions of the existing code as (most of) this information is already available in graph runtime, for example. I'm not sure if there are impacts on the rest of the codebase. What do you think? --- [Visit Topic](https://discuss.tvm.apache.org/t/c-c-runtime-multimodel-support/8518/1) to respond. You are receiving this because you enabled mailing list mode. To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/b053d87b5dca7f4135f9674b99c70725f4ada8491cd7af2cf57a8d39f05b9f7b).
