cbalint13 opened a new pull request, #19849:
URL: https://github.com/apache/tvm/pull/19849
This adds ```vm.builtin.shape_to_tensor``` builtin to runtime vm.
It replaces python ```relax.run.shape_to_tensor``` variant of the fuction.
---
#### Issue
Trying to do inference with a c++ only app (no public API yet) and
discovered the following issue:
```
#include <runtime/vm/vm.h>
#include <ffi/extra/module.h>
#include <runtime/vm/executable.h>
int main() {
auto mod_dso = tvm::ffi::Module::LoadFromFile("./lib.tar.so");
auto vme = mod_dso->GetFunction("vm_load_executable", false);
auto mod = (*vme)().cast<tvm::ffi::Module>();
tvm::ffi::Optional<tvm::ffi::Function> vm_ini =
mod->GetFunction("vm_initialization");
(*vm_ini)(static_cast<int>(kDLCPU),
static_cast<int>(0),
static_cast<int>(tvm::runtime::AllocatorType::kPooled),
static_cast<int>(kDLCPU),
0,
static_cast<int>(tvm::runtime::AllocatorType::kPooled));
return 0;
}
```
Exported DSO contain ```relax.run.shape_to_tensor``` call that works in
python env but not in pure runtime:
```
$ c++ -I/usr/include/tvm dso-run-tvm.cpp -o dso-run-tvm -ltvm_ffi
-ltvm_runtime
$ ./dso-run-tvm
{...}
DBG ModuleObj::GetFunction() [vm_load_executable][0]
HERE [vm_load_executable][ffi.Function]
DBG ModuleObj::GetFunction() [invoke_closure][0]
HERE [invoke_closure][ffi.Function]
{...}
DBG ModuleObj::GetFunction() [vm.builtin.shape_of][1]
DBG ModuleObj::GetFunction() [relax.run.shape_to_tensor][1] <- HERE
tvm.error.InternalError: Check failed: (func.has_value()) is false: Error:
Cannot find ffi::Function relax.run.shape_to_tensor in either Relax VM
kernel library,
or in TVM runtime ffi::Function registry, or in global Relax functions of
the VM executable
$ strings lib.tar.so.20260613 | grep relax.run
relax.run.shape_to_tensor
$
```
#### Solution
As fix, the ```relax.run.shape_to_tensor``` (python only) "leftover" it is
replaced with ```vm.builtin.shape_to_tensor```.
---
#### Results
[x] The exported DSO is instantiable in a pure C++ env having only
```tvm_runtime.so``` + ```tvm_ffi.so``` .
```
$ strings lib.tar.so | grep shape_to_tens
vm.builtin.shape_to_tensor
```
[x] Also the inference speed went up by almost 2x factor, even in a python
program (full environment):
* Before
```
[*] Analyzing with chunk size 512 and batch size 1 (Threshold: 0.2)...
Inference: 48%|██████████▋ | 9074/18737 [00:08<00:08,
1084.33chunk/s]
````
* After
```
[*] Analyzing with chunk size 512 and batch size 1 (Threshold: 0.2)...
Inference: 100%|█████████████████████| 18737/18737 [00:10<00:00,
1803.61chunk/s]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
