I don't have MATLAB on the system I'm on at the moment but trying your (slightly modified) example with octave instead works:
#+begin_src org ,#+BEGIN_SRC octave :results output pkg load symbolic syms a b A=[a,b;b,a] eig(A) ,#+END_SRC ,#+results: ,#+begin_example OctSymPy v2.2.4: this is free software without warranty, see source. Initializing communication with SymPy using a popen2() pipe. Some output from the Python subprocess (pid 28058) might appear next. OctSymPy: Communication established. SymPy v0.7.6.1. A = (sym 2×2 matrix) ⎡a b⎤ ⎢ ⎥ ⎣b a⎦ ans = (sym 2×1 matrix) ⎡ ____⎤ ⎢ ╱ 2 ⎥ ⎢a - ╲╱ b ⎥ ⎢ ⎥ ⎢ ____⎥ ⎢ ╱ 2 ⎥ ⎣a + ╲╱ b ⎦ ,#+end_example #+end_src I don't know if this helps you at all and/or whether the issue is with MATLAB... -- : Eric S Fraga (0xFFFCF67D), Emacs 25.0.92.1, Org release_8.3.4-668-g809a83