Panda or zef will have the same issue for me. What is the right place to put Perl6 modules?panda installed them in /usr/share/perl6/site/lib/ but Perl6 looks only in /usr/share/perl6/site/ in my case.
If you ran Configure with --prefix specifying the final install dir, not the temporary buildroot, might that help? Also, have you tried zef instead of panda? When I built perl6 on (NB _on_, not just _for_) Android, I found zef easier to get going. I put a note up about that on bet.github.io.