> What could be the cause? Is there any way to increase the MAXDSIZ to > nearly 3GB?
No. Our i386 architecture is a bit special. Since older machines don't have a NX bit, we invented a "line-in-the-sand" scheme using segment limits. In a userland process, this places code below 512MB, and data above that, and shared libraries stradle the distance if needed. We are doing PIE ASLR, so it is unclear exactly how large a linear-range of memory remains. This is what is causing the effects you are seeing. If you want more, move to amd64.
