Ah, the library I was thinking of was libsl. I just looked at the source and it's almost *perfect*. It provides the right amount of abstraction I want and basic drawing functions. All that I need is a text widget implemented. A menu lookk pretty simple to cook up and a text widget shouldn't be too difficult, considering there are functions that return character hints. Reminds me of libdraw actually :)
Maybe expect a patch to libsl in the near future!