We used YUI DataTables at my last company and they worked really well but we only got up to around 1000 records on a page (with very quick paging, filtering and sorting). I've been wanting to hook them up to Web2py but haven't needed it so far.
http://developer.yahoo.com/yui/datatable/