Hard to say. I don't know if this helps but here's how appadmin does it for a csv file:
def csv(): import gluon.contenttype response.headers['Content-Type'] = \ gluon.contenttype.contenttype('.csv') db = get_database(request) query = get_query(request) if not query: return None response.headers['Content-disposition'] = 'attachment; filename=%s_%s.csv'\ % tuple(request.vars.query.split('.')[:2]) return str(db(query).select())