Okay, I read the whole document. I think that for a government policy paper it is well written and it makes good points.
It may be somewhat over-optimistic because it is written by experts. They tend to have a gung-ho, can-do attitude. That's why they work in the field. It is like asking me about cold fusion. I like the way they point to previous delays and failures to meet goals, and on the other hand, they point to faster than expected progress in self-driving cars and multi-level deep learning. The part on page 33 is amusing, especially this discussion of a house-cleaning robot: Scalable Oversight: How can we efficiently ensure that the cleaning robot respects aspects of the objective that are too expensive to be frequently evaluated during training? For instance, it should throw out things that are unlikely to belong to anyone, but put aside things that might belong to someone (it should handle stray candy wrappers differently from stray cellphones). Asking the humans involved whether they lost anything can serve as a check on this, but this check might have to be relatively infrequent—can the robot find a way to do the right thing despite limited information? Safe Exploration: How do we ensure that the cleaning robot doesn’t make exploratory moves with very bad repercussions? For example, the robot should experiment with mopping strategies, but putting a wet mop in an electrical outlet is a very bad idea. It is like teaching a small child not to stick a fork into an electrical outlet. - Jed
