If you're doing this in RDS make sure to check what extensions each of your databases in your RDS instance has (we use PostGIS and hstore for example) and upgrade them incrementally. AWS provide the right libs installed to do those updates between each engine version, but not necessarily between engine versions more than 1 major version apart.
You can see which versions of the extensions are installed in RDS in each engine version on this page: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.FeatureSupport.Extensions
