There is a bug in pulsar-admin CLI 3.3.x & 4.0.x: "pulsar-admin topics set-retention" or "pulsar-admin topicPolicies set-retention" silently disables retention when the specified size is >2GB. The workaround is to use pulsar-admin or pulsar-shell from 3.0.x, or to use the Java API or REST API directly.
This is fixed by PR https://github.com/apache/pulsar/pull/23689. The fix will be released in 4.0.2. Since 3.3.x is now out of support (support ended December 5th), there is no plan to release 3.3.4. -Lari