It turns out the dude's smtpd processes were getting huge, like 25MiB.
The memory usage is caused by smtpd_recipient_restrictions = check_client_access cidr:/etc/postfix/dnswl/postfix-dnswl-header, check_client_access cidr:/etc/postfix/dnswl/postfix-dnswl-permit, This are whitelists downloaded from DNSWL every night. Each file is about 4 MBytes and bloats EACH smtpd-process by 10 MBytes.
CIDR tables are held in memory. As Wietse mentions, you can use proxymap as a "shared-memory" cache for this.
You'll have to add smtpd_recipient_restrictions or check_client_access to the list of proxy_read_maps, I'm not sure which is correct.