Postfix/Mailbox_quotas
This is a bit messy because quotas can be implemented in many ways. Dovecot is a nice LDA, but it doesn't have trivial quota support. Also, you need a hack if you want to reject during the initial transaction.
Dovecot's quota docs: http://wiki.dovecot.org/Quota
A not-bad alternative is to periodically populate a table with over-quota mailboxes, and use an access table in your smtpd restrictions. This might take a while if you have lots of mailboxes.
Consider a policyd service that queries mailbox sizes on demand. Or uses Dovecot's maildir++ quota files.