Mac OS X’s PDF encryption still uses 128-bit RC4 which is compatible with Adobe 5.0 and later. 128-bit RC4 is a weak encryption algorithm and people generally don’t recommend it. I use Adobe Acrobat X to encrypt my PDF files instead and choose the strongest AES256 algorithm compatible with Adobe Reader X or later. Note that there is AES256 encryption in Acrobat 9, but it has security flaw which makes it easier to brute force passwords than Acrobat 8.
Here is the setting that I use on my acrobat x to encrypt my PDF files: