I am trying to implement decryption of AES 256 Revision 6 (PDF 2.0) as described in the ISO 32000-2 spec and having some success but getting some peculiar results that I cannot resolve and would appreciate some suggestions.
Using Acrobat X on a PC and encrypting using password encryption compatible with Acrobat X, I created a set of about 20 Acrobat X encrypted PDF tests. When I ran these through our implementation to decrypt as following ISO 32000-2 particularly Algorithms 2.A and 2.B most decrypted successfully and produced correct output. However a few failed either in the authentication phase or in the intermediate key computation stage, with the latter showing an error by an invalid decryption of the first stream of PDF encountered. Next I tried another set of different tests and also got a similar pass rate. Finally I took one unencrypted PDF test and chose the same security settings of compatible with Acrobat X, restrict editing, and printing, and allowed print and used the same password for 15 generated versions of this PDF test. 13 of these 15 resultant encrypted tests ran successfully with our implementation of the Revision 6 decryption algorithm. Two failed, with one failing a match on both user and owner key and the other failing computing the intermediate owner key.
In the past when we have implemented earlier Revision 5 256 AES, or even older compatibility versions it always was the case that you either had the software working or you didn’t. And the new PDF 2.0 2B algorithm with 64+ hashes and 64+ aes encryptions of data along with 16 byte mod 3 math computations leaves little room for error.
I believe that Acrobat when encrypting is choosing a random AES IV and all data including input /U /O /UE, and /OE strings differ. Even for the case described above of the same input test, same password, and same Acrobat encryption options. Thus the input into Algorithm 2-B will differ but the output should for authentication match the first 32 bytes of the O or U key or should result in a correct final result for intermediate owner or user key if the corresponding match occurred above.
However for the few exceptions that fail the above decryption it is not easy to determine what went wrong. Just about any change to the implementation of Algorithm 2.B breaks all working test cases instead of giving a clue as to what the issue is. The possible suspects are the new SHA-384 and SHA-512 and the encrypt code. We have used SHA-256 and the AES decrypt portion in earlier implementation of revision 5 and had no problems. The AES and hash code we are using is from Gladman1. I was wondering what others are using? It looks like Acrobat X is using RSA BSAFE Crypto – C2 at least for FIPS. Could Leonard or somebody else at Adobe tell me if this RSA software is also used in general with Acrobat X?
And I think that it would be very beneficial to have and publish a set of test vectors given input into algorithm 2.B along with correct intermediate results for each step. For each hash – including which method used per step show hash results, and also encryption step results, number of steps beyond 64 minimum, as well as final result. For the 80% of tests I have working I could produce this info. For those tests I cannot get working I would need help. Perhaps someone at Adobe or elsewhere who has had greater success than I have can help? I can provide input for the problematic tests either through this forum or privately at David.Lincoln@csr.com.