Msg with invalid SecureData rejecting

Imported from previous forum

Dear community,

FIX 4.4 spec (vol. 2) says, that if “receive message with a SecureData value that cannot be decrypted” then acceptor should send Reject message in response. But should it be encrypted Reject (SessionRejectReason, RefSeqNum, etc. tags) or should it be unencrypted?

My vision is as next: if side A sends msg with incorrect SecureData, then, most likely, it will not be able to decode encrypted Reject and understand the reason of this reject.

FIX 4.4 spec (vol. 2) says, that if “receive message with a SecureData value that cannot be decrypted” then acceptor should send Reject message in response. But should it be encrypted Reject (SessionRejectReason, RefSeqNum, etc. tags) or should it be unencrypted?

My vision is as next: if side A sends msg with incorrect SecureData, then, most likely, it will not be able to decode encrypted Reject and understand the reason of this reject.

My understanding is that the spec calls out a very limited number of fields that must be unencrypted, and these are all in StandardHeader and StandardTrailer. Now there is disagreement over some of them; FIXimate says that for FIX 4.4 and FIXT.1.1, SenderCompID and TargetCompID must remain unencrypted, while the security document defining PGP/DES-MD5 encryption requires that they be encrypted. Still, I wouldn’t see any reason why you couldn’t encrypt SessionRejectReason and RefSeqNum.

Now there is a practical consideration with your example that may make this impossible. MsgSeqNum is supposed to be encrypted. It follows that if your engine can’t decrypt a message, it can’t extract MsgSeqNum, so it can’t send a Reject at all!

PGP/DES-MD5 has a number of issues and failure modes; this is just one of them. It also isn’t very secure, since each message is encrypted with 56-bit DES. Any determined adversary with even relatively modest funds would have little trouble breaking DES.

FPL recommended against usage of this kind of FIX encryption in a white paper:

http://www.fixprotocol.org/documents/3569/FIX%20Security%20White%20Paper-1.8-FINAL.pdf

If you want encryption, you should probably avoid PGP/DES-MD5, and either:

A. Wrap the session in SSL/TLS, or
B. Use an encrypted VPN, IPSec, etc.