SMTP BDAT support

[awelzel]

There's an SMTP extension named CHUNKING that adds the BDAT command in addition to the DATA command.

Is this worth adding the the old SMTP analyzer or spicy/future material?

---

Some references

RFC: https://www.rfc-editor.org/rfc/rfc3030#section-4.1

Example pcap from the wireshark mailing list: https://www.wireshark.org/lists/wireshark-bugs/201610/msg00410.html
foo.pcapng.zip

Postfix support: https://www.postfix.org/BDAT_README.html

Exim support (option chunking_advertises_hosts *): https://www.exim.org/exim-html-current/doc/html/spec_html/ch-main_configuration.html
Exim vulnerability in bdat support: https://lists.exim.org/lurker/message/20171125.034842.d1d75cac.en.html

Microsoft exchange non-delivery report explanation

If a message contains bare line feeds, the SMTP Chunking feature is required to transmit the message between email servers. Chunking uses the SMTP BDAT command as defined in RFC 3030. If the destination email server doesn't support BDAT, then it can't accept messages that contain bare line feeds.
...

Pointer at Suricata's support: https://github.com/OISF/suricata/blob/2158dbf3baf1e519b28569e57999a0e6d81279a4/src/app-layer-smtp.c#L722

---

Running above pcap with Zeek 5.0 looks as follows. Individual "data" lines are interpreted as commands:

smtp_request, C2BYnjHaArYFLdhMi, T, EHLO, testhost.test.ex                                                                                                                                                                                                                                                                                                  
smtp_request, C2BYnjHaArYFLdhMi, T, MAIL, FROM:<>                                                                                                                                                                                                                                                                                                           
smtp_request, C2BYnjHaArYFLdhMi, T, RCPT, TO:<[email protected]>                                                                                                                                                                                                                                                                                                    
smtp_request, C2BYnjHaArYFLdhMi, T, BDAT, 295                                                                                                                                                                                                                                                                                                               
smtp_unexpected, C2BYnjHaArYFLdhMi, unknown command, Received:                                                                                                                                                                                                                                                                                              
smtp_unexpected, C2BYnjHaArYFLdhMi, unknown command,                                                                                                                                                                                                                                                                                                        
smtp_unexpected, C2BYnjHaArYFLdhMi, unknown command,                                                                                                                                                                                                                                                                                                        
smtp_unexpected, C2BYnjHaArYFLdhMi, unknown command,                                                                                                                                                                                                                                                                                                        
smtp_unexpected, C2BYnjHaArYFLdhMi, unknown command, Subject:                                                                                                                                                                                                                                                                                               
smtp_unexpected, C2BYnjHaArYFLdhMi, unknown command, Message-Id:                                                                                                                                                                                                                                                                                            
smtp_unexpected, C2BYnjHaArYFLdhMi, unknown command, From:                                                                                                                                                                                                                                                                                                  
smtp_unexpected, C2BYnjHaArYFLdhMi, unknown command, Date:                                                                                                                                                                                                                                                                                                  
smtp_unexpected, C2BYnjHaArYFLdhMi, unknown command,                                                                                                                                                                                                                                                                                                        
smtp_request, C2BYnjHaArYFLdhMi, T, BDAT, 8380 LAST                                                                                                                                                                                                                                                                                                         
smtp_unexpected, C2BYnjHaArYFLdhMi, unknown command, 1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890                                                                                                                                                                                                   
smtp_unexpected, C2BYnjHaArYFLdhMi, unknown command, 1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890      
...

[ckreibich]

Interesting. Seems the main reason for this (per the RFC) is to switch from line-based messages to large binary data, for which it's handy to know the whole size as expressed in the BDAT line. That RFC is 20 years old, gulp...

I'm wondering if expanding the analyzer would be pretty expensive — understanding chunking & composing messages from it, perhaps needing additional content types, etc, which would make it a contender for Spicy. On the other hand SMTP may not be a great first Spicy analyzer in Zeek as it sits between the ContentLine and MIME analyzers.

@rsmmr / @bbannier, what do you guys think? I'm wondering how we can start tackling Spicy analyzers in Zeek.

[rsmmr]

Yeah, adding this to the existing analyzer doesn't seem trivial, and I agree that it looks like a good candidate to port to Spicy after we got some more experience with that. In terms of moving to Spicy, I'd start with a trivial analyzer, like Finger, so that we can focus on infrastructure first. Maybe I'll just go ahead with that, let me think about it.

[ckreibich]

I looked into this a bit today since another situation has come up where Zeek got confused by BDAT.

I noticed that it seems common that even though chunking support is supposed to enable large file transfer via multiple chunks, it's actually just a single chunk — so effectively "DATA with looking for a .-terminator" just becomes "read n bytes after the BDAT line". That could be a first-step implementation so we at least do better than right now. For a really minimal implementation we could even just ensure that we skip the BDAT chunks, to stop confusion of chunk content with SMTP verbs.

Btw RFC 3030 has two parts — chunked transport and binary MIME transport encoding. I think we already support the latter, so that's nice.

I'm attaching a zip with three pcap examples, found online (one from the Suricata guys, nice work team!), with a bit of cleanup to be on port 25 etc. None of them have multiple BDAT chunks.

smtp-bdat-pcap.zip

[awelzel]

@ckreibich - what's the origin of smtp-bdat-pipeline-8bitmime.pcap? It looks like a nice pcap to use, but I'm not sure it's okay to include it in our repo.