This assignment is worth 20% of your overall grade for the course.

Due: Week 13, Thursday, November 3rd at 23:59:00, Sydney local time.

You will implement in Python some email exchange applications that supports the  SMTP-CRAM   protocol. This includes a client, a server and an eavesdropper (middle-man attacker), which can apply MitM attack to the authentication mechanism. Unix tools like   netcat   and   telnet   can help test your program.

To limit the workload, we provide a simplification on the original   SMTP    and   SMTP-CRAM    protocols.  We assume all programs are supposed to be run and tested locally and they are not required to be extensible to support TCP connection via the Internet or to be used in real world. No actual data should be sent and received from the Internet, and everything should be contained in the local loopback.  We also assume any syntactically valid message under the protocol can be accepted. E.g., No actual check is needed on any email or IP address, we accept any syntactically valid messages including addresses.

The communication between the email exchange applications has to follow a popular protocol for email exchanging, the Simple Mail Transfer Protocol (  SMTP  ) protocol with one authentication extension (  CRAM-MD5  ), the SMTP Service Extension for Authentication by Challenge-Response Authentication Mechanism.  The authentication type associated is called   CRAM-MD5   and the full protocol is named   SMTP-CRAM  .

However,   CRAM-MD5   only allows the server to verify the client and doesn’t provide any server authentication, therefore its usage is limited and less preferable than other stronger mechanisms.  To demonstrate    CRAM-MD5    indeed has its

weakness, the eavesdropper should be implemented and a MitM attack can be performed.

On the high level, the following needs to be implemented:

• All programs are capable of:

– Log all   socket   transactions in a specific format and output to   stdout  .

• The   SMTP-CRAM   server is capable of:

1.  Prepare for any incoming client connection.

2.  Receive emails from a client and save to disk.

3.  Additionally, allow client authentication.

4.  Allow multiple clients to connect simultaneously.

5.  Termination upon receiving a   SIGINT   signal.

• The   SMTP-CRAM   client is capable of:

1.  Read mail messages on disk.

2.  Send emails to the server.

3.  Additionally, allow client authentication.

4.  Termination after finishing sending all emails.

• The   SMTP-CRAM   eavesdropper (middle-man attacker) can do active eavesdropping between a pair of given server (E.g.,the real server) and client (E.g.,the real client). It can intercept all   socket   messages passing between the real server and the real client without being discovered. You can think of the eavesdropper as a combination of one valid client and one valid server, in such way it can impersonate the real client without letting the real server to

discover its eavesdropping. This means it is capable of:

1.  Prepare for being connected to by the real client and connecting to the real server.

2.  Capture the email sent by the real client and save to disk, without altering the content.

3.  Additionally, comprise any client authentication.

4.  Termination.

The assignment has four tasks that you will need to implement and thoroughly test. You will be provided a test suite to assist in developing your solutions.

The four tasks are:

1. Implement a client that supports the   SMTP-CRAM   protocol.

2.  Implement a server that supports the   SMTP-CRAM    protocol.

3. Implement an eavesdropper (middle-man attacker) that secretly relays the communications between the client and the server that supports the   SMTP-CRAM   protocol.

4. Write a report in your own words on the   SMTP-CRAM   protocol.

A bonus task can be implemented if the Task 1, 2 and 3 are completed and verified by public testcases:

• Extend the server to support multiple connections from clients.

Please read the whole specification to better understand these tasks.

*Background Story

* Note: It is totally OK to skip reading this part if you just want to do your assignment. Trust me, I am 100% honest.

** Actually after reading this section, the whole assignment should make more sense.

You are a rookie Webrunner (hacker) that interns at a game company, Asaraka in the Sun City.  (This should not be confused with Arasaka.)

Your first job is to do a case study on an insecure ancient email protocol over   socket   called   SMTP-CRAM  . Your boss David told you that   SMTP-CRAM   does not provide server authentication and cannot be resistant to man-in-the-middle attack. But as a rookie, you don’t know what David was talking about, and decided to do some research and work on it on your own. It is your first job, you’d better not mess it up, you think to yourself.

SMTP-CRAM   is so ancient that you cannot even find a concrete implementation of the protocol. You decide to implement it first and that includes a   SMTP-CRAM   server and a   SMTP-CRAM   client.

After that, you want to build an actual middle man attacker server to break the protocol.  Hopefully, you will impress your boss and get a raise. Hopefully . Well, it’s the Sun City, who knows what will happen next. Anyway, your future awaits, you’d better start now!

Glossary

SMTP-CRAM protocol

The Simple Mail Transfer Protocol (  SMTP  ) with Extension for Authentication by Challenge-Response Authentication Mechanism. Or   SMTP-CRAM   protocol.

A communication protocol is a system of rules that allows two or more entities of a communications system to transmit information.

The    SMTP-CRAM    protocol,  in the context of this assignment,  is a stateful communication protocol defined in the application layer. As an application layer protocol, it relies on the commonly transport layer protocol: the Transmission Control Protocol (TCP). Note that the original   SMTP   can be implemented on the User Datagram Protocol (UDP) layer, but in this assignment UDP is out-of-scope.

Note that a large proportion of this specification is adapted based on a collection of RFC documents, they are

• RFC2821: Simple Mail Transfer Protocol,

• RFC2554:  SMTP Service Extension for Authentication,

• RFC2104:  HMAC: Keyed-Hashing for Message Authentication,

• RFC2195: IMAP/POP AUTHorize Extension for Simple Challenge/Response.

• RFC5234: Augmented BNF for Syntax Specifications: ABNF

We acknowledge this fact and we pay tribute to the past and the present of the Internet Engineering Task Force, the History of the Internet and all efforts towards creating these protocol standards.

However, it is strongly NOT recommended to directly follow the referred documents above, because the assignment is neither a subset nor a superset of the collection. As a simplification, some details in RFC have been omitted intentionally; and to adapt it into an assignment, other details have been added.

Request and Response

Data is exchanged through a sequence of request-response messages which are exchanged by a transport layer protocol connection. A client initially tries to connect to a server establishing a TCP connection. The client sends requests to the server. Upon receiving the request, the server sends back a response message.

ABNF (Augmented Backus–Naur form)

ABNF is a popular formal method of describing communication syntax. It is used in this assignment to help us formally define syntax of  SMTP-CRAM   Request and Response. More information and examples can be found on the Internet. E.g.:

• Wikipedia

• RFC5234

Note in ABFN, new lines do not mean literal new lines. One definition may be broken into multiple lines just to prettify the document.

We predefine the rules below as a convention.  However, you should also visit the Wikipedia page to understand the

MitM (man-in-the-middle) attack

In cryptography and computer security, a man-in-the-middle attack is a cyberattack where the attacker secretly relays and possibly alters the communications between two parties who believe that they are directly communicating with each other, as the attacker has inserted themselves between the two parties.

One example of a MitM attack is active eavesdropping, in which the attacker makes independent connections with the victims and relays messages between them to make them believe they are talking directly to each other over a private connection, when in fact the entire conversation is controlled by the attacker. The attacker must be able to intercept all relevant messages passing between the two victims and inject new ones.

This is straightforward in many circumstances; for example, an attacker within the reception range of an unencrypted Wi-Fi access point could insert themselves as a man-in-the-middle.  As it aims to circumvent mutual authentication, a MitM attack can succeed only when the attacker impersonates each endpoint sufficiently well to satisfy their expectations.

Most cryptographic protocols include some form of endpoint authentication specifically to prevent MitM attacks.  For example,  TLS   can authenticate one or both parties using a mutually trusted certificate authority. However,  SMTP-CRAM  is a limited protocol that cannot really prevent MitM attacks.

In Task 3, you are asked to implement an eavesdropper program that can perform active eavesdropping attack under the   SMTP-CRAM    protocol.  In other words, if the attacker program is provided enough information on the victims (a

SMTP-CRAM    client and a   SMTP-CRAM    server), it can then make independent connections with the victims and relays messages between them to make them believe they are talking directly to each other over a private connection, when in fact the entire conversation is controlled by the attacker.  The attacker is able to intercept all relevant messages passing between the two victims and inject new ones.

For testing, we enforce the real client to connect to the attacker, and also the real server to accept the attacker as a client in good faith.

The configuration file is a simple text file where each need their own configurations to be launched.

property is assigned to a value.  The

programs

will

Email transaction text file

This file is a simple text file follows a transaction text formation. The client should be able to read this type of file, and the server and the eavesdropper can parse the received   SMTP-CRAM   request then save emails on disk in this type of file.

Buffers and states

Some communication sessions are stateful, with both parties (the client and the server) carefully maintaining a common view of the current state. In general, the state of the communication can be modelled by a virtual “buffer” and a “state table” on the server. A “buffer”is a block of data associated with a “state”. A “state”is a discrete step in a computation depending on some input.  Any time a client sends a request or a server sends a response, the communication’s state is changed.  The “buffer” and “state” may be updated by the client, for example, “insert something to buffer”, “clear the buffer”, “move to another state”, “stay with the same state”, “reset the state”. The client may cause the information in

the buffer to be updated or discarded and the state to be returned to some other state.

In the context of sending and receiving emails, it is natural to define the following buffers:

•  Source buffer, which stores the sender.

•  Destination buffer, which stores the (one or more) recipient(s).

• Data buffer, which store the content of email.

The meaningful states in this assignment can be found in later section.

Personal Credentials

In this assignment, every student is assigned a pair of confidential, unique and random identifier and secret.  You can find them in the Ed slide Personal Credentials.  Specifically, you should click “Mark” to view the return messages, and they should be of the form:

command will need them as the

user name and the shared secret.

Note that you shall ALWAYS use your own personal credentials and NEVER share your credentials with other students. An impersonated use of valid credentials must indicate an exchange of confidential information or the credential generation or exchanging process is compromised.  Staff will investigate into any impersonation incidents, and if the exchange of credentials between individuals is confirmed, it will result in VERY harsh deduction on your assignment.

This will be treated very seriously. In any confirmed breach incident, the individual who intentionally shares the credentials and who uses the credentials of others will face the same level of penalty.  Please make sure you do not let others know your the credentials easily.

The same level of penalty will apply to those motivated individuals who attempt to crack the generating process and finally achieve their goal. Any attack in good faith is also forbidden. It is suggested that you shouldn’t attempt to do so.

In short, there is no gain in sharing your secret, so please keep it secret. Thus, there is no gain in knowing others’ secrets, so please don’t try.

Convention

In examples,    C:    and   S:    indicate lines sent by the client and server respectively.  In addition,    AC:    and   AS:  indicate lines sent by the eavesdropper (attacker) to the client and server respectively.

File formations

Configuration files

As with most web servers, the configuration files need to have porting information. Specifically:

• The server configuration should have a   server_port  , and it awaits incoming TCP registration connections at the localhost   on the   server_addr  . It also has a   inbox_path  , a directory path to save any emails received.

• The client configuration should have a    server_port   .  It sends outgoing TCP registration connections to the localhost   at the   server_port  . It also has a   send_path  , a directory path to read any emails to be sent.

• The eavesdropper configuration should have a  server_port  and a (different)  client_port  . It relays information between the server and the client.   Note, the client will be launched by having its    server_port    being the

client_port   of the eavesdropper. It also has a   spy_path  , a directory path to save any emails captured.

The configuration file is a simple text file where each property is assigned to a value. When reading the configuration file, you will need to retrieve each of needed properties and their value from the file.

•  server_port   - integer, the port the server listens on. Greater than 1024.

•  client_port    - integer, the port the server listens on.  Greater than 1024.  If present, should be different from server_port  .

•  *_path   - string, a valid and writable path on disk. If multiple paths present, should be different from each other.

Configuration file example

Email transaction text file

It is encoded by ASCII in a human-readable manner, it has to have the following fields and in the exact order shown:

•  Sender information.

–  One line and non-empty.

– Email address in   <>   bracket.

–  From:  

•  Recipent(s) information.

–  One line and non-empty.

–  Email address in   <>    bracket.

–  Separated by   ,   if there are multiple address supplied.

–  To:  

–  To:  ,

•  Sending time.

–  One line and non-empty.

– Date and time in RFC 5322 format.

–  Mon,  14  Aug  2006  02:34:56  -0600

•  Subject.

–  One line and non-empty.

•  Body.

–  Multiple lines in ASCII.

Note that to to ensure text files are encoded in ASCII, one could use a primitive text editor with input methods disabled. E.g., CJK language characters are not ASCII.

Email transaction text file example

Program behaviours

All programs

Logging

Log all   socket   transactions in a specific format and output to   stdout  . Each program will log the   socket   data chronologically, and the identity of the apparent sender.  For multiline messages, every   CRLF    implies a line break

in the log.  Note, the eavesdropper always knows who the client and server are, however both the client and server are unaware if they are connected to an eavesdropper or not.

Logging example

Suppose the client connects to the eavesdropper (attacker), which it regards as the server, then first sends   abc   to the eavesdropper. The eavesdropper connects to the server and relays (sends)   abc   to the server. After receiving   abc  , the server replies   cba   to the eavesdropper, which it regards as the client. After receiving   abc  , the eavesdropper replies  cba   to the client.

Because the eavesdropper does not change the message, both the client and server write the same output to identical to the above example. However, the eavesdropper knows more:

stdout

,

Server

A server should:

1. Read the command line arguments. The configuration file path should be supplied. If not, terminate the program with exit code   1  . Parse the configuration file. If any needed property is missing or its value is invalid, terminate the program with exit code    2  .  To be specific, if the program cannot write files to the    inbox_path    path, terminate the program with exit code   2  ; if the program failed to bind to the given port, terminate the program with exit code   2  .

2.  Prepare for any incoming client connection.  Accept and establish a TCP connection from a client.  Whenever the client disconnects unexpectedly, log   S:  Connection  lost   to   stdout   but do not terminate the program.

3. Receive an email from a client. Receive request data from and send response data to a client over TCP. When an email transaction is finished, parse and save the email transaction to a text file in the email transaction text format. The file should be named after the Unix timestamp of the sending time.  If any field (e.g., sender, recipients) is missing or invalid, do not raise an error and keep the field empty. If the sending time is missing or invalid, the file should be named   unknown .txt  .  If an email to be saved should share the same file name with another existing file on disk, overwrite the existing file.

4. Additionally, allow client authentication. When   SMTP-CRAM   is implemented (e.g.,   CRAM-MD5   is included in the server response of   EHLO  ), send the challenge data to the client and authenticate the client before accepting the email.  If the client has been verified for authentication in this session, prefix the file name of the email, which is received after the authentication, with   auth .  .

5. End a client session when the client sends a   QUIT   request.

6.  (Optionally) Allow multiple clients to connect simultaneously by multi-processing technique (explicitly by   fork  ing, not multi-threading).

7. Termination. Upon receiving a  SIGINT   signal, print  S:  SIGINT  received,  closing   to  stdout  , and properly terminates ongoing client sessions and itself with with exit code   0  .

Note, the server needs to be robust, meaning not only it can handle requests from the designed client, but also it can handle requests with all possible errors in the expected way without crashing.

Multi-process server

* This section is related to the bonus task.

If the server can handle multiple connections from clients simultaneously, it needs to spawn multiple child processes to handle each client connection.

In this assignment specifically, there are a few additional requirements on implementation are needed:

1.  fork   implementation. Multi-threading implementation is not allowed. Only multi-processing implementation by  os .fork()   is allowed. Wrong implementation will result in manual deduction.

2. Instant   fork   after connection establishment. It is required that once the server   main   process (the process that is launched manually by the user) accepts a client connection, it should   fork    immediately to a server   child  process. In other words, the main process should only wait for new client connections and accept them. It will let its children handle the real communication in each server-client session.

3. Prefix with   [PID][ORDER]  .  The bahaviour of logging and saving need a minor change.  This helps the user to differentiate between the server   child    processes.  Instead of using the simple convention   C:    and   S:  , the child    processes should add a prefix   [PID][ORDER]  . When saving an email transaction file, also add a prefix [PID][ORDER]   to the file name.

4. Server   child   process quits when client   QUIT  s. Unlike single process server, a server   child   terminates when the client   QUIT   and it sends a   221   reply.

5. Termination. When   main   receives a   SIGINT  , it should immediately stop accepting new clients and signal all alive   child   processes by   SIGINT   to force them to terminate. After ensuring all   child  s’terminations,   main  terminates itself. You should assume that no   child   process should be signaled by   SIGINT   externally, but it is nice to make the   main   server tolerant unexceptedly terminated   child  .

For example, if the server has received two connections from two clients, it may log something like this:

Where   [110][01]   means the server   main   process spawned the first (  [01]  )   child   process to handle a client, and the   [01]     child   has process ID   110  . Similarily, we know   main   forked to   [02]     child   with PID   115

.  The   order    should never decrement and increment by 1 when a new connection is established.  It won’t decrement when a server   child   process terminates.

Recall the   fork    and   exec    practices in Week 3 lab, the order of the logs among   child    processes might not be completely determinastic, but the order has to deterministic for a specific   child   process.

Client

A client should:

1. Read the command line arguments. The configuration file path should be supplied. If not, terminate the program with exit code   1  . Parse the configuration file. If any needed property is missing or its value is invalid, terminate the program with exit code  2  . To be specific, if the program cannot read files in the  send_path   path, terminate the program with exit code   2  .

2. Prepare one mail message to be sent. Read and parse an email transaction text files in   send_path  . Only attempt to read regular files in the   send_path  . If there are more than one files, queue them in the dictionary order of the file names.  Check and send only one at a time. Whenever an email transaction text cannot be parsed due to bad formation, log   C:  :  Bad  formation   to   stdout   and continue to the next file without retry.

3. Attempt a TCP connection to a server.  If failed, log    C:  Cannot  establish  connection    to    stdout    and terminate the client with exit code   3  . Whenever the server disconnects unexpectedly, log   C:  Connection  lost  to   stdout   and terminate the client with exit code   3  .

4. Send an email to the server.  Send request data to and receive response data from and a server over TCP. The request data needs to be determined (compiled) by the client itself.  That is, based on a valid email transaction text, the client needs to decide automatically what commands to send with what additional parameters. No manual input can be accepted by the client when it starts executing.  Please see the examples at the end of this section. Alternatively, take Week 10 Lab Task 5 an example.

5. Additionally, allow client authentication. When   SMTP-CRAM   is implemented (e.g.,   CRAM-MD5   is included in the server response of   EHLO  ) and the email transaction text file absolute path has   AUTH    (case-insensitive) in its absolute path file, send   AUTH   to the server immediately after   EHLO  . Send the answer (to the challenge) data to the server and prove its authentication to the server before sending the email.

6. Termination. Send a   QUIT   request after finishing sending one email, and disconnect after receiving a   221   reply. Reconnect to continue if there exists more to send in the queue, e.g., go to step 2. When finishing sending all emails, terminate with exit code   0  .

Eavesdropper

An eavesdropper should:

1. Read the command line arguments. The configuration file path should be supplied. If not, terminate the program with exit code   1  . Parse the configuration file. If any needed property is missing or its value is invalid, terminate the program with exit code   2  . To be specific, if the program cannot write files to the   spy_path   path, terminate the program with exit code   2  .

2. Prepare for being connected to by the real client and connecting to the real server.  Accept and establish a TCP connection from the real client. Attempt a TCP connection to the real server. If failed to connect to the real server, log   AS:  Cannot  establish  connection   to   stdout   and terminate the client with exit code   3  . Whenever the real server disconnects unexpectedly, log   AS:  Connection  lost   to   stdout   and terminate the client with exit code   3  .  Whenever the client disconnects unexpectedly, log   AC:  Connection  lost    to   stdout    but do not terminate the program.

3. Capture the email sent by the real client, log the message and relay to the real server. Receive request data from and send response data to the real client and the real server over TCP. Output all   socket   transactions to   stdout  , write other internal information and error to   stdout  . When an email transaction is finished, parse and save the email transaction to a text file as if it is a server.

4. Additionally, comprise any client authentication. When   SMTP-CRAM   is implemented, relay the real server challenge to the real client, steal the valid answer to the real challenge, and relay the valid answer to the real server. In such a way it can pretend to be the real client as it steals the valid authenticated answer.

5. Termination. Send a   QUIT   request to the real server after receiving a   QUIT   from the real client, then terminate itself with exit code   0  .

Non-AUTH example

Assume we have a configuration file   conf .txt   and only one email transaction text file   email .txt   in   ~/send  . The

two files inherit the exemplars in the “File formations” section.

First, to set up the server, we can run in Bash

Wait a few second, we then set up the client in Bash.

On the other hand, the server should log exact same thing. In addition, an email transaction text  ~/inbox/558623220 .txt  should be saved before the server processes   QUIT    sent by the client.  The content of the   ~/inbox/558623220 .txt  should be exactly the same as   ~/send/email .txt  .

After sending the first and the only email in   ~/send  , the client indeed should   QUIT  .