Difference between revisions of "CentOS 7.x Zimbra mail queue management"

From Notes_Wiki
(Created page with "<yambe:breadcrumb self="Zimbra mail queue management">CentOS_7.x_Zimbra_configuration|Zimbra configuration</yambe:breadcrumb> =CentOS 7.x Zimbra mail queue management= For gr...")
 
m
Line 2: Line 2:
=CentOS 7.x Zimbra mail queue management=
=CentOS 7.x Zimbra mail queue management=


==Graphical access to Zimbra email queue==
For graphical access to mail queue login as admin user and go to Monitor -> Mail queues.  Here Deferred queue is important and shown first.  This queue contains emails for which delivery was attempted and failed.  The option is to filter emails by Receiver domain, Origin IP, Sender domain, Receiver address, Sender address and Error.  We can click on a particular item (eg particular receiver domain) and in the messages pane below only deffered emails related to that particular item are shown.  '''If we hover mouse over''' any such deferred message row in messages list then error is displayed properly.
For graphical access to mail queue login as admin user and go to Monitor -> Mail queues.  Here Deferred queue is important and shown first.  This queue contains emails for which delivery was attempted and failed.  The option is to filter emails by Receiver domain, Origin IP, Sender domain, Receiver address, Sender address and Error.  We can click on a particular item (eg particular receiver domain) and in the messages pane below only deffered emails related to that particular item are shown.  '''If we hover mouse over''' any such deferred message row in messages list then error is displayed properly.


==Reading particular email using command-line==
If it is required to read email then we should note hexadecimal ID of the message in the below list.  Then go to folder:
If it is required to read email then we should note hexadecimal ID of the message in the below list.  Then go to folder:
<pre>
<pre>
Line 19: Line 22:


'''Avoid use of cat as it might change terminal settings due to binary data'''.  In case terminal settings are changed exit and reconnect.  Other option is to use '<tt>reset</tt>' command to reset terminal settings.
'''Avoid use of cat as it might change terminal settings due to binary data'''.  In case terminal settings are changed exit and reconnect.  Other option is to use '<tt>reset</tt>' command to reset terminal settings.
There is older article on mail queue management at [[Zimbra mail queue management]]




There is older article on mail queue management at [[Zimbra mail queue management]]
 
==Clearing long queue after attack or when the current active or incoming or deferred etc. queue are too big==
If you have recently faced attack (Often due to a particular user-ID / password getting leaked to attacker) then it is possible that Zimbra mail queue might become very big.  In such cases to clear the queue so that normal email functioning starts without loosing any of the important emails use:
 
# Login into Zimbra admin dashboard and look at email queue.  If there are many emails from same source IP, deny incoming emails from that IP using:
#:<pre>
#::    firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='185.235.15.224' reject"
#::    #firewall-cmd --add-rich-rule='rule family="ipv4" source address="185.235.15.224" reject'
#::    firewall-cmd --reload
#:</pre>
#: Refer:
#* https://fedoraproject.org/wiki/Features/FirewalldRichLanguage#Handle_rich_rules_with_the_command_line_client
#* https://access.redhat.com/discussions/1342573
# Validate that you have not blocked one or your own office IPs using: https://www.whatismyipaddress.com/ip/&lt;blocked-ip&gt;/ after replacing &lt;blocked-ip&gt; appropriately.
# Restart zimbra on another shell for server incoming connections to drop
#:<pre>
#::    su - zimbra
#::    zmcontrol restart
#:</pre>
# Go to various folders and sub-folders inside postfix/spool and move emails to hold folder
#:<pre>
#::    cd /opt/zimbra/data/postfix/spool/active
#::    mv * ../hold
#::
#::    cd /opt/zimbra/data/postfix/spool/defer
#::    #defer has 0 to f sub-folders inside
#::    for A in *; do mv -f $A/* ../hold; done
#::
#::    cd /opt/zimbra/data/postfix/spool/deferred
#::    #deferred has 0 to f sub-folders inside
#::    for A in *; do mv -f $A/* ../hold; done
#::
#::    cd /opt/zimbra/data/postfix/spool/incoming
#::    mv * ../hold
#:</pre>
# From hold folder move SPAM emails to another folder for deletion.  Note that use grep on * may not work if the no. of SPAM is too high with below output:
#:<pre>
#::    mkdir /root/2019-04-30-to-be-deleted-spam/
#::    [root@mail hold]# grep -l 185.235.15.224 * | xargs -I mv {} /root/2019-04-30-to-be-deleted-spam/
#::    -bash: /usr/bin/grep: Argument list too long
#:</pre>
#: Hence choose a smaller subset based on prefix such as:
#::<pre>
#:::    grep -l 185.235.15.224 00* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
#:</pre>
#::  Look at email count in to-be-deleted folder
#:::<pre>
#::::    ls -1 /root/2019-04-30-to-be-deleted-spam/ | wc -l
#:::</pre>
#:::    Also look at email count in hold queue:
#::::<pre>
#:::::    ls -1 | wc -l
#::::</pre>
# Ideally no. of emails to be deleted in /root/2019-04-30-to-be-deleted-spam folder should increase and emails in hold folder should decrease over time.
# Move emails with all prefixes 0 to f using:
#:<pre>
#::    grep -l 185.235.15.224 0* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
#::    grep -l 185.235.15.224 1* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
#::    grep -l 185.235.15.224 2* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
#::    grep -l 185.235.15.224 3* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
#::    grep -l 185.235.15.224 4* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
#::    grep -l 185.235.15.224 5* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
#::    grep -l 185.235.15.224 6* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
#::    grep -l 185.235.15.224 7* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
#::    grep -l 185.235.15.224 8* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
#::    grep -l 185.235.15.224 9* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
#::    grep -l 185.235.15.224 A* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
#::    grep -l 185.235.15.224 B* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
#::    grep -l 185.235.15.224 C* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
#::    grep -l 185.235.15.224 D* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
#::    grep -l 185.235.15.224 E* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
#::    grep -l 185.235.15.224 F* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
#:</pre>
# Again check "wc -l" on important folders and "Refresh" mail queue on Zimbra Admin web dashboard.  Now again search for bad origin or source IPs and filter emails based on them:
#:<pre>
#::    grep -l 105.112.33.150 * | wc -l
#::    mv $(grep -l 105.112.33.150 *) /root/2019-04-30-to-be-deleted-spam/
#:</pre>
# Again ensure that your own office IP is not blocked using:  https://www.whatismyipaddress.com/ip/&lt;blocked-ip&gt;
# If web browser queue refresh is not working use:
#:<pre>
#::    su - zimbra
#::    zmmtactl restart
#:</pre>
#:: and try to refresh mail queue in admin dashboard again.
# Often email responses with 'timed out' or 'refused to talk to me' can be part of hold queue since we moved everything.  These can again be moved to deleted folder using:
#:<pre>
#::    mkdir /root/2019-04-30-to-be-deleted-error-responses
#::    mv $(grep -l '^status=[0-9]\.[0-9]\.[0-9]' *) /root/2019-04-30-to-be-deleted-error-responses/
#:</pre>
#::'''If you search for sasl_username in most of these messages and it is the same user id then most likely that ID is compromised.  We should reset password of that ID immediately and inform user about it using other communication channels.'''
# Other option is to grep for different types of error messages and move emails containing those messages.  Ideally the status option suggested above should capture all the below types of messages automatically.
#:<pre>
#::      mv $(grep -l 'timed out' *) /root/2019-04-30-to-be-deleted-error-responses/
#::      mv $(grep -l 'refused to talk to me' *) /root/2019-04-30-to-be-deleted-error-responses/
#::      mv $(grep -l 'lost connection' * ) /root/2019-04-30-to-be-deleted-error-responses/
#::      mv $(grep -l 'no relay available' * ) /root/2019-04-30-to-be-deleted-error-responses/
#::      mv $(grep -l 'temporarily deferred' *) /root/2019-04-30-to-be-deleted-error-responses/
#::      mv $(grep -l 'the message has been blocked' * ) /root/2019-04-30-to-be-deleted-error-responses/
#::
#::      mv $(grep -l 'Temporarily unable to process your email' *) /root/2019-04-30-to-be-deleted-error-responses/
#::      mv $(grep -l 'Could not complete sender verify callout' *) /root/2019-04-30-to-be-deleted-error-responses/
#::      mv $(grep -l 'does not have an MX record' *) /root/2019-04-30-to-be-deleted-error-responses/
#::      mv $(grep -l 'Connection refused' *) /root/2019-04-30-to-be-deleted-error-responses/
#::      mv $(grep -l 'Host or domain name not found' *) /root/2019-04-30-to-be-deleted-error-responses/
#::      mv $(grep -l 'No route to host' *) /root/2019-04-30-to-be-deleted-error-responses/
#::      mv $(grep -l 'Client host rejected' *) /root/2019-04-30-to-be-deleted-error-responses/
#::      mv $(grep -l 'resource temporarily unavailable' *) /root/2019-04-30-to-be-deleted-error-responses/
#::      mv $(grep -l 'Relay access denied' *) /root/2019-04-30-to-be-deleted-error-responses/
#::      mv $(grep -l 'Transient reject by behaviour spam' *) /root/2019-04-30-to-be-deleted-error-responses/
#:</pre>
# Move domain specific emails to spam folder:
#:<pre>
#::      mv $(grep -l emailsecurity.com *) /root/2019-04-30-to-be-deleted-spam/
#:</pre>
# Requeue emails from held queue which are legitimate
# Create folder for emails that could be important but are not readable by graphical wizard (ID shown in messages pane below):
#:<pre>
#::      /root/2019-04-30-could-be-important
#::      mv 6C6C3F10AFBD /root/2019-04-30-could-be-important/
#:</pre>
# Ideally hold graphical pane should show details of sender, receiver, etc. and we should be able to requeue based on that information.
 






<yambe:breadcrumb self="Zimbra mail queue management">CentOS_7.x_Zimbra_configuration|Zimbra configuration</yambe:breadcrumb>
<yambe:breadcrumb self="Zimbra mail queue management">CentOS_7.x_Zimbra_configuration|Zimbra configuration</yambe:breadcrumb>

Revision as of 06:02, 26 September 2019

<yambe:breadcrumb self="Zimbra mail queue management">CentOS_7.x_Zimbra_configuration|Zimbra configuration</yambe:breadcrumb>

CentOS 7.x Zimbra mail queue management

Graphical access to Zimbra email queue

For graphical access to mail queue login as admin user and go to Monitor -> Mail queues. Here Deferred queue is important and shown first. This queue contains emails for which delivery was attempted and failed. The option is to filter emails by Receiver domain, Origin IP, Sender domain, Receiver address, Sender address and Error. We can click on a particular item (eg particular receiver domain) and in the messages pane below only deffered emails related to that particular item are shown. If we hover mouse over any such deferred message row in messages list then error is displayed properly.


Reading particular email using command-line

If it is required to read email then we should note hexadecimal ID of the message in the below list. Then go to folder:

cd /opt/zimbra/data/postfix/spool/ 

to see various folders related to various queues. For deferred emails visit deferred/<first-hex-letter> sub-folder inside above folder. Then use either:

less <message-ID>

while accepting warning to display binary message or use less-readable

xxd <message-ID>

Avoid use of cat as it might change terminal settings due to binary data. In case terminal settings are changed exit and reconnect. Other option is to use 'reset' command to reset terminal settings.

There is older article on mail queue management at Zimbra mail queue management


Clearing long queue after attack or when the current active or incoming or deferred etc. queue are too big

If you have recently faced attack (Often due to a particular user-ID / password getting leaked to attacker) then it is possible that Zimbra mail queue might become very big. In such cases to clear the queue so that normal email functioning starts without loosing any of the important emails use:

  1. Login into Zimbra admin dashboard and look at email queue. If there are many emails from same source IP, deny incoming emails from that IP using:
    firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='185.235.15.224' reject"
    #firewall-cmd --add-rich-rule='rule family="ipv4" source address="185.235.15.224" reject'
    firewall-cmd --reload
    Refer:
  2. Validate that you have not blocked one or your own office IPs using: https://www.whatismyipaddress.com/ip/<blocked-ip>/ after replacing <blocked-ip> appropriately.
  3. Restart zimbra on another shell for server incoming connections to drop
    su - zimbra
    zmcontrol restart
  4. Go to various folders and sub-folders inside postfix/spool and move emails to hold folder
    cd /opt/zimbra/data/postfix/spool/active
    mv * ../hold
    cd /opt/zimbra/data/postfix/spool/defer
    #defer has 0 to f sub-folders inside
    for A in *; do mv -f $A/* ../hold; done
    cd /opt/zimbra/data/postfix/spool/deferred
    #deferred has 0 to f sub-folders inside
    for A in *; do mv -f $A/* ../hold; done
    cd /opt/zimbra/data/postfix/spool/incoming
    mv * ../hold
  5. From hold folder move SPAM emails to another folder for deletion. Note that use grep on * may not work if the no. of SPAM is too high with below output:
    mkdir /root/2019-04-30-to-be-deleted-spam/
    [root@mail hold]# grep -l 185.235.15.224 * | xargs -I mv {} /root/2019-04-30-to-be-deleted-spam/
    -bash: /usr/bin/grep: Argument list too long
    Hence choose a smaller subset based on prefix such as:
    grep -l 185.235.15.224 00* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
    Look at email count in to-be-deleted folder
    ls -1 /root/2019-04-30-to-be-deleted-spam/ | wc -l
    Also look at email count in hold queue:
    ls -1 | wc -l
  6. Ideally no. of emails to be deleted in /root/2019-04-30-to-be-deleted-spam folder should increase and emails in hold folder should decrease over time.
  7. Move emails with all prefixes 0 to f using:
    grep -l 185.235.15.224 0* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
    grep -l 185.235.15.224 1* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
    grep -l 185.235.15.224 2* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
    grep -l 185.235.15.224 3* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
    grep -l 185.235.15.224 4* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
    grep -l 185.235.15.224 5* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
    grep -l 185.235.15.224 6* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
    grep -l 185.235.15.224 7* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
    grep -l 185.235.15.224 8* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
    grep -l 185.235.15.224 9* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
    grep -l 185.235.15.224 A* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
    grep -l 185.235.15.224 B* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
    grep -l 185.235.15.224 C* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
    grep -l 185.235.15.224 D* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
    grep -l 185.235.15.224 E* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
    grep -l 185.235.15.224 F* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
  8. Again check "wc -l" on important folders and "Refresh" mail queue on Zimbra Admin web dashboard. Now again search for bad origin or source IPs and filter emails based on them:
    grep -l 105.112.33.150 * | wc -l
    mv $(grep -l 105.112.33.150 *) /root/2019-04-30-to-be-deleted-spam/
  9. Again ensure that your own office IP is not blocked using: https://www.whatismyipaddress.com/ip/<blocked-ip>
  10. If web browser queue refresh is not working use:
    su - zimbra
    zmmtactl restart
    and try to refresh mail queue in admin dashboard again.
  11. Often email responses with 'timed out' or 'refused to talk to me' can be part of hold queue since we moved everything. These can again be moved to deleted folder using:
    mkdir /root/2019-04-30-to-be-deleted-error-responses
    mv $(grep -l '^status=[0-9]\.[0-9]\.[0-9]' *) /root/2019-04-30-to-be-deleted-error-responses/
    If you search for sasl_username in most of these messages and it is the same user id then most likely that ID is compromised. We should reset password of that ID immediately and inform user about it using other communication channels.
  12. Other option is to grep for different types of error messages and move emails containing those messages. Ideally the status option suggested above should capture all the below types of messages automatically.
    mv $(grep -l 'timed out' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'refused to talk to me' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'lost connection' * ) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'no relay available' * ) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'temporarily deferred' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'the message has been blocked' * ) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'Temporarily unable to process your email' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'Could not complete sender verify callout' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'does not have an MX record' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'Connection refused' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'Host or domain name not found' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'No route to host' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'Client host rejected' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'resource temporarily unavailable' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'Relay access denied' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'Transient reject by behaviour spam' *) /root/2019-04-30-to-be-deleted-error-responses/
  13. Move domain specific emails to spam folder:
    mv $(grep -l emailsecurity.com *) /root/2019-04-30-to-be-deleted-spam/
  14. Requeue emails from held queue which are legitimate
  15. Create folder for emails that could be important but are not readable by graphical wizard (ID shown in messages pane below):
    /root/2019-04-30-could-be-important
    mv 6C6C3F10AFBD /root/2019-04-30-could-be-important/
  16. Ideally hold graphical pane should show details of sender, receiver, etc. and we should be able to requeue based on that information.



<yambe:breadcrumb self="Zimbra mail queue management">CentOS_7.x_Zimbra_configuration|Zimbra configuration</yambe:breadcrumb>