mailstore-notes.dwlab.me - ServerUnmappableException

Extracting unknown e-mail addresses from journal-type profile's debug log

2023-01-21T00:00:00+00:00

🔗</a>Bulk import of messages without a list of all known e-mail addresses</h2>
🔗</a>Preface</h3>
This article describes how to perform a one-time archive or migration of messages into MailStore where the messages are available in EML files, PST files, an Exchange journal mailbox or a Multidrop mailbox and there is no definitive list of users / e-mail addresses available.</p>
All steps are to be performed in MailStore Client, logged in as an admin, unless otherwise described.</p>
I will be using `example.com</code> as the domain for my company in this article.</p>`
🔗</a>Licensing</h3> You will need a user license for each user for the archiving process. MailStore's 30-day trial has a 500-user limit and can be used for the initial archiving, or you could import users in batches instead.</p> 🔗</a>Creating the initial users (optional)</h3> To start off with, if you will eventually be using Directory Services to synchronize users then you can set it up first. This step is optional, and Directory Services is not actually needed at all, but by first synchronizing users it will reduce the number of users that need to be manually created.</p> I'll be using the full e-mail address in this article but if your Directory Services synchronizes with Active Directory then it may use only the local part. You can modify the "CreateUser" API call slightly to accommodate. While it is possible to rename users/archives after the fact it will save some time to use the correct configuration from the server.</p> If you aren't using Directory Services or simply are not ready to synchronize users, you just need at least one user with an e-mail address attached to their account. You could edit the admin account and set admin@example.com</code> for the address.</p> 🔗</a>Archiving Profile</h3> You can archive from EML files, PST files of type Multiple Users</em> or Exchange Journal</em>, or Multidrop</em> accounts, the process is similar for all message sources.</p> 🔗</a>EML files</h4> Go to Archive E-mail</em>.</li> Create a new profile --> E-mail Files</em> --> Multiple Users</em>.</li> Set the Directory</em> to a directory that contains the EML files to be archived.</li> Set "Messages with unknown e-mail addresses" to "Don't archive".</li> Leave "Delete them in origin mailbox" unchecked.</li> Run the archiving profile.</li> </ul> 🔗</a>Running the archiving profile for the first time</h4> You can start with a subdirectory that contains a few messages to understand the process, and later run the profile again against the same messages without duplicating any messages in the archive. Depending on the amount of mail you have to import, it may take several hours to perform the initial archiving run which generates the list we will use going forward.</p> Any user that already exists in MailStore at this point will have their messages archived, while messages that do not belong to any known user will have a debug log entry created, providing us a list of needed users.</p> 🔗</a>Get the debug log</h3> Click Details</em>.</li> You should see a number of messages that are skipped with the message because the target user archive could not be determined based on the e-mail addresses</code>.</li> Click on Debug Log</em> and Select in Explorer</em>.</li> Copy the log file to another directory as MailStore will remove the file when you close the dialog.</li> Return to the details dialog and close it (this dialog blocks further access to MailStore Client).</li> </ul> You may want to open the log to take a look, in which case look for the lines that look like this:</p> MailStore.Common.Interfaces.ServerUnmappableException: MailStore is unable to determine where to store this email. Please ensure that e-mail addresses are specified in the users' settings. Senders and recipients: noreply@mailstore.com, sherry.hall@example.com, frank.clark@example.com</code></p> noreply@mailstore.com</code> is an example of an external address, while the@example.com</code> addresses belong to our example company in this documentation. Note that there may be any number of addresses, but most messages will just have two.</p> 🔗</a>Get the e-mail addresses out of the debug log</h3> We need to get the e-mail addresses from the debug log lines above into a list, one e-mail address per line, with all other punctuation and spaces removed. Duplicates will not cause errors, but will take time to process, so it is ideal to remove them from the list but it is not required.</p> There are a lot of ways you can do this, including doing it entirely by hand for a small number of users, or you could use any regex capable editor or other tools. If you are doing it manually and miss a user, just create the user and run the archiving process again, MailStore will skip the messages already in the archive.</p> Aside from preparing this list manually, I will describe three ways to extract the data from the debug log.</p> The goal is to change this:</p> MailStore.Common.Interfaces.ServerUnmappableException: MailStore is unable to determine where to store this email. Please ensure that e-mail addresses are specified in the users' settings. Senders and recipients: noreply@mailstore.com, sherry.hall@example.com, frank.clark@example.com MailStore.Common.Interfaces.ServerUnmappableException: MailStore is unable to determine where to store this email. Please ensure that e-mail addresses are specified in the users' settings. Senders and recipients: joe.smith@example.com, frank.clark@example.com </code></pre> To a test file called userlist.txt</code> that looks like this:</p> sherry.hall@example.com frank.clark@example.com joe.smith@example.com </code></pre> Note that the original list had external addresses like noreply@mailstore.com</code> that were removed, and also frank.clark@example.com</code> was listed twice but is now only listed once.</p> If you built the userlist.txt</em> file manually, move ahead to Create the users</strong>, otherwise read on for automated ways of creating the list.</p> 🔗</a>Linux / bash command line</h4> The Linux/WSL steps are located in a separate article</a> and will only take a couple of minutes to complete.</p> 🔗</a>Manually (via Excel)</h4> The Excel steps are located in a separate article</a> and only needed if you do not have access to a Linux console for a few minutes.</p> 🔗</a>Create the users</h3> Return to MailStore Client</em>.</li> Right click on the profile created earlier and select Create Scheduled Task</em>.</li> Click Copy CMD Line</em>, then cancel the scheduled task dialog.</li> Paste the line into your favourite text editor.</li> </ul> The details of the line will vary, mine looks like this:</p> "C:\Program Files (x86)\MailStore\MailStore Server\MailStoreCmd.exe" --h="localhost" --cred="admin@127.0.0.1" -c import-execute --id=3 --user="admin" --verbose</code></p> We need the part up to and including -c</code>, so like this:</p> "C:\Program Files (x86)\MailStore\MailStore Server\MailStoreCmd.exe" --h="localhost" --cred="admin@127.0.0.1" -c</code></p> First we need to create each user, which we can do using a for</code> loop through the userlist.txt</em> file created above.</p> for /F %x in (userlist.txt) do "C:\Program Files (x86)\MailStore\MailStore Server\MailStoreCmd.exe" --h="localhost" --cred="admin@127.0.0.1" -c CreateUser --userName %x --privileges "none" </code></pre> Then set the e-mail addresses field:</p> for /F %x in (userlist.txt) do "C:\Program Files (x86)\MailStore\MailStore Server\MailStoreCmd.exe" --h="localhost" --cred="admin@127.0.0.1" -c SetUserEmailAddresses --userName %x --emailAddresses %x </code></pre> A note about using bare usernames (without the @example.com</code> domain portion): You could remove the domain from the lines in userlist.txt</em> and then when setting e-mail addresses replace the --emailAddresses %x</code> part with --emailAddresses %x@example.com</code> instead.</p> Creating totally different usernames and e-mail addresses (e.g. fclark</code> and frank.clark@example.com</code>) or listing multiple e-mail addresses is out of scope for this article, but you could modify the command lines above to match your environment.</p> 🔗</a>Running the archive profile</h3> Once your users are created, return to the Archive E-mail</em> panel and run your Multidrop Mailbox (File System)</code> profile again, this time it should process all messages.</p> Click the Details</em> link again to review the results. If there are any messages that still don't have a user associated with them, create the missing users and repeat the archive.</p> It is likely that you will have new messages</em> archived and some messages already existed in the archive</em>, but you should not have any further messages have been skipped</em>.</p> 🔗</a>Cleanup unneeded users</h3> At this point all of your source message files have been archived, and these files are no longer needed by MailStore.</p> It is likely you'll want to delete the users created, you can do this using this command:</p> for /F %x in (userlist.txt) do "C:\Program Files (x86)\MailStore\MailStore Server\MailStoreCmd.exe" --h="localhost" --cred="admin@127.0.0.1" -c DeleteUser --userName %x </code></pre> Note that this will remove all users listed, including those that were previously created by Directory Services.</p> Instead you could also use MailStore Client's Users</em> list. In this case, select all users, hold control and unselect the current admin user, then delete all users.</p> If any users were created by Directory Services and deleted above they will be recreated by Directory Services again, with their default set of permissions.</p> 🔗</a>Final notes</h3> A few more bits of information about the process that did not fit neatly above.</p> 🔗</a>Creating a user that has permissions to access the entire archive</h3> You will likely now need to create one or more users that have access to the entire archive. From the users editor you can give a user the ability to read a specific archive, but this would take a lot of clicking to give a single user access to many archives.</p> Instead you can create an Auditor</em> user that has read access to all archives without admin permissions. Go to Compliance</em> -> Compliance General</em> and create an auditor user.</p> You can create multiple auditor users if desired and once they're created you can rename them to the user's normal account name. If the target user already exists from Directory Services, delete their old account and rename the auditor user, then change their Authentication Type</em> to Directory Services</code>. Directory Services will update the full name and e-mail address the next time it synchronizes.</p> 🔗</a>User Interface</h4> In the Progress View</em> dialog when you click the Details</em> link, the dialog blocks you from accessing MailStore Client's interface until it is closed. I would therefore commend opening the debug log via Windows Explorer, copy the file, and return to MailStore Client to close this dialog otherwise it appears that MailStore Client is hung if you try to access other dialogs.</p> The debug log is only preserved while you leave Progress View</em> open, if the Debug Log</em> button is missing you are probably looking at Recent Results which does not contain the needed information.</p> 🔗</a>Archiving</h4> MailStore's archiving profiles all skip messages already in the target archive/folder, therefore there is no need to remove messages in the archive if you repeat the process after creating new users or adding more EML files.</p> If you change the folder names in the archiving profiles, this will</em> cause duplicates to be created. I would recommend using the defaults, or at least, be sure to get it right.</p> 🔗</a>Do not delete messages from the source until you're done</h4> While it might seem like you could save some time by having MailStore delete the source messages during each archiving profile execution so that you do not need to re-scan all messages that have already been archived, this is not recommended as it has an unexpected consequence.</p> The important thing to understand is that MailStore does not have any concept of your domain or any way to determine what is a local user or non-local user. MailStore just has the list of currently-licensed users and their e-mail addresses. Imagine we have frank@example.com</code> created, but bob@example.com</code> has not been created yet and then archive with the option to delete messages enabled, here is what will happen:</p> The first message is from noreply@mailstore.com</code> and to frank@example.com</code>, so it gets archived, and deleted. Great, this is what we want.</li> The second message is from noreply@mailstore.com</code> and to bob@example.com</code>, so it gets skipped and logged, but not deleted. Great, this is what we want.</li> The third message is from frank@example.com</code> and to bob@example.com</code>, so it gets archived to frank@example.com</code> only -- This message is successfully archived, therefore if the delete option is enabled the message is deleted. When you later create bob@example.com</code> the message is already gone from the source and therefore is never archived into into bob@example.com</code> at all.</li> </ul> You will likely not even notice that bob@example.com</code> is missing the messages, but even if noticed there is no way to find and fix this later. The message is not lost, it is in frank</code>'s archive, but this can cause problems when reviewing just bob</code>'s archive later.</p>



Extracting unknown e-mail addresses from journal-type profile's debug log via Excel
2023-01-21T00:00:00+00:00
🔗</a>Preface</h2>
This is part of a larger set of steps to set up a fresh MailStore Server and migrate messages from another archiving platform where the list of older users is not known.</p>
This article just covers the steps to use Excel and the Windows Command Line to get the e-mail addresses, the full process is described here</a>.</p>
You could do this by hand using Excel as well although this has a lot more individual steps.</p>
There are multiple ways to accomplish this, including using scripting, but I will attempt to provide easy-to-understand steps that allow you to see each step, and no scripting is required. Here is what I have done in the past:</p>
Start with Windows' find.exe</em> command to filter the log to just the relevant lines:</p>
find "MailStore.Common.Interfaces.ServerUnmappableException" < *.log > userlist.txt
</code></pre>

Open the resulting file in Excel.</li>
Use Data</em> -> Text to Columns</em>.</li>
Select Delimited</em>.</li>
Set Delimiters to both "Comma" and "Space" only.</li>
Enable Treat consecutive delimiters as one</em>.</li>
Click Finish</em>.</li>
Look for the columns that contains e-mail addresses at the end of each line. This should start at AA</em>.</li>
Copy the results and paste to a new sheet.</li>
Repeat for each column that has further e-mail addresses. There may be more than AA</em> and AB</em> as e-mail can have more than a single sender/recipient pair per message. Remember there may be lines that have no 3rd address followed by more lines that have a 3rd address, but if you miss any there is an opportunity to catch them later.</li>
</ul>
Now switch to the new sheet.</p>

Use Data</em> --> Remove Duplicates</em> to clean up the list.</li>
Insert a blank row at the top.</li>
Enable Data</em> --> Filter</em>.</li>
Click the down arrow on the top line to modify the filter.</li>
Type example.com</code> in the search</em> field and press enter.</li>
Copy the results and save it as userlist.txt</code></li>
</ul>


Extracting unknown e-mail addresses from journal-type profile's debug log via Linux_WSL
2023-01-21T00:00:00+00:00
🔗</a>Preface</h2>
This is part of a larger set of steps to set up a fresh MailStore Server and migrate messages from another archiving platform where the list of older users is not known.</p>
This article just covers the steps to use WSL (Windows Subsystem for Linux) to get the e-mail addresses, the full process is described here</a>.</p>
🔗</a>Linux / bash command line</h2>
From here we need to extract the e-mail addresses that match your users. If you are comfortable at a Linux command line we can do this easily:</p>
grep --only-matching --no-filename --extended-regex "[^ ]+@example\.com\b" *.log | sort | uniq > userlist.txt
</code></pre>
If completed this way, move ahead to Create the users</strong>.</p>
🔗</a>WSL (Windows Subsystem for Linux)</h2>
Installing and learning WSL is out of scope for this article, but if you already have WSL on a machine that you can use, then from cmd.exe</code> you can call WSL and convert the debug log into the needed text-file format like this:</p>
wsl grep --only-matching --no-filename --extended-regex "[^ ]+@example\.com\b" *.log ^| sort ^| uniq > userlist.txt
</code></pre>
If completed this way, move ahead to Create the users</em>.</p>