This topic lists the various troubleshooting procedures for ENS.

ENS2 Resubscription and Badge Count Accuracy Limitations

The ENS2 uses the Exchange Web Services (EWS) subscription to notify the Boxer application of any changes in an end-users mailbox, including the email notifications. The Boxer application initiates these subscriptions with the ENS and the ENS subscribes a user's account with the EWS.

The EWS is responsible for informing the ENS when there is a change in a user's mailbox. The subscriptions have limited lifetime due to the movement of mailbox, throttling, and so on. The Exchange can drop the EWS push subscriptions which are triggered by the Exchange and the ENS does not have control over the subscription lifetime. The EWS sends notification updates to the Boxer until the EWS subscription is active and alive.

To keep these subscriptions alive, the Boxer application has a check-in mechanism which validates if an EWS subscription is alive. In addition, the ENS2 is listening for status updates from the EWS. If the ENS2 does not receive a status update from the EWS, the ENS2 can send the Boxer a silent push notification to check in with the EWS.

The following figure describes the ENS resubscription process flow.

  1. The EWS sends a heartbeat signal to the ENS every 15 minutes.
  2. The ENS sends an acknowledgement to the EWS that the heartbeat signal is received.
  3. The ENS checks that the heartbeat signal is received every 30 minutes from the EWS.
  4. If the ENS does not receive a heartbeat signal, the ENS2 sends a silent notification to the Boxer application to initiate the resubscription process.
  5. The Boxer application initiates a resubscription process on receiving a silent notification.
  6. The Boxer application proactively checks the EWS subscription status with the ENS server to ensure the continuous delivery of notifications.

The ENS2 requests the Exchange to send heartbeat that a subscription is alive. When the ENS2 does not receive a heartbeat the ENS2 detects a drop in subscription from the EWS. If the subscription is not established and users are not receiving the ENS notifications, the users can manually trigger a resubscription. To trigger a resubscription or if you do not receive ENS notifications after you migrate from on-premises ENS to cloud ENS then change the notification sound in the email account settings section of the Boxer settings as shown in the following image.

The check-in mechanism used by ENS2 requires intervention from Boxer to renew the EWS subscriptions because the users credentials are required to open the subscription. These credentials are not stored in ENS. The functionality of ENS2 also depends on the Apple Push Notification Service (APNS) to deliver silent notifications to the device.

The following list describes the dependencies of the ENS2 on the EWS and APNS.

  • If the Boxer application is active and receives a silent notification, the Boxer application attempts to resubscribe. When the Boxer application receives a silent notification, the Boxer sends a resubscription request to the EWS using the employee credentials.
  • The iOS can stop the Boxer process without any warning due to various reasons. In such scenarios, the end users might see Boxer in the App Scroll of an iOS device, however, the Boxer process is stopped. The Boxer application has no control over this process and this state is called a killed state. If the Boxer application is in a killed state when it receives a silent notification, the Boxer application cannot resubscribe due to which the user can experience loss of notifications until the user opens the Boxer application. Opening the Boxer application triggers the ENS subscription again, and the user starts receiving notifications.
  • The end user might experience an inaccurate badge count when the time subscription is lost and before the Boxer application resubscribes.

The following list describes the badge count accuracy limitations on the Boxer application:

  • Sync window - The ENS checks the Inbox folder without the sync period and the Boxer unread messages are within the sync period. So, the users might have unread messages outside the sync window in the Inbox folder. The ENS reports these messages as unread while the user might not see these unread email messages in the Inbox.
  • Boxer application dependency on resubscription - When the ENS is going through resubscription, the ENS does not receive any notification or badge count. During this period, the ENS does not have the updated badge count.
  • Unmanaged accounts - When the user has both managed and unmanaged accounts like the Exchange account and Gmail account, the badge counts are not handled correctly.
  • Comparison with Outlook on MAC devices - The Outlook on MAC devices shows certain emails as read whereas the same emails show unread when opened using Boxer or Outlook for Web Access (OWA). So, the badge count is incorrect when compared with Outlook on MAC devices.

Troubleshooting Accessibility Issues to the ENS Server from a Cloud ENS

Probem: Check if the cloud ENS is accessible from the ENS server and confirm if the ENS server is accessible from the CAS or the Mailbox server.

  1. Access the following URL in a browser on all CAS or mailbox servers:

    https://{ENS cloud URL}/api/ens/alive.

  2. Select the ENS cloud URL based on your region.
    Region ENS Cloud URL
    North America https://ens.getboxer.com/api/ens
    Asia Pacific https://ens-apj.getboxer.com/api/ens
    European Union (EU) https://ens-eu.getboxer.com/api/ens
    United Kingdom https://ens-uk.getboxer.com/api/ens

Results:

You must receive the following response:

If you are unable to see a similar response, then allowlist the IP addresses and endpoints and validate the connection to the ENS server. To see the supported ENS2 API endpoints and to receive status updates on ENS2, refer the ENS Endpoints and IP Allowlist and Subscribe to ENS2 Cloud System Status section in the Configure Email Notification Service for Cloud Deployment topic.

Troubleshooting Accessibility Issues to the ENS Server from an On-Premises Installation

Problem: Check if the ENS server is accessible on an on-premises setup and is receiving the request. After an on-premises ENS installation, confirm that the ENS is installed and running on the ENS server.

  1. Navigate to the following URL in a web browser and check the same server where ENS is installed. The user localhost is mentioned as follows: https://localhost/MailNotificationService/api/ens/alive. To check from outside the ENS server, see http://{ENS server public url}/MailNotificationService/api/ens/alive and https://{ENS server public url/MailNotificationService/api/ens/alive. You must be able to see the following response:

  2. Confirm that a certificate is imported and 443 is bound to the website if you have an issue with the https 443 traffic.

Result:

Confirm if the ENS is receiving the request from outside (for example, receiving the request from a browser when you reach the alive endpoint). When verifying the ENS alive endpoint, the IIS logs are generated. The IIS logs are by default stored at the following path: %SystemDrive%\inetpub\logs\LogFiles. If you do not find the logs at the default path, then the logs for your IIS might be stored at a different location. To get the path for the IIS logs, check the following link: Managing IIS Log File Storage.

For other successful ENS traffic, you might see the following log entries in the IIS logs.

Test the Exchange Web Services URL

The Exchange Web Services (EWS) subscriptions notify changes in a users' mailbox. Use the Microsoft's Remote Connectivity Analyzer online tool to test the EWS URL. You can test the EWS URL only if the EWS is configured for the basic authentication and the EWS is publicly available.

  1. Open the Microsoft's Remote Connectivity Analyzer.
  2. Select the Synchronization, Notification, Availability, Automatic Replies under the Microsoft Exchange Web Services Connectivity Tests and click Next.
  3. Enter the Email address, Domain\User Name (or UPN), Password, and Confirm Password information.
  4. Enter the EWS URL manually, if the autodiscovery is not enabled or select the Use Auto-Discovery to detect server settings if autodiscovery is enabled.
  5. Click Verify account and perform the test.

Results:

If there are no issues, the following success message appears:

If the connectivity test fails for the following reasons, then expand the error to see more information.

You see the following 401 error when the user is unauthorized.

You see the following error when the autodiscovery is not enabled.

You see the following error when the Remote server cannot be resolved.

Troubleshooting the EWS Accessibility on an On-Premises ENS Installation

Use the EWSEditor tool to check if the EWS is internal and accessible from an on-premises ENS. The EWSEditor tool works only if you are using basic authentication or Open Authorization (OAuth).

  1. Download and extract the EWSEditor ZIP file from the EWSEditor.
  2. Run the EWSEditor.exe file.
  3. Navigate to the File > New Exchange Service and enter the Service URL, User Name, Password, and Domain.
  4. Click OK. If there is an error in the details entered, then an appropriate error message appears. If the details entered are correct, then the following message appears:
  5. Click Yes.
  6. Select the device for which you want to check the subscription and right-click on the device. Select Open Streaming Notifications Viewer.
  7. Click Subscribe and Clear Events.
  8. To test the notifications, send a test message to the device. If the test is successful, the following screen appears:

Troubleshooting ENS2 Configuration Issues in Workspace ONE UEM Console

For Workspace ONE UEM Console 2003 or lower

You can configure the ENS2 settings using the configuration key and configuration value provided by the Workspace ONE UEM console.

The following image shows the ENS2 settings when configured without EWS URL and with the EWS URL.

ENS2 settings configured without the EWS URL

ENS2 settings configured with the EWS URL

The following table lists the Workspace ONE UEM console configuration keys and values for ENS2.

Configuration Key Value Type Configuration Value
ENSLinkAddress String

Specify the URL address of the ENS server.

  • For cloud deployments, the URL must be https://ens.getboxer.com/api/ens. Based on your region, VMware provides the resolved name or IP address .
  • For on-premises deployments, the URL must be https://mycompany.com/MailNotificationService/api/ens, where mycompany.com is the IP address or domain name for your ENS server.
ENSAPIToken String VMware provides the API token to enable the ENS service. For the on-premises installation, the on-premises installer creates this token.
AccountNotifyPush Boolean This value must be True.
EWSUrl String Enables manual configuration of the Exchange Web Services (EWS) endpoint when the autodiscovery is disabled in your Exchange environment. Even for deployments where the autodiscovery is enabled, you must prefer to configure this option. The value of this option is your EWS endpoint. For example,

https://outlook.office365.com/EWS/Exchange.asmx (for Office 365)

For Workspace ONE UEM console version 2004, 2005, 2006, and 2007

You can configure the ENS2 settings and the EWS URL in Workspace ONE UEM console. However, configuring the EWS URL is not mandatory for ENS but it is recommended you configure the EWS URL.

You can configure the ENS2 specific settings and the EWS URL in the Email Settings > Notification section of the Boxer app assignment page. For more information on the ENS2 specific settings, see the Assign and Configure Workspace ONE Boxer Using the App Assignment Page section in the Workspace ONE Boxer Admin Guide and see the ENS2 Application Configuration for Boxer topic for more information on the ENS2 configuration for Boxer.

For Workspace ONE UEM console version 2008 and later

You can configure the ENS2 settings and the EWS URL in Workspace ONE UEM console. However, configuring the EWS URL is not mandatory for ENS but it is recommended you configure the EWS URL. The location where you set the ENS2 specific settings and the EWS URL are different.

You can configure the EWS URL in the Email Settings page and the ENS2 specific settings in the Email Settings > Notification page of the Boxer app assignment page. For more information, see the Assign and Configure Workspace ONE Boxer Using the App Assignment Page section in the Workspace ONE Boxer Admin Guide and see the ENS2 Application Configuration for Boxer topic for more information on the ENS2 configuration for Boxer.

Troubleshooting ENS2 Notification Issues

ENS notifications are applicable only for emails in the Inbox folder. The notifications do not work for Calendar events, sub folders, and so on. The following topics describes the steps to troubleshoot the ENS2 notification issues for emails in the Inbox folder.

Public Key Request from the ENS

The Boxer application requests the public key from the ENS. The public key is used to encrypt the user credentials. When the ENS processes the request, the ENS sends the public key and creates a user record in the database against the user ID. In the following sample, the ENS logs for the GetPublicKeyRequest, the Boxer application sends the SHA256 hash of the email address as the user ID.

2019/10/18 05:54:05.395 WIN-HTCPEDXIUVF 7b21cd56-4c45-4a7c-88d9-a7f225cea3b9    [0000000-0000000]   (5)     Debug   MailNotificationService.Controllers.EnsController.GetPublicKey  User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  Processing Get Public key request for Userid[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] 
2019/10/18 05:54:05.457 WIN-HTCPEDXIUVF 7b21cd56-4c45-4a7c-88d9-a7f225cea3b9    [0000000-0000000]   (5)     Debug   MailNotificationService.BusinessImpl.GetPublicKeyBusiness.ProcessGetPublicKeyRequestAsync   User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  Key generated for user id [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]   
2019/10/18 05:54:05.457 WIN-HTCPEDXIUVF 7b21cd56-4c45-4a7c-88d9-a7f225cea3b9    [0000000-0000000]   (5)     Debug   MailNotificationService.Controllers.EnsController.GetPublicKey  User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  Get Public Key request processed. HttpStatusCode:[OK] ResponseCode:[UpdateSuccess] 

The possible error types and solutions that you might see during a GetPublicKeyRequest is listed as follows:

Error: Unauthorized Request

If you see the following error when you send a GetPublicKeyRequest, then ensure that the provided API token is correct. Verify if the API token is the same at the following instances:

  • API token in the ENS logs - API token : [12341*********fasdf]
  • The Boxer application configuration in the UEM console. See, the Workspace ONE Boxer Admin Guide for more information on the Boxer application configuration values.
  • API token in the Boxer application logs - Verify the API token in the Boxer application logs.

Error: Unable to add a NULL value into the PublicKey column

Note: This section is applicable for an on-premises installation only.

When the available RSA keys in the database are exhausted, you might see the following error. This issue is automatically fixed when the RSAKey tracker service triggers and generates new keys again.

2019/10/18 12:20:04.121 WIN-HTCPEDXIUVF b4a42dc8-6896-4243-9a4c-8ed476ae94ab    [0000000-0000000]   (5)     Debug   MailNotificationService.Controllers.EnsController.GetPublicKey  User Id [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  Processing Get Public key request for Userid[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] 
2019/10/18 12:20:04.136 WIN-HTCPEDXIUVF b4a42dc8-6896-4243-9a4c-8ed476ae94ab    [0000000-0000000]   (5)     Debug   MailNotificationService.BusinessImpl.GetPublicKeyBusiness.ProcessGetPublicKeyRequestAsync   User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  Error: 515, Severity: 16, State: 2, Message: "Cannot insert the value NULL into column 'PublicKey', table 'onpremensdev.dbo.UserInfo'; column does not allow nulls. INSERT fails.", Procedure: "UserInfo_Save", Line: 39   
2019/10/18 12:20:04.136 WIN-HTCPEDXIUVF b4a42dc8-6896-4243-9a4c-8ed476ae94ab    [0000000-0000000]   (5)     Debug   MailNotificationService.Controllers.EnsController.GetPublicKey  User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  Get Public Key request processed. HttpStatusCode:[InternalServerError] ResponseCode:[UpdateFail]
Note: The RSAKey tracker trigger interval time is 120 minutes. If the number of keys available in the database during the tracker trigger time is less than 250, then the RSAKey tracker starts generating a new batch of RSA keys. By default, the RSAKey tracker generates 500 new keys at a time.

Ensure that the following values are present in the RSAKey tracker configuration file:

<add key="numberOfKeysToBeInserted" value="500"/>
<add key="wakeUpIntervalInMins" value="120"/>
<add key="keysThreshold" value="250"/>

Error: ENS service communication

When communicating with the ENS service, if you see the following error in the Boxer application logs, then ensure that your device has proper connectivity.

2019-11-04T12:23:12Z  E [710337] [ENS] An error occurred when communicating with the ENS service: Error Domain=NSURLErrorDomain Code=-1009 "The Internet connection appears to be offline." UserInfo={NSUnderlyingError=0x281fead90 {Error Domain=kCFErrorDomainCFNetwork Code=-1009 "The Internet connection appears to be offline." UserInfo={NSErrorFailingURLStringKey=https://ens-staging.getboxer.com/api/ens/getpublickey, NSErrorFailingURLKey=https://ens-staging.getboxer.com/api/ens/getpublickey, _kCFStreamErrorCodeKey=50, _kCFStreamErrorDomainKey=1, NSLocalizedDescription=The Internet connection appears to be offline.}}, NSErrorFailingURLStringKey=https://ens-staging.getboxer.com/api/ens/getpublickey, NSErrorFailingURLKey=https://ens-staging.getboxer.com/api/ens/getpublickey, _kCFStreamErrorDomainKey=1, _kCFStreamErrorCodeKey=50, NSLocalizedDescription=The Internet connection appears to be offline.} at URL: https://ens-staging.getboxer.com/api/ens/getpublickey. Data: . Response Code: 0
2019-11-04T12:23:12Z  E [710337] [ENS] Error registering new account: vmwUser4@awRed.onmicrosoft.com
Error:Error Domain=com.alamofire.serialization.response.error.response Code=-1 "invalid public key" UserInfo={NSLocalizedDescription=invalid public key}
2019-11-04T12:23:12Z  E [703318] [ENS] Error registering device for push notification
Error:Error Domain=com.alamofire.serialization.response.error.response Code=-1 "invalid public key" UserInfo={NSLocalizedDescription=invalid public key}
2019-11-04T12:23:12Z  E [726177] - Unexpected error: {
    BXLocalizedContextMessageErrorKey = "Could not update settings for the push notification service";
    BXLocalizedTitleErrorKey = "Could not update settings for the push notification service";
    NSLocalizedDescription = "Could not update settings for the push notification service. ";
    NSLocalizedFailureReason = "Failed to update push notification settings. Please contact your administrator.";
} context: 1

Register Device Request

The Boxer application sends a Register request to the ENS, a push subscription to the EWS, and a subscribe for notification. If the GetPublicKey request is successful, then the Boxer application sends a register request to the ENS with the necessary information required to register a device for notification.

Scenario 1: - If the EWS URL is not configured in the console, then the ENS tries autodiscovery to obtain the EWS URL to subscribe the user.

Scenario 2: - If the EWS URL is configured in the console, then the ENS uses the same EWS URL to subscribe the user.

When the subscription is successful, the ENS receives the [UserSubscribed] message with the subscription ID as mentioned in the following code snippet.

2019/11/05 08:18:49.674 A3  726c4072-5144-4450-848b-821f65174b89    [0000000-0000000]   (23)    Info    MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotificationsAsync  User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  User [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] subscribed with subscriptionId [JwBtbjJwcjE5bWIzMDA1Lm5hbXByZDE5LnByb2Qub3V0bG9vay5jb20QAAAAJ6RYazaIoUCfX7KheUsQYUQnw9rIYdcIEAAAAAQ9tcFCKSZFrTOxLbSCwj4=]
2019/11/05 08:18:49.767 A3  726c4072-5144-4450-848b-821f65174b89    [0000000-0000000]   (28)    Debug   MailNotificationService.Controllers.EnsController.RegisterDeviceV2  User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  Register device request processed. HttpStatusCode:[OK] ResponseCode:[UserSubscribed]

In the Android Boxer logs, you must see the following log entries to confirm a successful registration:

--------------------------------------------------------------
    ENS SETTINGS
--------------------------------------------------------------
ENS_LINK_ADDRESS = https://ens.getboxer.com/api/ens
ENS_API_TOKEN = 17413**********************88c08
POLICY_ACCOUNT_NOTIFY_PUSH = true
EWS_URL =
ENS_STATE = (8 -> Registered)
 
--------------------------------------------------------------
    HEALTH STATUS
--------------------------------------------------------------
App version health status: Green, Current app version: 5.11.0.4, New version: 5.10.0
Sync Health Status: Green, Sync durations in seconds: [0.522, 0.49, 0.416, 0.379, 0.424, 0.368, 0.465, 0.496, 0.565, 1.344], Sync results [OK, OK, OK, OK, OK, OK, OK, OK, OK, OK]
Ens health status: Green , Ens state: Registered
Overall health status: Green
Ens registration for account (id=8) is successful!

For the iOS Boxer logs, you must see the following log entries to confirm a successful registration:

For normal subscription
2019-11-11T09:31:41Z  I [12347] [ENS] Successfully registered account.
Note: For iOS Boxer logs, open the Boxer application, navigate to the Boxer Settings, click the VMware Secure Email, and ensure the Use Push Service switch is enabled to confirm a successful ENS registration.

The possible errors and solutions that you might see when you are unable to locate the autodiscover services are listed as follows:

Error: Unable to Locate the Autodiscover Services

If you see the following error, then ensure to enable autodiscovery, check the availability and connectivity of the autodiscovery server using the EWSEditor and the MS remote connectivity analyzer.

2019/11/06 07:01:56.207 A3  d252be19-1c5d-4e30-9155-a0ae3a529679    [0000000-0000000]   (94)    Warn    MailNotificationService.BusinessImpl.SubscriptionBusiness.SubscribeV2Async  User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  Exception while auto discovery occured for userId [20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586], Exception Message [The Autodiscover service couldn't be located.] , Exception [Microsoft.Exchange.WebServices.Data.AutodiscoverLocalException: The Autodiscover service couldn't be located.
   at Microsoft.Exchange.WebServices.Autodiscover.AutodiscoverService.InternalGetLegacyUserSettings[TSettings](String emailAddress, List`1 redirectionEmailAddresses, Int32& currentHop)
   at Microsoft.Exchange.WebServices.Autodiscover.AutodiscoverService.GetLegacyUserSettings[TSettings](String emailAddress)
   at Microsoft.Exchange.WebServices.Autodiscover.AutodiscoverService.InternalGetLegacyUserSettings(String emailAddress, List`1 requestedSettings)
   at Microsoft.Exchange.WebServices.Data.ExchangeService.GetAutodiscoverUrl(String emailAddress, ExchangeVersion requestedServerVersion, AutodiscoverRedirectionUrlValidationCallback validateRedirectionUrlCallback)
   at Microsoft.Exchange.WebServices.Data.ExchangeService.AutodiscoverUrl(String emailAddress, AutodiscoverRedirectionUrlValidationCallback validateRedirectionUrlCallback)
   at MailNotificationService.BusinessImpl.ExchangeServiceBusiness.<GetExchangeServiceViaAutoDiscovery>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MailNotificationService.BusinessImpl.ExchangeServiceBusiness.<GetExchangeServiceAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MailNotificationService.BusinessImpl.SubscriptionBusiness.<SubscribeV2Async>d__7.MoveNext()], Inner Exception [], Autodiscover url used [The Autodiscover service couldn't be located.]
2019/11/06 07:01:56.207 A3  d252be19-1c5d-4e30-9155-a0ae3a529679    [0000000-0000000]   (94)    Debug   MailNotificationService.Controllers.EnsController.RegisterDeviceV2  User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  Register device request processed. HttpStatusCode:[Conflict] ResponseCode:[SubscribeAgain]

Error: The remote server returned an error (403) Forbidden

If this error occurs during a subscription, then ensure to enter the proper EWS URL in the Boxer application KVP values of the UEM console. The EWSUrl used to subscribe must have the complete endpoint specified.

Example of a correct EWSUrl - [https://mail-mem13.xyz.com/EWS/exchange.asmx]

Example of an incorrect EWSUrl - [https://mail-xyz.com/]

To check the EWS URL availability and connectivity, check the EWSEditor and the MS remote connectivity analyzer.

2019/11/06 07:09:54.064 A3  f43eb3d0-e173-49de-9b52-3acb8a1107c4    [0000000-0000000]   (98)    Debug   MailNotificationService.Controllers.EnsController.RegisterDeviceV2  User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  Processing register device request for Userid[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]
2019/11/06 07:09:54.080 A3  f43eb3d0-e173-49de-9b52-3acb8a1107c4    [0000000-0000000]   (98)    Debug   MailNotificationService.BusinessImpl.RegisterDeviceBusiness.ProcessRegisterDeviceRequestAsyncV2 User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  Exchange version sent by boxer [2] 
2019/11/06 07:09:54.080 A3  f43eb3d0-e173-49de-9b52-3acb8a1107c4    [0000000-0000000]   (98)    Debug   MailNotificationService.BusinessImpl.ExchangeServiceBusiness.GetExchangeServiceAsync    User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  Using client ewsurl, mailServerUrlMatched : False, deletedEWSUrl: False
2019/11/06 07:09:54.080 A3  f43eb3d0-e173-49de-9b52-3acb8a1107c4    [0000000-0000000]   (98)    Debug   MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotificationsAsync  User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  EWSUrl used to subscribe: [https://mail-mem13.ssdevrd.com/]
2019/11/06 07:09:54.080 A3  f43eb3d0-e173-49de-9b52-3acb8a1107c4    [0000000-0000000]   (98)    Debug   MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotificationsAsync  User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  User subscribing with [Basic Auth] 
2019/11/06 07:09:54.173 A3  f43eb3d0-e173-49de-9b52-3acb8a1107c4    [0000000-0000000]   (98)    Warn    MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotifications   User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  Service request exception occured  for userId [20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586], Inner exception message [The remote server returned an error: (403) Forbidden.] Going for a retry,   
2019/11/06 07:09:54.173 A3  f43eb3d0-e173-49de-9b52-3acb8a1107c4    [0000000-0000000]   (98)    Debug   MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotificationsAsync  User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  EWSUrl used to subscribe: [https://mail-mem13.ssdevrd.com/]
2019/11/06 07:09:54.173 A3  f43eb3d0-e173-49de-9b52-3acb8a1107c4    [0000000-0000000]   (98)    Debug   MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotificationsAsync  User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  User subscribing with [Basic Auth] 
2019/11/06 07:09:54.205 A3  f43eb3d0-e173-49de-9b52-3acb8a1107c4    [0000000-0000000]   (98)    Warn    MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotifications   User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  Service request exception occured  for userId [20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586], Inner exception message [The remote server returned an error: (403) Forbidden.] Going for a retry,   
2019/11/06 07:09:54.205 A3  f43eb3d0-e173-49de-9b52-3acb8a1107c4    [0000000-0000000]   (98)    Debug   MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotificationsAsync  User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  EWSUrl used to subscribe: [https://mail-mem13.ssdevrd.com/]
2019/11/06 07:09:54.205 A3  f43eb3d0-e173-49de-9b52-3acb8a1107c4    [0000000-0000000]   (98)    Debug   MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotificationsAsync  User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  User subscribing with [Basic Auth] 
2019/11/06 07:09:54.236 A3  f43eb3d0-e173-49de-9b52-3acb8a1107c4    [0000000-0000000]   (98)    Warn    MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotifications   User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  Service request exception occured  for userId [20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586], Inner exception message [The remote server returned an error: (403) Forbidden.] Going for a retry,   
2019/11/06 07:09:54.236 A3  f43eb3d0-e173-49de-9b52-3acb8a1107c4    [0000000-0000000]   (98)    Debug   MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotificationsAsync  User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  EWSUrl used to subscribe: [https://mail-mem13.ssdevrd.com/]
2019/11/06 07:09:54.236 A3  f43eb3d0-e173-49de-9b52-3acb8a1107c4    [0000000-0000000]   (98)    Debug   MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotificationsAsync  User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  User subscribing with [Basic Auth] 
2019/11/06 07:09:54.251 A3  f43eb3d0-e173-49de-9b52-3acb8a1107c4    [0000000-0000000]   (98)    Warn    MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotifications   User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  Service request exception occured  for userId [20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586], Inner exception message [The remote server returned an error: (403) Forbidden.] Going for a retry,   
2019/11/06 07:09:54.251 A3  f43eb3d0-e173-49de-9b52-3acb8a1107c4    [0000000-0000000]   (98)    Warn    MailNotificationService.BusinessImpl.SubscriptionBusiness.SubscribeV2Async  User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  Service request exception occured  for userId [20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586], Inner exception message [The remote server returned an error: (403) Forbidden.] Going for a retry,   
2019/11/06 07:09:54.251 A3  f43eb3d0-e173-49de-9b52-3acb8a1107c4    [0000000-0000000]   (98)    Debug   MailNotificationService.Controllers.EnsController.RegisterDeviceV2  User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  Register device request processed. HttpStatusCode:[Conflict] ResponseCode:[SubscribeAgain] 

Sample error logs of Boxer during registration:

2019-11-11T09:13:43Z  E [9326] [ENS] An error occurred when communicating with the ENS service: Error Domain=com.alamofire.error.serialization.response Code=-1011 "Request failed: conflict (409)" UserInfo={NSLocalizedDescription=Request failed: conflict (409), NSErrorFailingURLKey=https://a3.ssdevrd.com/mailnotificationservice/api/ens/registerdevicev2, com.alamofire.serialization.response.error.data={length = 135, bytes = 0x7b227265 73706f6e 7365436f 6465223a ... 4f6e5072 656d227d }, com.alamofire.serialization.response.error.response=<NSHTTPURLResponse: 0x282db1fa0> { URL: https://a3.ssdevrd.com/mailnotificationservice/api/ens/registerdevicev2 } { Status Code: 409, Headers {
    "Content-Length" =     (
        135
    );
    "Content-Type" =     (
        "application/json; charset=utf-8"
    );
    Date =     (
        "Mon, 11 Nov 2019 09:13:40 GMT"
    );
    Server =     (
        "Microsoft-IIS/8.5"
    );
    "X-Powered-By" =     (
        "ASP.NET"
    );
} }} at URL: https://a3.ssdevrd.com/mailnotificationservice/api/ens/registerdevicev2. Data: {"responseCode":14,"errorMessage":"The Autodiscover service couldn't be located.","version":"1.5.7235.6268","environmentType":"OnPrem"}. Response Code: 409
2019-11-11T09:13:43Z  E [9326] [ENS] registerAccountOnENS: Error updating settings or credentials
Error:Error Domain=com.alamofire.error.serialization.response Code=-1011 "Request failed: conflict (409)" UserInfo={NSLocalizedDescription=Request failed: conflict (409), NSErrorFailingURLKey=https://a3.ssdevrd.com/mailnotificationservice/api/ens/registerdevicev2, com.alamofire.serialization.response.error.data={length = 135, bytes = 0x7b227265 73706f6e 7365436f 6465223a ... 4f6e5072 656d227d }, com.alamofire.serialization.response.error.response=<NSHTTPURLResponse: 0x282db1fa0> { URL: https://a3.ssdevrd.com/mailnotificationservice/api/ens/registerdevicev2 } { Status Code: 409, Headers {
    "Content-Length" =     (
        135
    );
    "Content-Type" =     (
        "application/json; charset=utf-8"
    );
    Date =     (
        "Mon, 11 Nov 2019 09:13:40 GMT"
    );
    Server =     (
        "Microsoft-IIS/8.5"
    );
    "X-Powered-By" =     (
        "ASP.NET"
    );
} }}
2019-11-11T09:13:43Z  E [9365] - Unexpected error: {
    BXLocalizedContextMessageErrorKey = "Could not update settings for the push notification service";
    BXLocalizedTitleErrorKey = "Could not update settings for the push notification service";
    NSLocalizedDescription = "Could not update settings for the push notification service. ";
    NSLocalizedFailureReason = "Failed to update push notification settings. Please contact your administrator.";
}
In the sample error logs of Boxer, you can see the following message:
{"responseCode":14,"errorMessage":"The Autodiscover service couldn't be
        located...

In this case, ensure that the autodiscovery URL is reachable from the ENS and the autodiscovery URL is configured correctly using the EWSEditor tool or MS connectivity analyzer tool.

If you are using the EWSUrl, ensure that the EWSUrl key is configured in the console with a correct value for the EWSUrl of their respective Exchange environments. To verify the EWSUrl is correct, open a browser, enter the EWSUrl, and ensure that you are prompted to enter the credentials.

You can find the error message and response code for different reasons. Based on the error message, you can start troubleshooting the issue.

Error: 403 or 401 error message

EWS must be accessible to the ENS application to subscribe the user for notification. If the EWS is not configured correctly, then you might receive 403 or 401 error. In such cases, refer the following documents:

Check the type of authentication you have enabled in the EWS. Ensure that the authentication is in parity with what the customer is using for ActiveSync (Basic, OAuth, and CBA). The Boxer application sends the user credentials to the ENS and the ENS uses the same credentials and the same type of authentication to communicate with the EWS.

Note: If the ENS can access the Office 365 and the Active Directory Federation Services (ADFS), then ensure that either the ENS IPs are allowlisted on the ADFS or the affected user has no block claim on the ADFS.

If you are using Office 365 and you receive a 401 error from the EWS URL, the reason for the error might be because the client access rules or ADFS claims are configured. In such scenarios, refer the following documents.

In a scenario where the ENS on-premises Exchange with CBA is enabled, you might need to confirm that the client certificate is arriving at the Exchange endpoint. To troubleshoot any errors, see the Troubleshooting ENS with On-Premise Exchange Server section.

Force Register or Re-register on the Boxer application:

On iOS devices only, you can manually perform a force subscription, in the following cases:

  • If there are any changes to the keys in console, then you must approximately wait for 1 hour and check if the users are still receiving the notification. If the users are not receiving notifications, you can proceed to re-register the Boxer application with the ENS2 service.
  • If you do not see any register request in the ENS logs from the Boxer application, then assume that the Boxer application has failed to send the register request automatically. Therefore, the ENS tries to re-register the Boxer application with the ENS2 service forcefully.

To force register or re-register on the Boxer application, perform the following steps:

  1. Open the Boxer application and click Settings.
  2. Under the Accounts tab, select your ENS-specific account.
  3. Turn off the Use Push Service option.
  4. Navigate to the Boxer application Settings screen.
  5. Repeat Step 2 through Step 4 to turn on the Use Push Service option.

When you perform either of the steps mentioned, then you can see the force register request in the ENS logs.

To confirm the force subscription in the ENS logs, search for the ForceSubscription and you must be able to see the following value: ForceSubscription : [True].

Registration Status Events

If the registration is successful, then the Exchange sends a status event to the ENS periodically against each subscription ID, to confirm the subscription. The ENS then sends an acknowledgment for each of the subscription IDs back to the Exchange.

2019/11/05 08:57:31.413 A3  1eb9186b-9370-45de-a172-0e452586f398    [0000000-0000000]   (58)    Debug   MailNotificationService.BusinessImpl.ExchangeNotificationParser.ScanEventNotificationAsync  User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  Received [StatusEvent] for subscription: [JwBtbjJwcjE5bWIzMDA1Lm5hbXByZDE5LnByb2Qub3V0bG9vay5jb20QAAAAl4H5dKboFUm1kJ8ZNBKkJILRTBjMYdcIEAAAAAQ9tcFCKSZFrTOxLbSCwj4=]
2019/11/05 08:57:31.413 A3  1eb9186b-9370-45de-a172-0e452586f398    [0000000-0000000]   (58)    Debug   MailNotificationService.BusinessImpl.PushNotificationBusiness.HandleExchangeEvents  User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  Status event received for user: [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]
If the ENS receives the status event for the old subscription ID, then the ENS responds to the Exchange with an unsubscribe response as shown in the following logs.
2019/11/05 08:49:20.123 A3  d2adec8a-73d7-48f2-ba14-abbd917844cd    [0000000-0000000]   (54)    Info    MailNotificationService.BusinessImpl.ExchangeNotificationParser.ScanEventNotificationAsync  User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  This JwBtbjJwcjE5bWIzMDA1Lm5hbXByZDE5LnByb2Qub3V0bG9vay5jb20QAAAAJ6RYazaIoUCfX7KheUsQYUQnw9rIYdcIEAAAAAQ9tcFCKSZFrTOxLbSCwj4= is old subscription for user 1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d, sending unsubscribe response  
2019/11/05 08:49:20.123 A3  d2adec8a-73d7-48f2-ba14-abbd917844cd    [0000000-0000000]   (54)    Debug   MailNotificationService.BusinessImpl.ExchangeNotificationParser.ScanEventNotificationAsync  User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  Sent Unsubscribe response to EWS successfully for subscriptionId: [JwBtbjJwcjE5bWIzMDA1Lm5hbXByZDE5LnByb2Qub3V0bG9vay5jb20QAAAAJ6RYazaIoUCfX7KheUsQYUQnw9rIYdcIEAAAAAQ9tcFCKSZFrTOxLbSCwj4=]   
2019/11/05 08:49:20.123 A3  d2adec8a-73d7-48f2-ba14-abbd917844cd    [0000000-0000000]   (54)    Debug   MailNotificationService.BusinessImpl.PushNotificationBusiness.ProcessPushNotificationV2Async    User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  ProcessNotificationResponse.IsUnSubscribeResponse is true

For more information on the status frequency, see the StatusFrequency topic.

ENS must receive the status events from the Exchange immediately after a subscription is successful. If the ENS is not receiving the status events, then check the following troubleshooting methods to verify the communication between the Exchange server and the ENS.

Error: Status event not received

If you do not see any status events in the ENS logs after a successful subscription, then check the communication between the Exchange server and the ENS. Access the following URLs in the browser on the CAS or the mailbox servers to check the communication between the Exchange and the ENS.

  • For on-premises ENS deployments, use the https://{ENS URL}/MailNotificationService/api/ens/alive.
  • For cloud ENS deployments, use the https://{ENS URL}/api/ens/alive. For example, https://ens.getboxer.com/api/ens/alive. Select the ENS cloud URL based on your region.

You must be able to see the following result when you browse the specified URLs from the browser.

  • For on-premises ENS deployments, use the https://{ENS URL}/MailNotificationService/api/ens/pushnotificationlistener.
  • For cloud ENS deployments, use the https://{ENS URL}/api/ens/pushnotificationlistener. Select the ENS cloud URL based on your region.
Note: When browsing the URLs, if you see any SSL error, then proceed to import the ENS certificate in the MMC of the server.

New Mail Event and Fetch Mail

When a device is successfully registered and the communication between the ENS and the Exchange is working correctly, the Exchange starts sending new mail events to the ENS whenever a new mail is received on the subscribed user mailbox. If the payloads of the created events contain an unread count, then the ENS uses the unread count, else the ENS gets the unread count from the EWS.

2019/11/05 09:39:56.608 A3  9f08ed6d-0726-430c-8440-9c396443c7ca    [0000000-0000000]   (74)    Debug   MailNotificationService.BusinessImpl.ExchangeNotificationParser.ScanEventNotificationAsync  User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  Received [CreatedEvent] for subscription: [JwBtbjJwcjE5bWIzMDA1Lm5hbXByZDE5LnByb2Qub3V0bG9vay5jb20QAAAAl4H5dKboFUm1kJ8ZNBKkJILRTBjMYdcIEAAAAAQ9tcFCKSZFrTOxLbSCwj4=]   
2019/11/05 09:39:56.639 A3  9f08ed6d-0726-430c-8440-9c396443c7ca    [0000000-0000000]   (74)    Debug   MailNotificationService.BusinessImpl.UnreadCountExchangeBusiness.GetUnReadCountV2   User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  EWSUrl used to get unread count: [https://outlook.office365.com/EWS/Exchange.asmx] 
2019/11/05 09:39:56.889 A3  9f08ed6d-0726-430c-8440-9c396443c7ca    [0000000-0000000]   (74)    Info    MailNotificationService.BusinessImpl.PushNotificationBusiness.HandleNewMailEvent    User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  Received new mail event for user [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] with BADGE count [893]

Whenever the ENS receives a new mail event, the ENS fetches the mail information from the Exchange. The possible errors and solutions that you might see during a fetch mail request is listed as follows:

Error: Stuck in EWSUrl used to sync email: [https://outlook.office365.com/EWS/Exchange.asmx] steps

When a mail event is received from the Exchange, the ENS tries to fetch all the information from the mail. If you are unable to see any ENS logs such as the Fetched email, then check the respective EWS logs in the Exchange. You can obtain the corresponding EWS logs using the client request ID or the activity ID.

Fetch New Mail Request

Sample client request ID or the activity ID: 03ea7f36-f72f-4322-8413-0dcd81c4ac78

Note: You can get the client request ID or the activity ID in the third column of the ENS logs. Copy that ID and search for the client request ID or the activity ID in the EWS logs.

ENS sends a push notification request to the CNS or the SNS

When the new mail information is fetched from the Exchange, the ENS composes and sends a notification payload to the CNS (for on-premises) or the SNS (for cloud).

Sample of sending a notification payload to the CNS (for on-premises)

2019/11/05 09:48:42.675 A3  fedf9a1d-6cc8-4607-acad-ae006766292a    [0000000-0000000]   (82)    Info    MailNotificationService.BusinessImpl.NotificationsProcessor.AddNotificationToBatch  User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  About to Post Notification for user : [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] and Device Id : [1]   
2019/11/05 09:48:42.690 A3  fedf9a1d-6cc8-4607-acad-ae006766292a    [0000000-0000000]   (82)    Info    MailNotificationService.BusinessImpl.NotificationsProcessor.AddNotificationToBatch  User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  About to Post Notification for user : [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] and Device Id : [5]   
2019/11/05 09:48:47.699 A3  7e45c693-511b-4c19-ae7c-305e5f8f9f0e    [0000000-0000000]   (8)     Info    MailNotificationService.BusinessImpl.CNSHelper.ComposeAPNSPushNotification  User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  Total unread count retrieved [894] for user [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] 
2019/11/05 09:48:47.699 A3  7e45c693-511b-4c19-ae7c-305e5f8f9f0e    [0000000-0000000]   (8)     Debug   MailNotificationService.BusinessImpl.CNSHelper.ComposeAPNSPushNotification  User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  Sending to :: User : [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d], DeviceId : [1], DeviceLogId : [], Message :  messageId: [AAMkAGMxYjUzZDA0LTI5NDItNDUyNi1hZDMzLWIxMmRiNDgyYzIzZQBGAAAAAAAOx2petA5rS4RDQM8RjW1TBwDnjcIsAp4/S4beDDAIaXMhAAAAAAEMAADnjcIsAp4/S4beDDAIaXMhAAGszQatAAA=]
2019/11/05 09:48:47.699 A3  7e45c693-511b-4c19-ae7c-305e5f8f9f0e    [0000000-0000000]   (8)     Info    MailNotificationService.BusinessImpl.CNSHelper.ComposeAPNSPushNotification  User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  Total unread count retrieved [894] for user [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] 
2019/11/05 09:48:47.699 A3  7e45c693-511b-4c19-ae7c-305e5f8f9f0e    [0000000-0000000]   (8)     Debug   MailNotificationService.BusinessImpl.CNSHelper.ComposeAPNSPushNotification  User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  Sending to :: User : [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d], DeviceId : [5], DeviceLogId : [61F9BB13-863C-444C-A300-4F888383ACDD-534-0000000CE599EDE0], Message :  messageId: [AAMkAGMxYjUzZDA0LTI5NDItNDUyNi1hZDMzLWIxMmRiNDgyYzIzZQBGAAAAAAAOx2petA5rS4RDQM8RjW1TBwDnjcIsAp4/S4beDDAIaXMhAAAAAAEMAADnjcIsAp4/S4beDDAIaXMhAAGszQatAAA=]   
2019/11/05 09:48:47.699 A3  7e45c693-511b-4c19-ae7c-305e5f8f9f0e    [0000000-0000000]   (8)     Debug   MailNotificationService.BusinessImpl.CNSHelper.CreateWebRequest User Id:[no-user-id]    CNS Url : [https://cns.awmdm.com/nws/notify/apns]  
2019/11/05 09:48:47.699 A3  7e45c693-511b-4c19-ae7c-305e5f8f9f0e    [0000000-0000000]   (8)     Debug   MailNotificationService.BusinessImpl.CertificateHelper.ComputeCmsSignature  User Id:[no-user-id]    Signing URL [/nws/notify/apns] with Cert [CN=AW Cloud Notification - aTest]
2019/11/05 09:48:48.558 A3  7e45c693-511b-4c19-ae7c-305e5f8f9f0e    [0000000-0000000]   (8)     Debug   MailNotificationService.BusinessImpl.CNSHelper.ReadResponse User Id:[no-user-id]    Response {"status":"success","errorReason":null}   
2019/11/05 09:48:48.558 A3  7e45c693-511b-4c19-ae7c-305e5f8f9f0e    [0000000-0000000]   (8)     Info    MailNotificationService.BusinessImpl.CNSHelper.ReadResponse User Id:[no-user-id]    ResponseCode OK

Sample of sending a notification payload to the SNS (for cloud)

2019-09-06 12:11:51.5380|INFO|MailNotificationService.BusinessImpl.NotificationsProcessor.AddNotificationToBatch|b1d8e164-c3fb-4f67-baa6-002dd3719c4e|User Id:[35045e4062200ca81c92d5b03928a7e86383ef8e9436d512187a711a4b18e94f] About to Post Notification for user [35045e4062200ca81c92d5b03928a7e86383ef8e9436d512187a711a4b18e94f]
 2019-09-06 12:11:52.5537|INFO|MailNotificationService.BusinessImpl.AmazonSNSHelper.PostNotifications|67d3c6f0-a197-4af4-958c-260eeedbf567|User Id:[35045e4062200ca81c92d5b03928a7e86383ef8e9436d512187a711a4b18e94f] Sending notification via SNS
 2019-09-06 12:11:52.5692|INFO|MailNotificationService.BusinessImpl.AmazonSNSHelper.PushNotificationViaSNS|67d3c6f0-a197-4af4-958c-260eeedbf567|User Id:[35045e4062200ca81c92d5b03928a7e86383ef8e9436d512187a711a4b18e94f] Notification successfully sent via SNS for [424716]

To confirm if your Android device is receiving notifications from the ENS, enable the Boxer application passcode and restart the device after a successful registration. You might see a notification, that is, a banner containing the email address configured. On the banner notification if you cannot perform actions such as, Delete, Reply, and Read option then, the notification is a push notification that is sent from the ENS and not locally from the Boxer application itself. If the notification banner contains notification actions such as Delete, Reply, Read, and so on, then the notification is a local notification from the Boxer application and not a push notification from the ENS.

The possible errors and solutions that you might see during a push notification request is listed as follows:

Error: The remote server returned an error: (400) Bad Request

Sample error log:

apis
[0000000-0000000]  (5) Error     MailNotificationService.BusinessImpl.CNSHelper.ReadResponse             User Id:[no-user-id]               Failed To Post to CNS [https://cns.awmdm.com/nws/notify/apns] Error: [The remote server returned an error: (400) Bad Request.] Response: [{"status":"failure","errorReason":"Unable to process json input, errors are Unregistered, requestId 8f8e1939-3660-43d9-b873-a7ae61ea2b7c"}]    ”fcm
[0000000-0000000] (128) Error MailNotificationService.BusinessImpl.CNSHelper.ReadResponse User Id:[no-user-id] Failed To Post to CNS [https://cns.awmdm.com/cns/services/api/notifications/fcm] Error: [The remote server returned an error: (400) Bad Request.] Response: [[{"fcmResults":[{"fcmMessageId":null,"canonicalRegistrationId":null,"errorCode":"NotRegistered"}],"messageId":"AM0PR03MB4067FD025796AB3867E3C5AEEA4F9@AM0PR03MB4067.eurprd03.prod.outlook.com","fcmMulticastId":3844875711768273544,"successCount":0,"failureCount":1,"allFcmCloudError":false,"allMsgsInNonFcmError":true,"fewMsgsInNonFcmError":false,"errCode":5004}]]"

To troubleshoot the issue, share the trace level logs with the VMware Support team. For more information, see the Enable Trace Level Logging for Enhanced Debugging section.

Error: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel

2019/11/06 09:03:48.218 A3  aa57f568-6871-42cc-8b8d-39c77a15af41    [0000000-0000000]   (40)    Error   MailNotificationService.BusinessImpl.CNSHelper.ReadResponse User Id:[no-user-id]    Failed To Post to CNS [https://cns.awmdm.com/nws/notify/apns] Error: [The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.] Response: []
Note: Ensure you have followed the steps as mentioned in the Configure CNS and Download Email Notification Service Configuration Files section in the Configure Email Notification Service for On-Premises Deployment topic.

If the issue still persists, download the latest public CNS certificate from the CNS Public Certificate and perform the following steps:

  1. Click the SSLPinningCertTool shortcut present in the ENS server or click <ENS_INSTALL_DIR>\Email Notification Service\Tools\SSLPinningCertTool\SSLPinningCertTool.exe.
  2. Click the Upload CNS Certificate button.
  3. Select the certificate to be uploaded and click Submit. If the following screen appears, then the certificate is successfully added.
    Note: After uploading the SSL pinning certificate on the ENS, the tool adds the public key of the certificate to the ENS configuration. When the ENS posts payload to the CNS, the certificate validation is done against the newly added certificate public key.
  4. If the following screen appears, the certificate is successfully added to the resubscription configuration file.
    Note: After uploading the SSL pinning certificate, the tool adds the public key of the certificate to the resubscription configuration file. For the resubscription mechanism, after payload (silent notification) to the CNS, the certificate validation is done against the newly added certificate public key.
  5. If the certificate is already present in both the configuration files, then you are prompted with the following message.
    Note: The upload pinning certificate occurs as follows:
    • The tool tries to upload the certificate to the ENS configuration file only if the provided certificate is not present in the ENS configuration file. If the given certificate is already present, then the tool does not prompt any message and continues to upload the same certificate to the resubscription configuration file.
    • The tool tries to add a certificate to the resubscription configuration file only if the provided certificate is not present in the resubscription configuration file. If the given certificate is present, then the tool does not prompt any message to the user.
  6. If the certificate is added to the resubscription configuration file, then navigate to Services and restart the AirWatch Resubscription Mechanism service.

Error: The remote server returned an error: (401) Unauthorized.

Sample error log:

2019/11/06 09:25:13.688 A3  6c041e00-c909-45ff-b340-283844376c06    [0000000-0000000]   (6)     Error   MailNotificationService.BusinessImpl.CNSHelper.ReadResponse User Id:[no-user-id]    Failed To Post to CNS [https://cns.awmdm.com/nws/notify/apns] Error: [The remote server returned an error: (401) Unauthorized.] Response: [{"code":2007,"message":"Unable to verify if the signer cert as trusted. The associated request id is 154e9542-b695-497b-9896-a8fd9cb13e84."}]

If you see a 401 error while posting a notification and the UEM console is on-premises, then navigate to System > Advanced > Secure Channel Certificate and select the Download CNS Secure Channel Certificate Installer. You can also open a Zendesk ticket with the SaasOps > CNS Upload Request category. To install the certificate on the CNS server, send a request to the VMware Support team.

Error: ENS has posted notification to CNS/SNS successfully, but we don't see any notification on the device.

This error occurs due to the APNS or the GCM token issue. To verify the APNS or the GCM tokens, perform the following steps:

  1. Log in to the Workspace ONE UEM console and navigate to the organization group where the device is enrolled.
  2. Navigate to the Devices > List View and select the device.
  3. Click the SEND > PUSH NOTIFICATION and select the application as Boxer from the drop-down.
  4. Enter the Message Body and click SEND. After you click SEND, you must be able to see the notification on the device if the APNS token is correct.

Unregistered ENS Logs

The Boxer application sends an unregister request to the ENS in the following scenarios:

  • When a device account is removed from the Boxer application
  • When a device is deleted from the Workspace ONE UEM console.
  • During an enterprise wipe from the Workspace ONE UEM console.
  • Toggle off the push notification button in the Boxer application settings.

Sample of unregistered ENS logs:

2019/11/06 10:33:23.976 A3  2bd0af6a-ba08-479e-a606-b1326281902c    [0000000-0000000]   (53)    Debug   MailNotificationService.Controllers.EnsController.Unregister    User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  Processing Unregister request. UserId:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]   
2019/11/06 10:33:24.054 A3  2bd0af6a-ba08-479e-a606-b1326281902c    [0000000-0000000]   (55)    Debug   MailNotificationService.BusinessImpl.UnregisterBusiness.ProcessUnregisterRequestAsync   User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  Device Unregistered for user:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]
2019/11/06 10:33:24.054 A3  2bd0af6a-ba08-479e-a606-b1326281902c    [0000000-0000000]   (55)    Debug   MailNotificationService.Controllers.EnsController.Unregister    User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  Unregister request processed. HttpStatusCode:[OK] ResponseCode:[DeviceUnregistered]

When the ENS receives an unregister request, the ENS processes the request and sends an unsubscribe request to the Exchange and deletes the records from the database. The possible errors and solutions that you might see when you unregister is listed as follows:

Error: 401 error Unauthorized

The following logs are seen when the Boxer application sends an unregister request with a wrong API token. You can confirm the API token comparing the API token logged in the ENS logs and present in the Boxer application logs.

ENS logs: API token : [12341*********fasdf]

Boxer application logs: ensapitoken: 17413********************88c08

Sample of UnAuthorizedRequest log:

2019/11/06 10:38:20.413 KAVINASH-W03    cd790dc0-ca7e-4f3d-b468-3c5181c34063    [0000000-0000000]   (31)    Warn    MailNotificationService.BusinessImpl.ApiKeyRepository.ValidateAsync User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  ApiKey header present [True], Value Empty/Null: [False] API key dictionary has keys:[True] Key: [12341:fasdf]  
2019/11/06 10:38:20.424   cd790dc0-ca7e-4f3d-b468-3c5181c34063    [0000000-0000000]   (31)    Debug   MailNotificationService.Controllers.EnsController.Unregister    user Id [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] API token : [12341*********fasdf]   
2019/11/06 10:38:20.444   cd790dc0-ca7e-4f3d-b468-3c5181c34063    [0000000-0000000]   (31)    Warn    MailNotificationService.Controllers.EnsController.Unregister    User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  Error Code:'23' Error message: 'UnAuthorizedRequest'
Stack Trace:   at MailNotificationService.Controllers.EnsController.<Unregister>d__21.MoveNext() in C:\Stash\MailNotificationService\Controllers\EnsController.cs:line 926  

Badge Update for ENS Logs

Badge update is only supported for iOS devices. The badge notification starts displaying only after the badge receives the first notification from ENS. The badge count is not seen in Boxer immediately after the badge counter is configured and subscribed.

Sample of badge update ENS logs:

2019/11/11 12:27:55.416 A3  04f06dcb-a721-4a90-a2ff-2be8e007f533    [0000000-0000000]   (52)    Debug   MailNotificationService.BusinessImpl.ExchangeNotificationParser.ScanEventNotificationAsync  User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  Received [ModifiedEvent] for subscription: [EgBleGNoMjAxMy5tZW0xMy5vcmcQAAAAjoO0qTL7hk2FF7QvXOHC1BLv0sChZtcIEAAAACanmwmX5x5OpwfUW+dfdrQ=]  
2019/11/11 12:27:55.525 A3  04f06dcb-a721-4a90-a2ff-2be8e007f533    [0000000-0000000]   (52)    Info    MailNotificationService.BusinessImpl.PushNotificationBusiness.HandleMoveModifiedEventAsync  User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  -BADGE UPDATE- [5422] previous BADGE count is [5422] Received modified event for user [20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]   
2019/11/11 12:27:55.525 A3  04f06dcb-a721-4a90-a2ff-2be8e007f533    [0000000-0000000]   (52)    Info    MailNotificationService.BusinessImpl.NotificationsProcessor.AddNotificationToBatch  User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  About to Post Notification for user : [20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] and Device Id : [9]   
2019/11/11 12:28:00.531 A3  d7893c08-3a06-46a1-a8a7-45361572b573    [0000000-0000000]   (16)    Info    MailNotificationService.BusinessImpl.CNSHelper.ComposeAPNSPushNotification  User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  Total unread count retrieved [5422] for user [20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]
2019/11/11 12:28:00.531 A3  d7893c08-3a06-46a1-a8a7-45361572b573    [0000000-0000000]   (16)    Debug   MailNotificationService.BusinessImpl.CNSHelper.ComposeAPNSPushNotification  User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]  Sending to :: User : [20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586], DeviceId : [9], DeviceLogId : [], Message :  messageId: []
2019/11/11 12:28:00.531 A3  d7893c08-3a06-46a1-a8a7-45361572b573    [0000000-0000000]   (16)    Debug   MailNotificationService.BusinessImpl.CNSHelper.CreateWebRequest User Id:[no-user-id]    CNS Url : [https://cns.awmdm.com/nws/notify/apns]  
2019/11/11 12:28:00.531 A3  d7893c08-3a06-46a1-a8a7-45361572b573    [0000000-0000000]   (16)    Debug   MailNotificationService.BusinessImpl.CertificateHelper.ComputeCmsSignature  User Id:[no-user-id]    Signing URL [/nws/notify/apns] with Cert [CN=AW Cloud Notification - aTest]
2019/11/11 12:28:00.748 A3  d7893c08-3a06-46a1-a8a7-45361572b573    [0000000-0000000]   (16)    Debug   MailNotificationService.BusinessImpl.CNSHelper.ReadResponse User Id:[no-user-id]    Response {"status":"success","errorReason":null}   
2019/11/11 12:28:00.748 A3  d7893c08-3a06-46a1-a8a7-45361572b573    [0000000-0000000]   (16)    Info    MailNotificationService.BusinessImpl.CNSHelper.ReadResponse User Id:[no-user-id]    ResponseCode OK

Understanding ENS Logs

The ENS logs contain information about registration, subscriptions, notifications, and the CNS or the APNS delivery status. For the on-premises ENS, you can find the ENS2 logs files at: %ENS Installed Directory%\Logs\Email Notification Service. For example, the ENS2 log file can be at: C:\AirWatch\Logs\Email Notification Service. The name of the log file is ENS.log.

Sample ENS2 log file:

2019/11/05 09:39:56.608 A3  9f08ed6d-0726-430c-8440-9c396443c7ca    [0000000-0000000]   (74)    Debug   
MailNotificationService.BusinessImpl.ExchangeNotificationParser.ScanEventNotificationAsync  
User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]  Received [CreatedEvent] for subscription: [JwBtbjJwcjE5bWIzMDA1Lm5hbXByZDE5LnByb2Qub3V0bG9vay5jb20QAAAAl4H5dKboFUm1kJ8ZNBKkJILRTBjMYdcIEAAAAAQ9tcFCKSZFrTOxLbSCwj4=]   

The following table provides a description of a sample ENS2 log file.

Log Format Value
Date 2019/11/05 09:39:56.608. The date is mentioned in the UTC format.
machinename A3
ActivityId 9f08ed6d-0726-430c-8440-9c396443c7ca
threadid (74)
logLevel Debug
Logger MailNotificationService.BusinessImpl.ExchangeNotificationParser.ScanEventNotificationAsync
Message UserId:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] Received [CreatedEvent] for subscription: [JwBtbjJwcjE5bWIzMDA1Lm5hbXByZDE5LnByb2Qub3V0bG9vay5jb20QAAAAl4H5dKboFUm1kJ8ZNBKkJILRTBjMYdcIEAAAAAQ9tcFCKSZFrTOxLbSCwj4=]
Note:

In the logs, you can find the user name or email address in the alphanumeric format and not in the plain text format. For example, the user ID is mentioned as an alphanumeric string such as, 4e9dc715faba719b266fe90f866caf8e377c08984cd1fd005bac72c7eba4db02. This string is a hash value that is calculated from the email address.

You can use the SHA-256 hash calculator to translate any email address to a hash value. You can then use the hash value to search logs for any user.

To obtain the logs for the cloud ENS, use the ENS2 Self-Help website based on your region.

  • For the US region - https://enshelp.getboxer.com
  • For the EU region - https://enshelp-eu.getboxer.com
  • For the APJ region - https://enshelp-apj.getboxer.com
  • For the UK region - https://enshelp-uk.getboxer.com

Troubleshooting the ENS2 SEG Errors

This section describes the troubleshooting steps you might have to perform due to communication errors between the ENS2 and Exchange with SEGv2 as the proxy.

The following steps describe the interaction between the ENS2 and Exchange with SEGv2 as the proxy.

  1. Boxer application requests a public key from the ENS.
  2. Boxer application encrypts the user credentials using the public key and sends a subscription request to the ENS.
  3. ENS requests a subscription to the Exchange server using the SEG URL which also contains the encrypted credentials. The ENS also sends a client certificate. If the client certificate is configured on the Boxer application profile, then the authentication received from the Boxer profile is sent. For certificate-based authentication (CBA), when a register device request is sent to the cloud ENS server, the ENS routes the request to the SEG with the certificate information. The SEG follows the same token retrieval process similar to the ActiveSync request.
  4. SEG forwards the subscription request to the Exchange to complete the subscription. The same authentication method configured in the Boxer application profile is used for subscription. The ENS server callback URL is used to subscribe.
  5. The Exchange server receives an email.
  6. The Exchange server notifies the ENS callback URL of the subscriber to inform that a new email has arrived, hence update the email client with the notification. The ENS fetches the details of the email from the SEG.
  7. The ENS server requests the CNS or SNS to send notification to the Boxer application or the device of the subscriber.
  8. The CNS or the SNS server contacts the Apple Push Notifications (APNs for iOS devices) or GCM or FCM (for Android devices).
  9. The APNS or GCM server pushes the email notification to the device.

Using the transaction ID received in the ews-transaction log, you can search the ews-proxy.log. For example, if the transaction ID is 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 then search for 544ef2b7.

When you see 200 in the ENS transaction log, you can confirm if the notifications are going through the CNS communication.

The Ews-transaction log sample.

Time, LogLevel, Thread Id, Message, HTTP-Method, Remote-Host, X-Forwarded-For, SEG TransactionId, Request-DeviceId, EnsDevices, EmailServerResponseStatus, SegResponseStatus, EmailRequestBodySize, EmailResponseBodySize, TimeTakenByKerberosService(ms), TimeTakenBySeg(ms), TimeTakenByEmailServer(ms), BeginningOfRequest
2018-12-03 17:20:15.696, DEBUG, (vert.x-eventloop-thread-0), Responding back to ENS,POST,192.168.2.34,null,544ef2b7-9ca3-4009-b116-8a9f6513f2c7,6C30D0304E7A4EE795494DEB0F465B72,"6C30D0304E7A4EE795494DEB0F465B72:200",200,200,1243,1147,2547,0,16,1543875613133
2018-12-03 17:20:18.274, DEBUG, (vert.x-eventloop-thread-0), Responding back to ENS,POST,192.168.2.34,null,d77ae46b-2d38-46b5-9548-3bcc25a1bf03,6C30D0304E7A4EE795494DEB0F465B72,"6C30D0304E7A4EE795494DEB0F465B72:200",200,200,673,1806,2547,0,16,1543875615711
2018-12-03 17:20:41.430, DEBUG, (vert.x-eventloop-thread-0), Responding back to ENS,POST,192.168.2.34,null,b639bdee-0cfa-42b5-82ea-0629ab1d586a,6C30D0304E7A4EE795494DEB0F465B72,"6C30D0304E7A4EE795494DEB0F465B72:200",200,200,1632,2464,2562,0,47,1543875638821
2018-12-03 17:21:16.462, DEBUG, (vert.x-eventloop-thread-1), Responding back to ENS,POST,192.168.2.34,null,ed0db6c1-9dd4-420a-83d3-e746cb17445c,82B15D853CC14CA3989020257158BFC1,"82B15D853CC14CA3989020257158BFC1:200",200,200,1632,3028,2563,0,47,1543875673852
2018-12-03 17:21:26.493, DEBUG, (vert.x-eventloop-thread-1), Responding back to ENS,POST,192.168.2.34,null,425fc495-4ae1-4c26-abc5-c30f34a376cf,82B15D853CC14CA3989020257158BFC1,"82B15D853CC14CA3989020257158BFC1:200",200,200,673,1815,2547,16,15,1543875683915
2018-12-03 17:22:46.649, DEBUG, (vert.x-eventloop-thread-1), Responding back to ENS,POST,192.168.2.34,null,ba0b13ad-b341-43e3-a4a9-d1a79c5330e0,82B15D853CC14CA3989020257158BFC1,"82B15D853CC14CA3989020257158BFC1:200",200,200,1632,3028,2547,15,32,1543875764055
2018-12-03 17:23:01.649, DEBUG, (vert.x-eventloop-thread-1), Responding back to ENS,POST,192.168.2.34,null,262cc2b2-8ae4-4ea7-b062-da2b2eb42a68,82B15D853CC14CA3989020257158BFC1,"82B15D853CC14CA3989020257158BFC1:200",200,200,673,1815,2547,0,15,1543875779087
2018-12-03 17:26:47.353, DEBUG, (vert.x-eventloop-thread-3), Responding back to ENS,POST,192.168.2.34,null,c7e5a6c9-b1b0-4739-9132-49470306882c,6C30D0304E7A4EE795494DEB0F465B72,"6C30D0304E7A4EE795494DEB0F465B72:200",200,200,673,1806,2547,0,94,1543876004712
2018-12-03 17:26:51.884, DEBUG, (vert.x-eventloop-thread-3), Responding back to ENS,POST,192.168.2.34,null,d5cf2470-d818-45f6-ab0e-dd68599d4aa8,6C30D0304E7A4EE795494DEB0F465B72,"6C30D0304E7A4EE795494DEB0F465B72:200",200,200,673,1806,2547,0,15,1543876009322
2018-12-03 22:06:55.421, DEBUG, (vert.x-eventloop-thread-2), Responding back to ENS,POST,192.168.2.34,null,93f7f097-bda5-417a-ac67-5667b4088c84,6C30D0304E7A4EE795494DEB0F465B72,"6C30D0304E7A4EE795494DEB0F465B72:200",200,200,673,1806,12000,16,234,1543892803171
2018-12-03 22:07:00.031, DEBUG, (vert.x-eventloop-thread-0), Responding back to ENS,POST,192.168.2.34,null,d10c08a4-49cd-4240-bcc0-ba9bb81f74f0,82B15D853CC14CA3989020257158BFC1,"82B15D853CC14CA3989020257158BFC1:200",200,200,673,1815,11969,0,188,1543892807874
2018-12-04 10:31:33.786, DEBUG, (vert.x-eventloop-thread-2), Responding back to ENS,POST,192.168.2.34,null,3844719b-73c6-4b77-91d8-8a7d8b9a97c0,82B15D853CC14CA3989020257158BFC1,"82B15D853CC14CA3989020257158BFC1:200",200,200,1632,3028,2563,15,516,1543937490692

The Ews-transaction log sample filtered using the 544ef2b7.

2018-12-03 17:20:13.133 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsRequestReadHandler] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - Incoming EWS request, Path: /EWS/Exchange.asmx. Headers are
2018-12-03 17:20:13.133 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsHelper] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - Collected ENS devices: [6C30D0304E7A4EE795494DEB0F465B72]
2018-12-03 17:20:13.133 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsRequestReadHandler] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - Getting device policy for request device 6C30D0304E7A4EE795494DEB0F465B72
2018-12-03 17:20:13.133 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsComplianceCheckHandler] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - Device list: [6C30D0304E7A4EE795494DEB0F465B72]
2018-12-03 17:20:13.133 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsComplianceCheckHandler] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - Checking compliance for device 6C30D0304E7A4EE795494DEB0F465B72
2018-12-03 17:20:13.133 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsComplianceCheckHandler] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - Device 6C30D0304E7A4EE795494DEB0F465B72 is compliant
2018-12-03 17:20:13.133 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsRequestProxyHandler] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 KCD authentication is (true), upn is TUSER1@MILKYWAY.LOCAL.
2018-12-03 17:20:15.680 DEBUG (pool-7-thread-5) [c.a.s.e.h.EwsRequestProxyHandler] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - Successfully got kerberos token for UPN TUSER1@MILKYWAY.LOCAL - token length 2024
2018-12-03 17:20:15.680 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsRequestProxyHandler] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - Proxying request to EWS
2018-12-03 17:20:15.680 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsRequestProxyHandler] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - EWS client request headers:
2018-12-03 17:20:15.696 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsRequestProxyHandler] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - EWS client response headers:
2018-12-03 17:20:15.696 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsHelper] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - Response headers from SEG to ENS:
X-AW-SEG-TRANSACTION-ID : 544ef2b7-9ca3-4009-b116-8a9f6513f2c7
2018-12-03 17:20:15.696 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsRequestProxyHandler] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - EWS response status 200, length 1147

The possible errors and solutions you might see during an interaction between the ENS2 and Exchange with SEGv2 as the proxy is listed as follows:

Error: 404 / https://[segURL]/EWS/Exchange.asmx is not found

If you see this error in the ENS logs, then ensure you have enabled the EWS proxy in the SEG server. If you have not enabled the EWS proxy in the SEG server then perform the following steps.

  1. Navigate to the SEG > Config folder using the File explorer.
  2. Select the application.properties file and edit the file.
  3. Select the enable.boxer.ens.ews.proxy value and update the value to enable.boxer.ens.ews.proxy=true.
  4. Save the file.
  5. Restart the VMware AirWatch Secure Email Gateway service.

Sample of the application.properties file.

#######################################################################################################################################
############################# Start - HTTP endpoint path for SEG active-sync, syncML and REST API.   ##################################
#######################################################################################################################################
# SEG HTTP server context path. This should be same as the context path of Email/Exchange server as Device won't know
# if it's sending request to email server or SEG Proxy. This value generally don't change but we want to give
# the ability to the Admin to change it, if needed in some exceptional cases.
# Right now Vertx doesn't support "ignore-case" on path, and also doesn't allow mounting sub-routers on RegEx.
# For now we're trying to avoid using RegEx anyway - https://groups.google.com/forum/#!topic/vertx/ck95b4juj4A
activesync.context.paths=/Microsoft-Server-ActiveSync,/microsoft-server-activesync# Context path when SEG works as EWS proxy for ENS. EWS endpoint will be disabled by default.
enable.boxer.ens.ews.proxy=true
ews.proxy.context.paths=/EWS,/ews
 
# Flag used to remove unsupported www-authenticate header such as NTLM and Negotiate (in absense of certificate) from EWS response to ENS.
 
remove.unsupported.auth.for.ews=true

Error: 401 - Please check the authentication type enabled in exchange (EWS endpoint)

If you see this error in the ENS logs, then the SEGv2 does not support the NTLM authentication. If both the Basic and NTLM authentication mechanisms are enabled for the EWS endpoint, then the SEGv2 version prior to version 2.9.0.1 cannot prefer Basic authentication over the unsupported NTLM authentication.

This results in the ENS attempting the NTLM-based authentication for requests through the SEG, that eventually causes 401 error responses as observed in the ews-transaction.log. If the user is unable to disable the NTLM authentication mechanism for the EWS endpoint, and is using any lower version of the SEG, then setup the KCD authentication for the ENS-SEG integration to work correctly.

If you connect directly to the EWS endpoint on the SEGv2 proxy through the https://[segURL]/EWS/Exchange.asmx URL, you might receive a 400 error message unless you connect using a permitted device.

Error: The request was aborted: Could not create SSL/TLS secure channel

In the ENS logs, if you see the following error during the registration process, then the error might be due to a cipher mismatch.

2019-12-05 15:33:40.5081|DEBUG|MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotificationsAsync|3ed2219d-42f2-4a2a-b857-ab7639ad1858|User Id:[af03aa8bb3cae692442ec673b207fbe5666e0762bf3ca62cbaaa61c4208cd7bd] EWSUrl used to subscribe: [https://uag.testdomain.com/ews/exchange.asmx]
 2019-12-05 15:33:40.5550|WARN|MailNotificationService.BusinessImpl.SubscriptionBusiness.SubscribeV2Async|3ed2219d-42f2-4a2a-b857-ab7639ad1858|User Id:[af03aa8bb3cae692442ec673b207fbe5666e0762bf3ca62cbaaa61c4208cd7bd] Service request exception occured for userId [af03aa8bb3cae692442ec673b207fbe5666e0762bf3ca62cbaaa61c4208cd7bd], Inner exception message [The request was aborted: Could not create SSL/TLS secure channel.].

To fix the cipher mismatch error, perform the following steps:

  1. Run a TCP dump on the UAG or SEG. Check the reason for the handshake failure, using the following commands. See the Troubleshooting Firewall and Connection Issues section in the Deploying and Configuring VMware Unified Access Gateway guide.
    /etc/vmware/gss-support/install.sh 
    tcpdump -i any -n -v tcp port any -w /tmp/vmware/capture.pcap
  2. Open the TCP dump logs using the Wireshark or any supported application. Filter the logs based on the IP source and IP destination and check for the client hello request as shown in the following log.

    Use the tls.alert_message.level filter to search for the SSL error or alert in the Wireshark. Identify the source and destination IP, right click, and select Follow > Follow → TLS stream.

  3. Right click and open the Client Hello information.
  4. Click the Show packet > TLS 1.2 Record Layer > Handshake Protocol : Client Hello > Transport Layer Security > Cipher Suits. You can see a list of cipher suites that the client ENS is sending to initiate a secure communication as shown in the following image.
  5. Ensure that the UAG or the SEG server has enabled the ciphers listed in the Client hello Request.
    Note: To check for the enabled cipher suites in the UAG or the SEG server, you can use the SSL report. Enter your SEG or UAG URL and wait for the test to complete. When the test is complete, you might see the following result.

The following table lists all the response codes and messages in the SEG logs.

Response Code Message Description
204 No Content Indicates that the policy data is not loaded in the SEG to run the compliance check on the requesting devices.
403 Forbidden Indicates that none of the devices listed in the ENS request headers are compliant.
400 Bad Request Indicates that none of the devices listed in the ENS request header are found in the SEG device policy cache.
5xx Indicates the server errors.

Troubleshooting Connection Issues to the ENS Database

When installing ENS, use the SQL authentication and not the Windows authentication to access the ENS database. This topic is applicable for ENS on-premise installation only.

Problem: In case you connect to the ENS database using the Windows authentication then you might receive the following error:

2018/11/05 19:55:40.800 EUROPA  80000005-0001-ff00-b63f-84710c7967bb    [0000000-0000000]   (35)    Error   MailNotificationService.ProviderImpl.ApiTokensDataHandler.ApiTokensAsync    User Id:[ ] Error While loading the api tokens Exception [Cannot open database "ENS" requested by the login. The login failed.
Login failed for user 'NT AUTHORITY\LOCAL SERVICE'.] StackTrace[   at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, DbConnectionPool pool, String accessToken, Boolean applyTransientFaultHandling)

Cause:

Connecting to the ENS database using the Windows authentication might cause this issue.

Use the SQL authentication to connect to your ENS database. In the solutions steps provided, the NT AUTHORITY\LOCAL SERVICE is the name of the user account and the database role membership for the NT AUTHORITY\LOCAL SERVICE account must have the db_owner and public enabled.

To add an SQL account to the ENS database, perform the following steps:

  1. Open the SQL Server Management Studio.
  2. Navigate to Security > Logins and add NT AUTHORITY\LOCAL SERVICE.
  3. Navigate to Security > Logins > NT AUTHORITY\LOCAL SERVICE > User Mapping

  4. Select the ENS database and add the required permissions.

Troubleshooting SSL Errors

Use the SSL pinning certificate tool when the notifications are not delivered to the devices. This tool is only used for troubleshooting purpose and is not a mandatory step during installation. The following error message appears in the ENS logs while posting the notifications to CNS: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.

Before you begin:

Download the latest certificate from the CNS Public Certificate.

The following procedure describes the steps to upload the SSL pinning certificate to the ENS.

  1. Click the SSLPinningCertTool shortcut on the ENS server, or navigate to the <ENS_INSTALL_DIR>\Email Notification Service\Tools\SSLPinningCertTool\SSLPinningCertTool.exe file.
  2. Click the Upload CNS Certificate button.
  3. Select the certificate to be uploaded and click Submit.

    Results: If the following screen appears, then the certificate is successfully added to the ENS configuration file. Click OK to continue. After uploading the SSL pinning certificate on the ENS, the tool adds the public key of the certificate to the ENS configuration file. When the ENS posts payload to the CNS, the certificate validation is done against the newly added certificate public key.

    If the following screen appears, then the certificate is added successfully to the resubscription configuration file. After uploading the SSL pinning certificate, the tool adds the public key of the certificate to the resubscription configuration file. When the resubscription mechanism posts payload to the CNS, the certificate validation is done against the newly added certificate public key.

    If the certificate is already present in both the configuration files, then the following prompt message appears:

    The upload pinning certificate process works as follows:

    • The SSL pinning certificate tool tries to upload the certificate to the ENS configuration file only if the provided certificate is not present in the ENS configuration file. If the given certificate is already present, then the tool does not prompt any message and continues to upload the same certificate to the resubscription configuration file.
    • The SSL pinning certificate tool tries to add the certificate to the resubscription configuration file only if the provided certificate is not present in the resubscription configuration file. If the given certificate is present, then the tool does not prompt any message to user.
    • If the certificate is added to the resubscription configuration file, then restart the AirWatch Resubscription Mechanism service in the Services tab.

Troubleshooting AirWatch AutoDiscovery Checker Service Errors

ENS2 depends on the CNS service to deliver email notifications to the devices. If the existing CNS certificate expires then the CNS certificate rotation occurs. The autodiscovery checker service gets the latest certificate and adds the certificate in the ENS server. If the autodiscovery checker service fails to fetch the CNS certificate and add the certificate in the ENS, then ENS cannot send the notifications to devices and receives an SSL error. Perform the following steps to troubleshoot the Airwatch AutoDiscovery Checker service error.

  1. Ensure that the Airwatch AutoDiscovery Checker service is running correctly.

  2. Review the logs for the service at \{ENS installation directory}\Email Notification Service\Services and ensure you are able to see the following log statements without errors: New Certificate Added Successfully.

  3. Review the file at \{ENS installation directory}\Email Notification Service\Website\web.config and ensure that at least 8 pinned certificate elements are listed under the <pinnedCertificates> section.

Troubleshooting Installer Error

The following VMware AirWatch root certificate error might occur if the installer is unable to install the VMware AirWatch root certificate. To resolve this problem, ensure that the installer has the appropriate privileges to install the certificate on the server.

Troubleshooting AutoDiscoveryChecker.log File Errors

The following list describes the possible errors for the AutoDiscoveryChecker.log file:

  • Error while searching for public key in an existing config file
  • Error occurred while updating the config file
  • Exception while obtaining the latest certificates from auto discovery

The following errors might be displayed for the AutoDiscoveryChecker.log file:

  • If the following URL is not reachable https://awtrustdiscovery.awmdm.com/autodiscovery/HostRegistry.aws?URL=cns.awmdm.com
  • If the error is a result of a temporary network failure, the service must attempt to connect to the endpoint again after 24 hours.
  • If the ENS server is configured behind a reverse proxy, or if the outgoing traffic is going through a proxy then the auto discovery service does not go through the proxy and the firewall rules must be updated to allow the IP address 192.30.68.111 for the ENS auto discovery service to be able to reach the following URL http://awtrustdiscovery.awmdm.com/autodiscovery/HostRegistry.aws?URL=cns.awmdm.com.

ENS2 Response Code and Error Code Details

If the Boxer application sends a request to the ENS, the ENS processes the request and sends a response with a response code and message to the Boxer application. The following table lists all the response codes and messages in the Boxer application logs and the ENS logs.

Response Code Message Description
14 SubscribeAgain

If a subscription failed, then the ENS sends a subscribe again message to the Boxer application.

8

ErrorSubscribeOrUpdateDb

When you try to add a user or device details to the database during subscription, you might receive this error .

23

UnAuthorizedRequest

Authentication failed (API token mismatch) for the request from the Boxer application.

32 Failed (Handled Exception)

Registration failed with a handled exception. For example, the URL is not in the correct format.

17

Success

Indicates that the registration is successful.
3 UpdateSuccess

The Boxer application receives this response when the:

  • Get the Public Key request is successful and the database is updated accordingly.
  • The synchronization key for the user success and the database is updated accordingly.
  • Any device details updated in database, such as, update device token is successful.
4 UpdateFail ENS sends this response for multiple reasons. When you receive this response, verify the corresponding ENS logs and troubleshoot based on the message in the logs.
5

TokenDoesNotExists

ENS sends this response when the device record is absent.

When you send a force register, (by changing the notification sound in the Boxer application setting) a new device is created on the ENS server.

6

UserAlreadySubscribed

ENS sends this response when a user is already subscribed on the ENS server.

7

UserSubscribed

User subscription is successful.

9

NoRecordExists

ENS sends this response when a user record is absent.

When you send a force register, (by changing the notification sound in the Boxer application setting) a new user is created on the ENS server.

15

UserSubscribedNotUpdatedInDb

User subscribed but failed to add device details in the database. In this case, ensure that the connection from the ENS to the database is working correctly or the user has permission to update the database.

16

FailedToGetEwsUrlFromAutoDiscovery

Unable to determine the Exchange version after autodiscovery.

21

EmailFetchfailed

Fetching email information from the EWS failed.

24 DeviceUnregisteredUserUnsubscribed

Unsubscribe successful and the user is unregistered.

25

DeviceUnregistered

The device is deleted from the database.

26

DeviceNotRegistered

The device is not registered.

28

UserSubscriptionNotFound

User record does not exist.

29

UserRecordPresentNotSubscribed

User records are present but not subscribed.

30

SubscribedNeedsUpdate

User has already subscribed and must be added to the database.

34

InvalidDecryptedPayload

Payload is encrypted with a wrong public key.

35

EWSUrlMismatch

Unsubscribing with the wrong EWS URL. The EWSUrl for the register request and Exchange service is not the same.

36

InvalidAuthType

Indicates the invalid authentication type.

Troubleshooting the CNS Errors

By default, ENS 21.04 and later versions use CNS2 APIs (https://prod.cns.vmwservices.com), whereas earlier versions of ENS use CNS1 APIs (https://cns.awmdm.com).

If you are having trouble receiving notifications after upgrading your ENS to 21.04, make sure the URL "https://prod.cns.vmwservices.com" is allowlisted. For more information, see the CNS Server IP Allowlist section in ENS2 Requirements and Prerequisites. If the issue persists despite allowlisting this URL, you must update the configuration settings to use the previous CNS version as a temporary workaround and contact VMware support.
  1. Update the web.config file.
    1. Navigate to ENS installed Directory > Email Notification Service > WebSite.
    2. Open the web.config file and look for uri="https://prod.cns.vmwservices.com" and enableCNSv2="true" in airWatchCloudNotificationService.
    3. Change the URI to "https://cns.awmdm.com" and set enableCNSv2="false".
    4. Save the web.config file.
  2. Update the ResubscriptionMechanism.config file.
    1. Navigate to ENS installed Directory > Email Notification Service > Services.
    2. Open the ReSubscriptionMechanism.exe.config file and look for uri="https://prod.cns.vmwservices.com" and enableCNSv2="true" in airWatchCloudNotificationService.
    3. Change the URI to "https://cns.awmdm.com" and set enableCNSv2="false".
    4. Save the ReSubscriptionMechanism.exe.config file.
    5. Restart AirWatch ReSubscription Mechanism from services.

Enable Trace Level Logging for Enhanced Debugging

To help debug and troubleshoot rare issues, ENS2 version 1.5 or later supports trace level logging. To enable trace level logging, you must upgrade to ENS 1.5 or later and perform the following steps to change the log level to ActivityTrace.

  1. Navigate to <INSTALL_FOLDER>\Email Notification Service\WebSite\Web.config path.
  2. Change the value of <loggingConfiguration level="Verbose" /> to <loggingConfiguration level="ActivityTrace" />
  3. Navigate to <INSTALL_FOLDER>\Email Notification Service\Services\ReSubscriptionMechanism.exe.config path.
  4. Change the value of <loggingConfiguration level="Verbose" /> to <loggingConfiguration level="ActivityTrace" />

Result

You can view the trace level logs in the log files within 30 minutes to an hour.

Note: The trace logs might contain PII information which the customer might not wish to share. In such cases, you can mask the PII information.

To mask the PII information, you must manually replace the text with *. For more information, see the following example.

Original trace log without masking:

Trace logs after masking, that is, manually replacing text with *.

Disable Trace Logging

You must disable trace logging after you enable trace logging and debug the issue.

To disable trace logging, change the loggingConfiguration level value from ActivityTrace to Verbose in all the configuration files where the changes were previously made.