This topic lists the various troubleshooting procedures for ENS.
ENS2 Resubscription and Badge Count Accuracy Limitations
The ENS2 uses the Exchange Web Services (EWS) subscription to notify the Boxer application of any changes in an end-users mailbox, including the email notifications. The Boxer application initiates these subscriptions with the ENS and the ENS subscribes a user's account with the EWS.
The EWS is responsible for informing the ENS when there is a change in a user's mailbox. The subscriptions have limited lifetime due to the movement of mailbox, throttling, and so on. The Exchange can drop the EWS push subscriptions which are triggered by the Exchange and the ENS does not have control over the subscription lifetime. The EWS sends notification updates to the Boxer until the EWS subscription is active and alive.
To keep these subscriptions alive, the Boxer application has a check-in mechanism which validates if an EWS subscription is alive. In addition, the ENS2 is listening for status updates from the EWS. If the ENS2 does not receive a status update from the EWS, the ENS2 can send the Boxer a silent push notification to check in with the EWS.
The following figure describes the ENS resubscription process flow.
- The EWS sends a heartbeat signal to the ENS every 15 minutes.
- The ENS sends an acknowledgement to the EWS that the heartbeat signal is received.
- The ENS checks that the heartbeat signal is received every 30 minutes from the EWS.
- If the ENS does not receive a heartbeat signal, the ENS2 sends a silent notification to the Boxer application to initiate the resubscription process.
- The Boxer application initiates a resubscription process on receiving a silent notification.
- The Boxer application proactively checks the EWS subscription status with the ENS server to ensure the continuous delivery of notifications.
The ENS2 requests the Exchange to send heartbeat that a subscription is alive. When the ENS2 does not receive a heartbeat the ENS2 detects a drop in subscription from the EWS. If the subscription is not established and users are not receiving the ENS notifications, the users can manually trigger a resubscription. To trigger a resubscription or if you do not receive ENS notifications after you migrate from on-premises ENS to cloud ENS then change the notification sound in the email account settings section of the Boxer settings as shown in the following image.
The check-in mechanism used by ENS2 requires intervention from Boxer to renew the EWS subscriptions because the users credentials are required to open the subscription. These credentials are not stored in ENS. The functionality of ENS2 also depends on the Apple Push Notification Service (APNS) to deliver silent notifications to the device.
The following list describes the dependencies of the ENS2 on the EWS and APNS.
- If the Boxer application is active and receives a silent notification, the Boxer application attempts to resubscribe. When the Boxer application receives a silent notification, the Boxer sends a resubscription request to the EWS using the employee credentials.
- The iOS can stop the Boxer process without any warning due to various reasons. In such scenarios, the end users might see Boxer in the App Scroll of an iOS device, however, the Boxer process is stopped. The Boxer application has no control over this process and this state is called a killed state. If the Boxer application is in a killed state when it receives a silent notification, the Boxer application cannot resubscribe due to which the user can experience loss of notifications until the user opens the Boxer application. Opening the Boxer application triggers the ENS subscription again, and the user starts receiving notifications.
- The end user might experience an inaccurate badge count when the time subscription is lost and before the Boxer application resubscribes.
The following list describes the badge count accuracy limitations on the Boxer application:
- Sync window - The ENS checks the Inbox folder without the sync period and the Boxer unread messages are within the sync period. So, the users might have unread messages outside the sync window in the Inbox folder. The ENS reports these messages as unread while the user might not see these unread email messages in the Inbox.
- Boxer application dependency on resubscription - When the ENS is going through resubscription, the ENS does not receive any notification or badge count. During this period, the ENS does not have the updated badge count.
- Unmanaged accounts - When the user has both managed and unmanaged accounts like the Exchange account and Gmail account, the badge counts are not handled correctly.
- Comparison with Outlook on MAC devices - The Outlook on MAC devices shows certain emails as read whereas the same emails show unread when opened using Boxer or Outlook for Web Access (OWA). So, the badge count is incorrect when compared with Outlook on MAC devices.
Troubleshooting Accessibility Issues to the ENS Server from a Cloud ENS
Probem: Check if the cloud ENS is accessible from the ENS server and confirm if the ENS server is accessible from the CAS or the Mailbox server.
- Access the following URL in a browser on all CAS or mailbox servers:
https://{ENS cloud URL}/api/ens/alive.
- Select the ENS cloud URL based on your region.
Region ENS Cloud URL North America https://ens.getboxer.com/api/ens Asia Pacific https://ens-apj.getboxer.com/api/ens European Union (EU) https://ens-eu.getboxer.com/api/ens United Kingdom https://ens-uk.getboxer.com/api/ens
Results:
For example, if you enter https://ens.getboxer.com/api/ens/alive you must receive the following response:
This XML file does not appear to have any style information associated with it.
The document tree is shown below.<string xmlns="http://schemas.microsoft.com/2003/10/Serialization/">ens.getboxer.com is alive Version = 1.5.257.10706 Environment = Productio InstanceId = i - 042168040ee9293ac </string>
If you are unable to see a similar response, then allowlist the IP addresses and endpoints and validate the connection to the ENS server. To see the supported ENS2 API endpoints and to receive status updates on ENS2, refer the ENS Endpoints and IP Allowlist and Subscribe to ENS2 Cloud System Status section in the Configure your Email Notification Service for Cloud Deployment topic.
Troubleshooting Accessibility Issues to the ENS Server from an On-Premises Installation
Problem: ENS version 1.10 and later installations fail because the ENSCertificateManager is unable to communicate through proxy.
Support for proxy is added in the ENSCertificateManager service. As a workaround, you must deactivate proxy while installing ENS so that there is a direct communication between the service and the signing.awmdm.com. ENS does not communicate to the signing.awmdm.com after the installation and ENS will function normally.
Problem: Check if the ENS server is accessible on an on-premises setup and is receiving the request. After an on-premises ENS installation, confirm that the ENS is installed and running on the ENS server.
- Navigate to the following URL in a web browser and select the same server where ENS is installed. The user localhost is mentioned as follows: https://localhost/MailNotificationService/api/ens/alive. To check from outside the ENS server, see http://{ENS server public url}/MailNotificationService/api/ens/alive and https://{ENS server public url/MailNotificationService/api/ens/alive. You must be able to view the following response:
This XML file does not appear to have any style information associated with it. The document tree is shown below. <string xmlns="http://schemas.microsoft.com/2003/10/Serialization/"> is alive. Version = 1.5.7249.1115 Enviroment = OnPRem InstanceId = A1 </string>
- Confirm that a certificate is imported and 443 is bound to the website if you have an issue with the https 443 traffic.
Result:
Confirm if the ENS is receiving the request from outside (for example, receiving the request from a browser when you reach the alive endpoint). When verifying the ENS alive endpoint, the IIS logs are generated. The IIS logs are by default stored at the following path: %SystemDrive%\inetpub\logs\LogFiles. If you do not find the logs at the default path, then the logs for your IIS might be stored at a different location. To get the path for the IIS logs, check the following link: Managing IIS Log File Storage.
For other successful ENS traffic, you might see the following log entries in the IIS logs.
Test the Exchange Web Services URL
The Exchange Web Services (EWS) subscriptions notify changes in a users' mailbox. Use the Microsoft's Remote Connectivity Analyzer online tool to test the EWS URL. You can test the EWS URL only if the EWS is configured for the basic authentication and the EWS is publicly available.
- Open the Microsoft's Remote Connectivity Analyzer.
- Select the Synchronization, Notification, Availability, Automatic Replies under the Microsoft Exchange Web Services Connectivity Tests and click Next.
- Enter the Email address, Domain\User Name (or UPN), Password, and Confirm Password information.
- Enter the EWS URL manually, if the autodiscovery is not enabled or select the Use Auto-Discovery to detect server settings if autodiscovery is enabled.
- Click Verify account and perform the test.
Results:
If there are no issues, the following success message appears:
If the connectivity test fails for the following reasons, then expand the error to see more information.
You see the following 401 error when the user is unauthorized.
You see the following error when the autodiscovery is not enabled.
You see the following error when the Remote server cannot be resolved.
Troubleshooting the EWS Accessibility on an On-Premises ENS Installation
Use the EWSEditor tool to check if the EWS is internal and accessible from an on-premises ENS. The EWSEditor tool works only if you are using basic authentication or Open Authorization (OAuth).
- Download and extract the EWSEditor ZIP file from the EWSEditor.
- Run the EWSEditor.exe file.
- Navigate to the and enter the Service URL, User Name, Password, and Domain.
- Click OK. If there is an error in the details entered, then an appropriate error message appears. If the details entered are correct, then the following message appears:
- Click Yes.
- Select the device for which you want to check the subscription and right-click on the device. Select Open Streaming Notifications Viewer.
- Click Subscribe and Clear Events.
- To test the notifications, send a test message to the device. If the test is successful, the following screen appears: EWSEditor
Troubleshooting ENS2 Configuration Issues in Workspace ONE UEM Console
For Workspace ONE UEM Console 2003 or lower
You can configure the ENS2 settings using the configuration key and configuration value provided by the Workspace ONE UEM console.
The following image shows the ENS2 settings when configured without EWS URL and with the EWS URL.
The following table lists the Workspace ONE UEM console configuration keys and values for ENS2.
| Configuration Key | Value Type | Configuration Value |
|---|---|---|
| ENSLinkAddress | String | Specify the URL address of the ENS server.
|
| ENSAPIToken | String | VMware provides the API token to enable the ENS service. For the on-premises installation, the on-premises installer creates this token. |
| AccountNotifyPush | Boolean | This value must be True. |
| EWSUrl | String | Enables manual configuration of the Exchange Web Services (EWS) endpoint when the autodiscovery is deactivated in your Exchange environment. Even for deployments where the autodiscovery is enabled, you must prefer to configure this option. The value of this option is your EWS endpoint. For example, https://outlook.office365.com/EWS/Exchange.asmx (for Office 365) |
For Workspace ONE UEM console version 2004, 2005, 2006, and 2007
You can configure the ENS2 settings and the EWS URL in Workspace ONE UEM console. However, configuring the EWS URL is not mandatory for ENS but it is recommended you configure the EWS URL.
You can configure the ENS2 specific settings and the EWS URL in the section of the Boxer app assignment page. For more information on the ENS2 specific settings, see the Assign and Configure Workspace ONE Boxer Using the App Assignment Page section in the Workspace ONE Boxer Admin Guide and see the Configure ENS2 with Application Configuration Values for Boxer topic for more information on the ENS2 configuration for Boxer.
For Workspace ONE UEM console version 2008 and later
You can configure the ENS2 settings and the EWS URL in Workspace ONE UEM console. However, configuring the EWS URL is not mandatory for ENS but it is recommended you configure the EWS URL. The location where you set the ENS2 specific settings and the EWS URL are different.
You can configure the EWS URL in the Email Settings page and the ENS2 specific settings in the page of the Boxer app assignment page. For more information, see the Assign and Configure Workspace ONE Boxer Using the App Assignment Page section in the Workspace ONE Boxer Admin Guide and see the Configure ENS2 with Application Configuration Values for Boxer topic for more information on the ENS2 configuration for Boxer.
Troubleshooting ENS2 Notification Issues
ENS notifications are applicable only for emails in the Inbox folder. The notifications do not work for Calendar events, sub folders, and so on. The following topics describes the steps to troubleshoot the ENS2 notification issues for emails in the Inbox folder.
Public Key Request from the ENS
The Boxer application requests the public key from the ENS. The public key is used to encrypt the user credentials. When the ENS processes the request, the ENS sends the public key and creates a user record in the database against the user ID. In the following sample, the ENS logs for the GetPublicKeyRequest, the Boxer application sends the SHA256 hash of the email address as the user ID.
2019/10/18 05:54:05.395 WIN-HTCPEDXIUVF 7b21cd56-4c45-4a7c-88d9-a7f225cea3b9 [0000000-0000000] (5) Debug MailNotificationService.Controllers.EnsController.GetPublicKey User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] Processing Get Public key request for Userid[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] 2019/10/18 05:54:05.457 WIN-HTCPEDXIUVF 7b21cd56-4c45-4a7c-88d9-a7f225cea3b9 [0000000-0000000] (5) Debug MailNotificationService.BusinessImpl.GetPublicKeyBusiness.ProcessGetPublicKeyRequestAsync User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] Key generated for user id [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] 2019/10/18 05:54:05.457 WIN-HTCPEDXIUVF 7b21cd56-4c45-4a7c-88d9-a7f225cea3b9 [0000000-0000000] (5) Debug MailNotificationService.Controllers.EnsController.GetPublicKey User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] Get Public Key request processed. HttpStatusCode:[OK] ResponseCode:[UpdateSuccess]
The possible error types and solutions that you might see during a GetPublicKeyRequest is listed as follows:
Error: Unauthorized Request
If you see the following error when you send a GetPublicKeyRequest, then ensure that the provided API token is correct. Verify if the API token is the same at the following instances:
- API token in the ENS logs - API token : [12341*********fasdf]
- The Boxer application configuration in the UEM console. See, the Workspace ONE Boxer Admin Guide for more information on the Boxer application configuration values.
- API token in the Boxer application logs - Verify the API token in the Boxer application logs.
Error: Unable to add a NULL value into the PublicKey column
When the available RSA keys in the database are exhausted, you might see the following error. This issue is automatically fixed when the RSAKey tracker service triggers and generates new keys again.
2019/10/18 12:20:04.121 WIN-HTCPEDXIUVF b4a42dc8-6896-4243-9a4c-8ed476ae94ab [0000000-0000000] (5) Debug MailNotificationService.Controllers.EnsController.GetPublicKey User Id [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] Processing Get Public key request for Userid[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] 2019/10/18 12:20:04.136 WIN-HTCPEDXIUVF b4a42dc8-6896-4243-9a4c-8ed476ae94ab [0000000-0000000] (5) Debug MailNotificationService.BusinessImpl.GetPublicKeyBusiness.ProcessGetPublicKeyRequestAsync User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] Error: 515, Severity: 16, State: 2, Message: "Cannot insert the value NULL into column 'PublicKey', table 'onpremensdev.dbo.UserInfo'; column does not allow nulls. INSERT fails.", Procedure: "UserInfo_Save", Line: 39 2019/10/18 12:20:04.136 WIN-HTCPEDXIUVF b4a42dc8-6896-4243-9a4c-8ed476ae94ab [0000000-0000000] (5) Debug MailNotificationService.Controllers.EnsController.GetPublicKey User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] Get Public Key request processed. HttpStatusCode:[InternalServerError] ResponseCode:[UpdateFail]
Ensure that the following values are present in the RSAKey tracker configuration file:
<add key="numberOfKeysToBeInserted" value="500"/> <add key="wakeUpIntervalInMins" value="120"/> <add key="keysThreshold" value="250"/>
Error: ENS service communication
When communicating with the ENS service, if you see the following error in the Boxer application logs, then ensure that your device has proper connectivity.
2019-11-04T12:23:12Z E [710337] [ENS] An error occurred when communicating with the ENS service: Error Domain=NSURLErrorDomain Code=-1009 "The Internet connection appears to be offline." UserInfo={NSUnderlyingError=0x281fead90 {Error Domain=kCFErrorDomainCFNetwork Code=-1009 "The Internet connection appears to be offline." UserInfo={NSErrorFailingURLStringKey=https://ens-staging.getboxer.com/api/ens/getpublickey, NSErrorFailingURLKey=https://ens-staging.getboxer.com/api/ens/getpublickey, _kCFStreamErrorCodeKey=50, _kCFStreamErrorDomainKey=1, NSLocalizedDescription=The Internet connection appears to be offline.}}, NSErrorFailingURLStringKey=https://ens-staging.getboxer.com/api/ens/getpublickey, NSErrorFailingURLKey=https://ens-staging.getboxer.com/api/ens/getpublickey, _kCFStreamErrorDomainKey=1, _kCFStreamErrorCodeKey=50, NSLocalizedDescription=The Internet connection appears to be offline.} at URL: https://ens-staging.getboxer.com/api/ens/getpublickey. Data: . Response Code: 0
2019-11-04T12:23:12Z E [710337] [ENS] Error registering new account: vmwUser4@awRed.onmicrosoft.com
Error:Error Domain=com.alamofire.serialization.response.error.response Code=-1 "invalid public key" UserInfo={NSLocalizedDescription=invalid public key}
2019-11-04T12:23:12Z E [703318] [ENS] Error registering device for push notification
Error:Error Domain=com.alamofire.serialization.response.error.response Code=-1 "invalid public key" UserInfo={NSLocalizedDescription=invalid public key}
2019-11-04T12:23:12Z E [726177] - Unexpected error: {
BXLocalizedContextMessageErrorKey = "Could not update settings for the push notification service";
BXLocalizedTitleErrorKey = "Could not update settings for the push notification service";
NSLocalizedDescription = "Could not update settings for the push notification service. ";
NSLocalizedFailureReason = "Failed to update push notification settings. Please contact your administrator.";
} context: 1
Register Device Request
The Boxer application sends a Register request to the ENS, a push subscription to the EWS, and a subscribe for notification. If the GetPublicKey request is successful, then the Boxer application sends a register request to the ENS with the necessary information required to register a device for notification.
Scenario 1: - If the EWS URL is not configured in the console, then the ENS tries autodiscovery to obtain the EWS URL to subscribe the user.
Scenario 2: - If the EWS URL is configured in the console, then the ENS uses the same EWS URL to subscribe the user.
When the subscription is successful, the ENS receives the [UserSubscribed] message with the subscription ID as mentioned in the following code snippet.
2019/11/05 08:18:49.674 A3 726c4072-5144-4450-848b-821f65174b89 [0000000-0000000] (23) Info MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotificationsAsync User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] User [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] subscribed with subscriptionId [JwBtbjJwcjE5bWIzMDA1Lm5hbXByZDE5LnByb2Qub3V0bG9vay5jb20QAAAAJ6RYazaIoUCfX7KheUsQYUQnw9rIYdcIEAAAAAQ9tcFCKSZFrTOxLbSCwj4=] 2019/11/05 08:18:49.767 A3 726c4072-5144-4450-848b-821f65174b89 [0000000-0000000] (28) Debug MailNotificationService.Controllers.EnsController.RegisterDeviceV2 User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] Register device request processed. HttpStatusCode:[OK] ResponseCode:[UserSubscribed]
In the Android Boxer logs, you must see the following log entries to confirm a successful registration:
--------------------------------------------------------------
ENS SETTINGS
--------------------------------------------------------------
ENS_LINK_ADDRESS = https://ens.getboxer.com/api/ens
ENS_API_TOKEN = 17413**********************88c08
POLICY_ACCOUNT_NOTIFY_PUSH = true
EWS_URL =
ENS_STATE = (8 -> Registered)
--------------------------------------------------------------
HEALTH STATUS
--------------------------------------------------------------
App version health status: Green, Current app version: 5.11.0.4, New version: 5.10.0
Sync Health Status: Green, Sync durations in seconds: [0.522, 0.49, 0.416, 0.379, 0.424, 0.368, 0.465, 0.496, 0.565, 1.344], Sync results [OK, OK, OK, OK, OK, OK, OK, OK, OK, OK]
Ens health status: Green , Ens state: Registered
Overall health status: Green
Ens registration for account (id=8) is successful!
For the iOS Boxer logs, you must see the following log entries to confirm a successful registration:
For normal subscription 2019-11-11T09:31:41Z I [12347] [ENS] Successfully registered account.
The possible errors and solutions that you might see when you are unable to locate the autodiscover services are listed as follows:
Error: Unable to Locate the Autodiscover Services
If you see the following error, then ensure to enable autodiscovery, check the availability and connectivity of the autodiscovery server using the EWSEditor and the MS remote connectivity analyzer.
2019/11/06 07:01:56.207 A3 d252be19-1c5d-4e30-9155-a0ae3a529679 [0000000-0000000] (94) Warn MailNotificationService.BusinessImpl.SubscriptionBusiness.SubscribeV2Async User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] Exception while auto discovery occured for userId [20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586], Exception Message [The Autodiscover service couldn't be located.] , Exception [Microsoft.Exchange.WebServices.Data.AutodiscoverLocalException: The Autodiscover service couldn't be located. at Microsoft.Exchange.WebServices.Autodiscover.AutodiscoverService.InternalGetLegacyUserSettings[TSettings](String emailAddress, List`1 redirectionEmailAddresses, Int32& currentHop) at Microsoft.Exchange.WebServices.Autodiscover.AutodiscoverService.GetLegacyUserSettings[TSettings](String emailAddress) at Microsoft.Exchange.WebServices.Autodiscover.AutodiscoverService.InternalGetLegacyUserSettings(String emailAddress, List`1 requestedSettings) at Microsoft.Exchange.WebServices.Data.ExchangeService.GetAutodiscoverUrl(String emailAddress, ExchangeVersion requestedServerVersion, AutodiscoverRedirectionUrlValidationCallback validateRedirectionUrlCallback) at Microsoft.Exchange.WebServices.Data.ExchangeService.AutodiscoverUrl(String emailAddress, AutodiscoverRedirectionUrlValidationCallback validateRedirectionUrlCallback) at MailNotificationService.BusinessImpl.ExchangeServiceBusiness.<GetExchangeServiceViaAutoDiscovery>d__10.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at MailNotificationService.BusinessImpl.ExchangeServiceBusiness.<GetExchangeServiceAsync>d__6.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at MailNotificationService.BusinessImpl.SubscriptionBusiness.<SubscribeV2Async>d__7.MoveNext()], Inner Exception [], Autodiscover url used [The Autodiscover service couldn't be located.] 2019/11/06 07:01:56.207 A3 d252be19-1c5d-4e30-9155-a0ae3a529679 [0000000-0000000] (94) Debug MailNotificationService.Controllers.EnsController.RegisterDeviceV2 User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] Register device request processed. HttpStatusCode:[Conflict] ResponseCode:[SubscribeAgain]
Error: The remote server returned an error (403) Forbidden
If this error occurs during a subscription, then ensure to enter the proper EWS URL in the Boxer application KVP values of the UEM console. The EWSUrl used to subscribe must have the complete endpoint specified.
Example of a correct EWSUrl - [https://mail-mem13.xyz.com/EWS/exchange.asmx]
Example of an incorrect EWSUrl - [https://mail-xyz.com/]
To check the EWS URL availability and connectivity, check the EWSEditor and the MS remote connectivity analyzer.
2019/11/06 07:09:54.064 A3 f43eb3d0-e173-49de-9b52-3acb8a1107c4 [0000000-0000000] (98) Debug MailNotificationService.Controllers.EnsController.RegisterDeviceV2 User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] Processing register device request for Userid[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] 2019/11/06 07:09:54.080 A3 f43eb3d0-e173-49de-9b52-3acb8a1107c4 [0000000-0000000] (98) Debug MailNotificationService.BusinessImpl.RegisterDeviceBusiness.ProcessRegisterDeviceRequestAsyncV2 User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] Exchange version sent by boxer [2] 2019/11/06 07:09:54.080 A3 f43eb3d0-e173-49de-9b52-3acb8a1107c4 [0000000-0000000] (98) Debug MailNotificationService.BusinessImpl.ExchangeServiceBusiness.GetExchangeServiceAsync User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] Using client ewsurl, mailServerUrlMatched : False, deletedEWSUrl: False 2019/11/06 07:09:54.080 A3 f43eb3d0-e173-49de-9b52-3acb8a1107c4 [0000000-0000000] (98) Debug MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotificationsAsync User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] EWSUrl used to subscribe: [https://mail-mem13.ssdevrd.com/] 2019/11/06 07:09:54.080 A3 f43eb3d0-e173-49de-9b52-3acb8a1107c4 [0000000-0000000] (98) Debug MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotificationsAsync User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] User subscribing with [Basic Auth] 2019/11/06 07:09:54.173 A3 f43eb3d0-e173-49de-9b52-3acb8a1107c4 [0000000-0000000] (98) Warn MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotifications User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] Service request exception occured for userId [20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586], Inner exception message [The remote server returned an error: (403) Forbidden.] Going for a retry, 2019/11/06 07:09:54.173 A3 f43eb3d0-e173-49de-9b52-3acb8a1107c4 [0000000-0000000] (98) Debug MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotificationsAsync User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] EWSUrl used to subscribe: [https://mail-mem13.ssdevrd.com/] 2019/11/06 07:09:54.173 A3 f43eb3d0-e173-49de-9b52-3acb8a1107c4 [0000000-0000000] (98) Debug MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotificationsAsync User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] User subscribing with [Basic Auth] 2019/11/06 07:09:54.205 A3 f43eb3d0-e173-49de-9b52-3acb8a1107c4 [0000000-0000000] (98) Warn MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotifications User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] Service request exception occured for userId [20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586], Inner exception message [The remote server returned an error: (403) Forbidden.] Going for a retry, 2019/11/06 07:09:54.205 A3 f43eb3d0-e173-49de-9b52-3acb8a1107c4 [0000000-0000000] (98) Debug MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotificationsAsync User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] EWSUrl used to subscribe: [https://mail-mem13.ssdevrd.com/] 2019/11/06 07:09:54.205 A3 f43eb3d0-e173-49de-9b52-3acb8a1107c4 [0000000-0000000] (98) Debug MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotificationsAsync User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] User subscribing with [Basic Auth] 2019/11/06 07:09:54.236 A3 f43eb3d0-e173-49de-9b52-3acb8a1107c4 [0000000-0000000] (98) Warn MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotifications User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] Service request exception occured for userId [20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586], Inner exception message [The remote server returned an error: (403) Forbidden.] Going for a retry, 2019/11/06 07:09:54.236 A3 f43eb3d0-e173-49de-9b52-3acb8a1107c4 [0000000-0000000] (98) Debug MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotificationsAsync User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] EWSUrl used to subscribe: [https://mail-mem13.ssdevrd.com/] 2019/11/06 07:09:54.236 A3 f43eb3d0-e173-49de-9b52-3acb8a1107c4 [0000000-0000000] (98) Debug MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotificationsAsync User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] User subscribing with [Basic Auth] 2019/11/06 07:09:54.251 A3 f43eb3d0-e173-49de-9b52-3acb8a1107c4 [0000000-0000000] (98) Warn MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotifications User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] Service request exception occured for userId [20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586], Inner exception message [The remote server returned an error: (403) Forbidden.] Going for a retry, 2019/11/06 07:09:54.251 A3 f43eb3d0-e173-49de-9b52-3acb8a1107c4 [0000000-0000000] (98) Warn MailNotificationService.BusinessImpl.SubscriptionBusiness.SubscribeV2Async User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] Service request exception occured for userId [20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586], Inner exception message [The remote server returned an error: (403) Forbidden.] Going for a retry, 2019/11/06 07:09:54.251 A3 f43eb3d0-e173-49de-9b52-3acb8a1107c4 [0000000-0000000] (98) Debug MailNotificationService.Controllers.EnsController.RegisterDeviceV2 User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] Register device request processed. HttpStatusCode:[Conflict] ResponseCode:[SubscribeAgain]
Sample error logs of Boxer during registration:
2019-11-11T09:13:43Z E [9326] [ENS] An error occurred when communicating with the ENS service: Error Domain=com.alamofire.error.serialization.response Code=-1011 "Request failed: conflict (409)" UserInfo={NSLocalizedDescription=Request failed: conflict (409), NSErrorFailingURLKey=https://a3.ssdevrd.com/mailnotificationservice/api/ens/registerdevicev2, com.alamofire.serialization.response.error.data={length = 135, bytes = 0x7b227265 73706f6e 7365436f 6465223a ... 4f6e5072 656d227d }, com.alamofire.serialization.response.error.response=<NSHTTPURLResponse: 0x282db1fa0> { URL: https://a3.ssdevrd.com/mailnotificationservice/api/ens/registerdevicev2 } { Status Code: 409, Headers {
"Content-Length" = (
135
);
"Content-Type" = (
"application/json; charset=utf-8"
);
Date = (
"Mon, 11 Nov 2019 09:13:40 GMT"
);
Server = (
"Microsoft-IIS/8.5"
);
"X-Powered-By" = (
"ASP.NET"
);
} }} at URL: https://a3.ssdevrd.com/mailnotificationservice/api/ens/registerdevicev2. Data: {"responseCode":14,"errorMessage":"The Autodiscover service couldn't be located.","version":"1.5.7235.6268","environmentType":"OnPrem"}. Response Code: 409
2019-11-11T09:13:43Z E [9326] [ENS] registerAccountOnENS: Error updating settings or credentials
Error:Error Domain=com.alamofire.error.serialization.response Code=-1011 "Request failed: conflict (409)" UserInfo={NSLocalizedDescription=Request failed: conflict (409), NSErrorFailingURLKey=https://a3.ssdevrd.com/mailnotificationservice/api/ens/registerdevicev2, com.alamofire.serialization.response.error.data={length = 135, bytes = 0x7b227265 73706f6e 7365436f 6465223a ... 4f6e5072 656d227d }, com.alamofire.serialization.response.error.response=<NSHTTPURLResponse: 0x282db1fa0> { URL: https://a3.ssdevrd.com/mailnotificationservice/api/ens/registerdevicev2 } { Status Code: 409, Headers {
"Content-Length" = (
135
);
"Content-Type" = (
"application/json; charset=utf-8"
);
Date = (
"Mon, 11 Nov 2019 09:13:40 GMT"
);
Server = (
"Microsoft-IIS/8.5"
);
"X-Powered-By" = (
"ASP.NET"
);
} }}
2019-11-11T09:13:43Z E [9365] - Unexpected error: {
BXLocalizedContextMessageErrorKey = "Could not update settings for the push notification service";
BXLocalizedTitleErrorKey = "Could not update settings for the push notification service";
NSLocalizedDescription = "Could not update settings for the push notification service. ";
NSLocalizedFailureReason = "Failed to update push notification settings. Please contact your administrator.";
}
{"responseCode":14,"errorMessage":"The Autodiscover service couldn't be
located...
In this case, ensure that the autodiscovery URL is reachable from the ENS and the autodiscovery URL is configured correctly using the EWSEditor tool or MS connectivity analyzer tool.
If you are using the EWSUrl, ensure that the EWSUrl key is configured in the console with a correct value for the EWSUrl of their respective Exchange environments. To verify the EWSUrl is correct, open a browser, enter the EWSUrl, and ensure that you are prompted to enter the credentials.
You can find the error message and response code for different reasons. Based on the error message, you can start troubleshooting the issue.
Error: 403 or 401 error message
EWS must be accessible to the ENS application to subscribe the user for notification. If the EWS is not configured correctly, then you might receive 403 or 401 error. In such cases, refer the following documents:
- Getting started with the EWS Managed API 2.0
- Managing access for EWS Managed API 2.0 applications
- Authentication and EWS in Exchange
Check the type of authentication you have enabled in the EWS. Ensure that the authentication is in parity with what the customer is using for ActiveSync (Basic, OAuth, and CBA). The Boxer application sends the user credentials to the ENS and the ENS uses the same credentials and the same type of authentication to communicate with the EWS.
If you are using Office 365 and you receive a 401 error from the EWS URL, the reason for the error might be because the client access rules or ADFS claims are configured. In such scenarios, refer the following documents.
In a scenario where the ENS on-premises Exchange with CBA is enabled, you might need to confirm that the client certificate is arriving at the Exchange endpoint. To troubleshoot any errors, see the Troubleshooting ENS with On-Premise Exchange Server section.
Force Register or Re-register on the Boxer application:
On iOS devices only, you can manually perform a force subscription, in the following cases:
- If there are any changes to the keys in console, then you must approximately wait for 1 hour and check if the users are still receiving the notification. If the users are not receiving notifications, you can proceed to re-register the Boxer application with the ENS2 service.
- If you do not see any register request in the ENS logs from the Boxer application, then assume that the Boxer application has failed to send the register request automatically. Therefore, the ENS tries to re-register the Boxer application with the ENS2 service forcefully.
To force register or re-register on the Boxer application, perform the following steps:
- Open the Boxer application and click Settings.
- Under the Accounts tab, select your ENS-specific account.
- Turn off the Use Push Service option.
- Navigate to the Boxer application Settings screen.
- Repeat Step 2 through Step 4 to turn on the Use Push Service option.
When you perform either of the steps mentioned, then you can see the force register request in the ENS logs.
To confirm the force subscription in the ENS logs, search for the ForceSubscription and you must be able to see the following value: ForceSubscription : [True].
Registration Status Events
If the registration is successful, then the Exchange sends a status event to the ENS periodically against each subscription ID, to confirm the subscription. The ENS then sends an acknowledgment for each of the subscription IDs back to the Exchange.
2019/11/05 08:57:31.413 A3 1eb9186b-9370-45de-a172-0e452586f398 [0000000-0000000] (58) Debug MailNotificationService.BusinessImpl.ExchangeNotificationParser.ScanEventNotificationAsync User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] Received [StatusEvent] for subscription: [JwBtbjJwcjE5bWIzMDA1Lm5hbXByZDE5LnByb2Qub3V0bG9vay5jb20QAAAAl4H5dKboFUm1kJ8ZNBKkJILRTBjMYdcIEAAAAAQ9tcFCKSZFrTOxLbSCwj4=] 2019/11/05 08:57:31.413 A3 1eb9186b-9370-45de-a172-0e452586f398 [0000000-0000000] (58) Debug MailNotificationService.BusinessImpl.PushNotificationBusiness.HandleExchangeEvents User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] Status event received for user: [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]
2019/11/05 08:49:20.123 A3 d2adec8a-73d7-48f2-ba14-abbd917844cd [0000000-0000000] (54) Info MailNotificationService.BusinessImpl.ExchangeNotificationParser.ScanEventNotificationAsync User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] This JwBtbjJwcjE5bWIzMDA1Lm5hbXByZDE5LnByb2Qub3V0bG9vay5jb20QAAAAJ6RYazaIoUCfX7KheUsQYUQnw9rIYdcIEAAAAAQ9tcFCKSZFrTOxLbSCwj4= is old subscription for user 1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d, sending unsubscribe response 2019/11/05 08:49:20.123 A3 d2adec8a-73d7-48f2-ba14-abbd917844cd [0000000-0000000] (54) Debug MailNotificationService.BusinessImpl.ExchangeNotificationParser.ScanEventNotificationAsync User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] Sent Unsubscribe response to EWS successfully for subscriptionId: [JwBtbjJwcjE5bWIzMDA1Lm5hbXByZDE5LnByb2Qub3V0bG9vay5jb20QAAAAJ6RYazaIoUCfX7KheUsQYUQnw9rIYdcIEAAAAAQ9tcFCKSZFrTOxLbSCwj4=] 2019/11/05 08:49:20.123 A3 d2adec8a-73d7-48f2-ba14-abbd917844cd [0000000-0000000] (54) Debug MailNotificationService.BusinessImpl.PushNotificationBusiness.ProcessPushNotificationV2Async User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] ProcessNotificationResponse.IsUnSubscribeResponse is true
For more information on the status frequency, see the StatusFrequency topic.
ENS must receive the status events from the Exchange immediately after a subscription is successful. If the ENS is not receiving the status events, then check the following troubleshooting methods to verify the communication between the Exchange server and the ENS.
Error: Status event not received
If you do not see any status events in the ENS logs after a successful subscription, then check the communication between the Exchange server and the ENS. Access the following URLs in the browser on the CAS or the mailbox servers to check the communication between the Exchange and the ENS.
- For on-premises ENS deployments, use the https://{ENS URL}/MailNotificationService/api/ens/alive.
- For cloud ENS deployments, use the https://{ENS URL}/api/ens/alive. For example, https://ens.getboxer.com/api/ens/alive. Select the ENS cloud URL based on your region.
You must be able to see the following result when you browse the specified URLs from the browser.
This XML file does not appear to have any style information associated with it. The document tree is shown below. <string xmlns="http://schemas.microsoft.com/2003/10/Serialization/"> is alive. Version = 1.5.7227.9937 Enviroment = Production InstanceId = i-04676f24928463e31 </string>
- For on-premises ENS deployments, use the https://{ENS URL}/MailNotificationService/api/ens/pushnotificationlistener.
- For cloud ENS deployments, use the https://{ENS URL}/api/ens/pushnotificationlistener. Select the ENS cloud URL based on your region.
<Error> <Message> The requested resource does not support http method 'GET'. </Message> </Error>When browsing the URLs, if you see any SSL error, then proceed to import the ENS certificate in the MMC of the server.
pushnotificationlistener logs. Right click on the
Application node and navigate to
Find. If there are any entries then check if there are any communication errors while pushing the notification to the ENS.
New Mail Event and Fetch Mail
When a device is successfully registered and the communication between the ENS and the Exchange is working correctly, the Exchange starts sending new mail events to the ENS whenever a new mail is received on the subscribed user mailbox. If the payloads of the created events contain an unread count, then the ENS uses the unread count, else the ENS gets the unread count from the EWS.
2019/11/05 09:39:56.608 A3 9f08ed6d-0726-430c-8440-9c396443c7ca [0000000-0000000] (74) Debug MailNotificationService.BusinessImpl.ExchangeNotificationParser.ScanEventNotificationAsync User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] Received [CreatedEvent] for subscription: [JwBtbjJwcjE5bWIzMDA1Lm5hbXByZDE5LnByb2Qub3V0bG9vay5jb20QAAAAl4H5dKboFUm1kJ8ZNBKkJILRTBjMYdcIEAAAAAQ9tcFCKSZFrTOxLbSCwj4=] 2019/11/05 09:39:56.639 A3 9f08ed6d-0726-430c-8440-9c396443c7ca [0000000-0000000] (74) Debug MailNotificationService.BusinessImpl.UnreadCountExchangeBusiness.GetUnReadCountV2 User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] EWSUrl used to get unread count: [https://outlook.office365.com/EWS/Exchange.asmx] 2019/11/05 09:39:56.889 A3 9f08ed6d-0726-430c-8440-9c396443c7ca [0000000-0000000] (74) Info MailNotificationService.BusinessImpl.PushNotificationBusiness.HandleNewMailEvent User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] Received new mail event for user [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] with BADGE count [893]
Whenever the ENS receives a new mail event, the ENS fetches the mail information from the Exchange. The possible errors and solutions that you might see during a fetch mail request is listed as follows:
Error: Stuck in EWSUrl used to sync email: [https://outlook.office365.com/EWS/Exchange.asmx] steps
When a mail event is received from the Exchange, the ENS tries to fetch all the information from the mail. If you are unable to see any ENS logs such as the Fetched email, then check the respective EWS logs in the Exchange. You can obtain the corresponding EWS logs using the client request ID or the activity ID.
Fetch New Mail Request
Sample client request ID or the activity ID: 03ea7f36-f72f-4322-8413-0dcd81c4ac78
ENS sends a push notification request to the CNS or the SNS
When the new mail information is fetched from the Exchange, the ENS composes and sends a notification payload to the CNS (for on-premises) or the SNS (for cloud).
Sample of sending a notification payload to the CNS (for on-premises)
2019/11/05 09:48:42.675 A3 fedf9a1d-6cc8-4607-acad-ae006766292a [0000000-0000000] (82) Info MailNotificationService.BusinessImpl.NotificationsProcessor.AddNotificationToBatch User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] About to Post Notification for user : [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] and Device Id : [1]
2019/11/05 09:48:42.690 A3 fedf9a1d-6cc8-4607-acad-ae006766292a [0000000-0000000] (82) Info MailNotificationService.BusinessImpl.NotificationsProcessor.AddNotificationToBatch User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] About to Post Notification for user : [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] and Device Id : [5]
2019/11/05 09:48:47.699 A3 7e45c693-511b-4c19-ae7c-305e5f8f9f0e [0000000-0000000] (8) Info MailNotificationService.BusinessImpl.CNSHelper.ComposeAPNSPushNotification User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] Total unread count retrieved [894] for user [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]
2019/11/05 09:48:47.699 A3 7e45c693-511b-4c19-ae7c-305e5f8f9f0e [0000000-0000000] (8) Debug MailNotificationService.BusinessImpl.CNSHelper.ComposeAPNSPushNotification User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] Sending to :: User : [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d], DeviceId : [1], DeviceLogId : [], Message : messageId: [AAMkAGMxYjUzZDA0LTI5NDItNDUyNi1hZDMzLWIxMmRiNDgyYzIzZQBGAAAAAAAOx2petA5rS4RDQM8RjW1TBwDnjcIsAp4/S4beDDAIaXMhAAAAAAEMAADnjcIsAp4/S4beDDAIaXMhAAGszQatAAA=]
2019/11/05 09:48:47.699 A3 7e45c693-511b-4c19-ae7c-305e5f8f9f0e [0000000-0000000] (8) Info MailNotificationService.BusinessImpl.CNSHelper.ComposeAPNSPushNotification User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] Total unread count retrieved [894] for user [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d]
2019/11/05 09:48:47.699 A3 7e45c693-511b-4c19-ae7c-305e5f8f9f0e [0000000-0000000] (8) Debug MailNotificationService.BusinessImpl.CNSHelper.ComposeAPNSPushNotification User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] Sending to :: User : [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d], DeviceId : [5], DeviceLogId : [61F9BB13-863C-444C-A300-4F888383ACDD-534-0000000CE599EDE0], Message : messageId: [AAMkAGMxYjUzZDA0LTI5NDItNDUyNi1hZDMzLWIxMmRiNDgyYzIzZQBGAAAAAAAOx2petA5rS4RDQM8RjW1TBwDnjcIsAp4/S4beDDAIaXMhAAAAAAEMAADnjcIsAp4/S4beDDAIaXMhAAGszQatAAA=]
2019/11/05 09:48:47.699 A3 7e45c693-511b-4c19-ae7c-305e5f8f9f0e [0000000-0000000] (8) Debug MailNotificationService.BusinessImpl.CNSHelper.CreateWebRequest User Id:[no-user-id] CNS Url : [https://cns.awmdm.com/nws/notify/apns]
2019/11/05 09:48:47.699 A3 7e45c693-511b-4c19-ae7c-305e5f8f9f0e [0000000-0000000] (8) Debug MailNotificationService.BusinessImpl.CertificateHelper.ComputeCmsSignature User Id:[no-user-id] Signing URL [/nws/notify/apns] with Cert [CN=AW Cloud Notification - aTest]
2019/11/05 09:48:48.558 A3 7e45c693-511b-4c19-ae7c-305e5f8f9f0e [0000000-0000000] (8) Debug MailNotificationService.BusinessImpl.CNSHelper.ReadResponse User Id:[no-user-id] Response {"status":"success","errorReason":null}
2019/11/05 09:48:48.558 A3 7e45c693-511b-4c19-ae7c-305e5f8f9f0e [0000000-0000000] (8) Info MailNotificationService.BusinessImpl.CNSHelper.ReadResponse User Id:[no-user-id] ResponseCode OK
Sample of sending a notification payload to the SNS (for cloud)
2019-09-06 12:11:51.5380|INFO|MailNotificationService.BusinessImpl.NotificationsProcessor.AddNotificationToBatch|b1d8e164-c3fb-4f67-baa6-002dd3719c4e|User Id:[35045e4062200ca81c92d5b03928a7e86383ef8e9436d512187a711a4b18e94f] About to Post Notification for user [35045e4062200ca81c92d5b03928a7e86383ef8e9436d512187a711a4b18e94f] 2019-09-06 12:11:52.5537|INFO|MailNotificationService.BusinessImpl.AmazonSNSHelper.PostNotifications|67d3c6f0-a197-4af4-958c-260eeedbf567|User Id:[35045e4062200ca81c92d5b03928a7e86383ef8e9436d512187a711a4b18e94f] Sending notification via SNS 2019-09-06 12:11:52.5692|INFO|MailNotificationService.BusinessImpl.AmazonSNSHelper.PushNotificationViaSNS|67d3c6f0-a197-4af4-958c-260eeedbf567|User Id:[35045e4062200ca81c92d5b03928a7e86383ef8e9436d512187a711a4b18e94f] Notification successfully sent via SNS for [424716]
To confirm if your Android device is receiving notifications from the ENS, enable the Boxer application passcode and restart the device after a successful registration. You might see a notification, that is, a banner containing the email address configured. On the banner notification if you cannot perform actions such as, Delete, Reply, and Read option then, the notification is a push notification that is sent from the ENS and not locally from the Boxer application itself. If the notification banner contains notification actions such as Delete, Reply, Read, and so on, then the notification is a local notification from the Boxer application and not a push notification from the ENS.
The possible errors and solutions that you might see during a push notification request is listed as follows:
Error: The remote server returned an error: (400) Bad Request
Sample error log:
apis
[0000000-0000000] (5) Error MailNotificationService.BusinessImpl.CNSHelper.ReadResponse User Id:[no-user-id] Failed To Post to CNS [https://cns.awmdm.com/nws/notify/apns] Error: [The remote server returned an error: (400) Bad Request.] Response: [{"status":"failure","errorReason":"Unable to process json input, errors are Unregistered, requestId 8f8e1939-3660-43d9-b873-a7ae61ea2b7c"}] ”fcm
[0000000-0000000] (128) Error MailNotificationService.BusinessImpl.CNSHelper.ReadResponse User Id:[no-user-id] Failed To Post to CNS [https://cns.awmdm.com/cns/services/api/notifications/fcm] Error: [The remote server returned an error: (400) Bad Request.] Response: [[{"fcmResults":[{"fcmMessageId":null,"canonicalRegistrationId":null,"errorCode":"NotRegistered"}],"messageId":"AM0PR03MB4067FD025796AB3867E3C5AEEA4F9@AM0PR03MB4067.eurprd03.prod.outlook.com","fcmMulticastId":3844875711768273544,"successCount":0,"failureCount":1,"allFcmCloudError":false,"allMsgsInNonFcmError":true,"fewMsgsInNonFcmError":false,"errCode":5004}]]"
To troubleshoot the issue, share the trace level logs with the VMware Support team. For more information, see the Enable Trace Level Logging for Enhanced Debugging section.
Error: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel
2019/11/06 09:03:48.218 A3 aa57f568-6871-42cc-8b8d-39c77a15af41 [0000000-0000000] (40) Error MailNotificationService.BusinessImpl.CNSHelper.ReadResponse User Id:[no-user-id] Failed To Post to CNS [https://cns.awmdm.com/nws/notify/apns] Error: [The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.] Response: []
If the issue still persists, download the latest public CNS certificate from the CNS Public Certificate and perform the following steps:
- Click the SSLPinningCertTool shortcut present in the ENS server or click <ENS_INSTALL_DIR>\Email Notification Service\Tools\SSLPinningCertTool\SSLPinningCertTool.exe.
- Click the Upload CNS Certificate button.
- Select the certificate to be uploaded and click Submit. If the following screen appears, then the certificate is successfully added.
Note: After uploading the SSL pinning certificate on the ENS, the tool adds the public key of the certificate to the ENS configuration. When the ENS posts payload to the CNS, the certificate validation is done against the newly added certificate public key. - If the following screen appears, the certificate is successfully added to the resubscription configuration file.
Note: After uploading the SSL pinning certificate, the tool adds the public key of the certificate to the resubscription configuration file. For the resubscription mechanism, after payload (silent notification) to the CNS, the certificate validation is done against the newly added certificate public key. - If the certificate is already present in both the configuration files, then you are prompted with the following message.
Note: The upload pinning certificate occurs as follows:- The tool tries to upload the certificate to the ENS configuration file only if the provided certificate is not present in the ENS configuration file. If the given certificate is already present, then the tool does not prompt any message and continues to upload the same certificate to the resubscription configuration file.
- The tool tries to add a certificate to the resubscription configuration file only if the provided certificate is not present in the resubscription configuration file. If the given certificate is present, then the tool does not prompt any message to the user.
- If the certificate is added to the resubscription configuration file, then navigate to Services and restart the AirWatch Resubscription Mechanism service.
Error: The remote server returned an error: (401) Unauthorized.
Sample error log:
2019/11/06 09:25:13.688 A3 6c041e00-c909-45ff-b340-283844376c06 [0000000-0000000] (6) Error MailNotificationService.BusinessImpl.CNSHelper.ReadResponse User Id:[no-user-id] Failed To Post to CNS [https://cns.awmdm.com/nws/notify/apns] Error: [The remote server returned an error: (401) Unauthorized.] Response: [{"code":2007,"message":"Unable to verify if the signer cert as trusted. The associated request id is 154e9542-b695-497b-9896-a8fd9cb13e84."}]
If you see a 401 error while posting a notification and the UEM console is on-premises, then navigate to and select the Download CNS Secure Channel Certificate Installer. You can also open a Zendesk ticket with the category. To install the certificate on the CNS server, send a request to the VMware Support team.
Error: ENS has posted notification to CNS/SNS successfully, but we don't see any notification on the device.
This error occurs due to the APNS or the GCM token issue. To verify the APNS or the GCM tokens, perform the following steps:
- Log in to the Workspace ONE UEM console and navigate to the organization group where the device is enrolled.
- Navigate to the and select the device.
- Click the and select the application as Boxer from the drop-down.
- Enter the Message Body and click SEND. After you click SEND, you must be able to see the notification on the device if the APNS token is correct.
Unregistered ENS Logs
The Boxer application sends an unregister request to the ENS in the following scenarios:
- When a device account is removed from the Boxer application
- When a device is deleted from the Workspace ONE UEM console.
- During an enterprise wipe from the Workspace ONE UEM console.
- Toggle off the push notification button in the Boxer application settings.
Sample of unregistered ENS logs:
2019/11/06 10:33:23.976 A3 2bd0af6a-ba08-479e-a606-b1326281902c [0000000-0000000] (53) Debug MailNotificationService.Controllers.EnsController.Unregister User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] Processing Unregister request. UserId:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] 2019/11/06 10:33:24.054 A3 2bd0af6a-ba08-479e-a606-b1326281902c [0000000-0000000] (55) Debug MailNotificationService.BusinessImpl.UnregisterBusiness.ProcessUnregisterRequestAsync User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] Device Unregistered for user:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] 2019/11/06 10:33:24.054 A3 2bd0af6a-ba08-479e-a606-b1326281902c [0000000-0000000] (55) Debug MailNotificationService.Controllers.EnsController.Unregister User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] Unregister request processed. HttpStatusCode:[OK] ResponseCode:[DeviceUnregistered]
When the ENS receives an unregister request, the ENS processes the request and sends an unsubscribe request to the Exchange and deletes the records from the database. The possible errors and solutions that you might see when you unregister is listed as follows:
Error: 401 error Unauthorized
The following logs are seen when the Boxer application sends an unregister request with a wrong API token. You can confirm the API token comparing the API token logged in the ENS logs and present in the Boxer application logs.
ENS logs: API token : [12341*********fasdf]
Boxer application logs: ensapitoken: 17413********************88c08
Sample of UnAuthorizedRequest log:
2019/11/06 10:38:20.413 KAVINASH-W03 cd790dc0-ca7e-4f3d-b468-3c5181c34063 [0000000-0000000] (31) Warn MailNotificationService.BusinessImpl.ApiKeyRepository.ValidateAsync User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] ApiKey header present [True], Value Empty/Null: [False] API key dictionary has keys:[True] Key: [12341:fasdf] 2019/11/06 10:38:20.424 cd790dc0-ca7e-4f3d-b468-3c5181c34063 [0000000-0000000] (31) Debug MailNotificationService.Controllers.EnsController.Unregister user Id [1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] API token : [12341*********fasdf] 2019/11/06 10:38:20.444 cd790dc0-ca7e-4f3d-b468-3c5181c34063 [0000000-0000000] (31) Warn MailNotificationService.Controllers.EnsController.Unregister User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] Error Code:'23' Error message: 'UnAuthorizedRequest' Stack Trace: at MailNotificationService.Controllers.EnsController.<Unregister>d__21.MoveNext() in C:\Stash\MailNotificationService\Controllers\EnsController.cs:line 926
Badge Update for ENS Logs
Badge update is only supported for iOS devices. The badge notification starts displaying only after the badge receives the first notification from ENS. The badge count is not seen in Boxer immediately after the badge counter is configured and subscribed.
Sample of badge update ENS logs:
2019/11/11 12:27:55.416 A3 04f06dcb-a721-4a90-a2ff-2be8e007f533 [0000000-0000000] (52) Debug MailNotificationService.BusinessImpl.ExchangeNotificationParser.ScanEventNotificationAsync User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] Received [ModifiedEvent] for subscription: [EgBleGNoMjAxMy5tZW0xMy5vcmcQAAAAjoO0qTL7hk2FF7QvXOHC1BLv0sChZtcIEAAAACanmwmX5x5OpwfUW+dfdrQ=]
2019/11/11 12:27:55.525 A3 04f06dcb-a721-4a90-a2ff-2be8e007f533 [0000000-0000000] (52) Info MailNotificationService.BusinessImpl.PushNotificationBusiness.HandleMoveModifiedEventAsync User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] -BADGE UPDATE- [5422] previous BADGE count is [5422] Received modified event for user [20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]
2019/11/11 12:27:55.525 A3 04f06dcb-a721-4a90-a2ff-2be8e007f533 [0000000-0000000] (52) Info MailNotificationService.BusinessImpl.NotificationsProcessor.AddNotificationToBatch User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] About to Post Notification for user : [20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] and Device Id : [9]
2019/11/11 12:28:00.531 A3 d7893c08-3a06-46a1-a8a7-45361572b573 [0000000-0000000] (16) Info MailNotificationService.BusinessImpl.CNSHelper.ComposeAPNSPushNotification User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] Total unread count retrieved [5422] for user [20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586]
2019/11/11 12:28:00.531 A3 d7893c08-3a06-46a1-a8a7-45361572b573 [0000000-0000000] (16) Debug MailNotificationService.BusinessImpl.CNSHelper.ComposeAPNSPushNotification User Id:[20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586] Sending to :: User : [20943ad3f74ef04b3a2394b968cb46cc498f54994bdec0b3520d965e35356586], DeviceId : [9], DeviceLogId : [], Message : messageId: []
2019/11/11 12:28:00.531 A3 d7893c08-3a06-46a1-a8a7-45361572b573 [0000000-0000000] (16) Debug MailNotificationService.BusinessImpl.CNSHelper.CreateWebRequest User Id:[no-user-id] CNS Url : [https://cns.awmdm.com/nws/notify/apns]
2019/11/11 12:28:00.531 A3 d7893c08-3a06-46a1-a8a7-45361572b573 [0000000-0000000] (16) Debug MailNotificationService.BusinessImpl.CertificateHelper.ComputeCmsSignature User Id:[no-user-id] Signing URL [/nws/notify/apns] with Cert [CN=AW Cloud Notification - aTest]
2019/11/11 12:28:00.748 A3 d7893c08-3a06-46a1-a8a7-45361572b573 [0000000-0000000] (16) Debug MailNotificationService.BusinessImpl.CNSHelper.ReadResponse User Id:[no-user-id] Response {"status":"success","errorReason":null}
2019/11/11 12:28:00.748 A3 d7893c08-3a06-46a1-a8a7-45361572b573 [0000000-0000000] (16) Info MailNotificationService.BusinessImpl.CNSHelper.ReadResponse User Id:[no-user-id] ResponseCode OK
Understanding ENS Logs
The ENS logs contain information about registration, subscriptions, notifications, and the CNS or the APNS delivery status. For the on-premises ENS, you can find the ENS2 logs files at: %ENS Installed Directory%\Logs\Email Notification Service. For example, the ENS2 log file can be at: C:\AirWatch\Logs\Email Notification Service. The name of the log file is ENS.log.
Sample ENS2 log file:
2019/11/05 09:39:56.608 A3 9f08ed6d-0726-430c-8440-9c396443c7ca [0000000-0000000] (74) Debug
MailNotificationService.BusinessImpl.ExchangeNotificationParser.ScanEventNotificationAsync
User Id:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] Received [CreatedEvent] for subscription: [JwBtbjJwcjE5bWIzMDA1Lm5hbXByZDE5LnByb2Qub3V0bG9vay5jb20QAAAAl4H5dKboFUm1kJ8ZNBKkJILRTBjMYdcIEAAAAAQ9tcFCKSZFrTOxLbSCwj4=]
The following table provides a description of a sample ENS2 log file.
| Log Format | Value |
|---|---|
| Date | 2019/11/05 09:39:56.608. The date is mentioned in the UTC format. |
| machinename | A3 |
| ActivityId | 9f08ed6d-0726-430c-8440-9c396443c7ca |
| threadid | (74) |
| logLevel | Debug |
| Logger | MailNotificationService.BusinessImpl.ExchangeNotificationParser.ScanEventNotificationAsync |
| Message | UserId:[1743604ea20cda831dc7aea285e7fdc011ca233caf0fa7d5d926916622dd182d] Received [CreatedEvent] for subscription: [JwBtbjJwcjE5bWIzMDA1Lm5hbXByZDE5LnByb2Qub3V0bG9vay5jb20QAAAAl4H5dKboFUm1kJ8ZNBKkJILRTBjMYdcIEAAAAAQ9tcFCKSZFrTOxLbSCwj4=] |
In the logs, you can find the user name or email address in the alphanumeric format and not in the plain text format. For example, the user ID is mentioned as an alphanumeric string such as, 4e9dc715faba719b266fe90f866caf8e377c08984cd1fd005bac72c7eba4db02. This string is a hash value that is calculated from the email address.
You can use the SHA-256 hash calculator to translate any email address to a hash value. You can then use the hash value to search logs for any user.
To obtain the logs for the cloud ENS, you can access ENS2 logs through the LogInsight.
Troubleshooting the ENS2 SEG Errors
This section describes the troubleshooting steps you might have to perform due to communication errors between the ENS2 and Exchange with SEGv2 as the proxy.
The following steps describe the interaction between the ENS2 and Exchange with SEGv2 as the proxy.
- Boxer application requests a public key from the ENS.
- Boxer application encrypts the user credentials using the public key and sends a subscription request to the ENS.
- ENS requests a subscription to the Exchange server using the SEG URL which also contains the encrypted credentials. The ENS also sends a client certificate. If the client certificate is configured on the Boxer application profile, then the authentication received from the Boxer profile is sent. For certificate-based authentication (CBA), when a register device request is sent to the cloud ENS server, the ENS routes the request to the SEG with the certificate information. The SEG follows the same token retrieval process similar to the ActiveSync request.
- SEG forwards the subscription request to the Exchange to complete the subscription. The same authentication method configured in the Boxer application profile is used for subscription. The ENS server callback URL is used to subscribe.
- The Exchange server receives an email.
- The Exchange server notifies the ENS callback URL of the subscriber to inform that a new email has arrived, hence update the email client with the notification. The ENS fetches the details of the email from the SEG.
- The ENS server requests the CNS or SNS to send notification to the Boxer application or the device of the subscriber.
- The CNS or the SNS server contacts the Apple Push Notifications (APNs for iOS devices) or GCM or FCM (for Android devices).
- The APNS or GCM server pushes the email notification to the device.
Using the transaction ID received in the ews-transaction log, you can search the ews-proxy.log. For example, if the transaction ID is 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 then search for 544ef2b7.
When you see 200 in the ENS transaction log, you can confirm if the notifications are going through the CNS communication.
The Ews-transaction log sample.
Time, LogLevel, Thread Id, Message, HTTP-Method, Remote-Host, X-Forwarded-For, SEG TransactionId, Request-DeviceId, EnsDevices, EmailServerResponseStatus, SegResponseStatus, EmailRequestBodySize, EmailResponseBodySize, TimeTakenByKerberosService(ms), TimeTakenBySeg(ms), TimeTakenByEmailServer(ms), BeginningOfRequest 2018-12-03 17:20:15.696, DEBUG, (vert.x-eventloop-thread-0), Responding back to ENS,POST,192.168.2.34,null,544ef2b7-9ca3-4009-b116-8a9f6513f2c7,6C30D0304E7A4EE795494DEB0F465B72,"6C30D0304E7A4EE795494DEB0F465B72:200",200,200,1243,1147,2547,0,16,1543875613133 2018-12-03 17:20:18.274, DEBUG, (vert.x-eventloop-thread-0), Responding back to ENS,POST,192.168.2.34,null,d77ae46b-2d38-46b5-9548-3bcc25a1bf03,6C30D0304E7A4EE795494DEB0F465B72,"6C30D0304E7A4EE795494DEB0F465B72:200",200,200,673,1806,2547,0,16,1543875615711 2018-12-03 17:20:41.430, DEBUG, (vert.x-eventloop-thread-0), Responding back to ENS,POST,192.168.2.34,null,b639bdee-0cfa-42b5-82ea-0629ab1d586a,6C30D0304E7A4EE795494DEB0F465B72,"6C30D0304E7A4EE795494DEB0F465B72:200",200,200,1632,2464,2562,0,47,1543875638821 2018-12-03 17:21:16.462, DEBUG, (vert.x-eventloop-thread-1), Responding back to ENS,POST,192.168.2.34,null,ed0db6c1-9dd4-420a-83d3-e746cb17445c,82B15D853CC14CA3989020257158BFC1,"82B15D853CC14CA3989020257158BFC1:200",200,200,1632,3028,2563,0,47,1543875673852 2018-12-03 17:21:26.493, DEBUG, (vert.x-eventloop-thread-1), Responding back to ENS,POST,192.168.2.34,null,425fc495-4ae1-4c26-abc5-c30f34a376cf,82B15D853CC14CA3989020257158BFC1,"82B15D853CC14CA3989020257158BFC1:200",200,200,673,1815,2547,16,15,1543875683915 2018-12-03 17:22:46.649, DEBUG, (vert.x-eventloop-thread-1), Responding back to ENS,POST,192.168.2.34,null,ba0b13ad-b341-43e3-a4a9-d1a79c5330e0,82B15D853CC14CA3989020257158BFC1,"82B15D853CC14CA3989020257158BFC1:200",200,200,1632,3028,2547,15,32,1543875764055 2018-12-03 17:23:01.649, DEBUG, (vert.x-eventloop-thread-1), Responding back to ENS,POST,192.168.2.34,null,262cc2b2-8ae4-4ea7-b062-da2b2eb42a68,82B15D853CC14CA3989020257158BFC1,"82B15D853CC14CA3989020257158BFC1:200",200,200,673,1815,2547,0,15,1543875779087 2018-12-03 17:26:47.353, DEBUG, (vert.x-eventloop-thread-3), Responding back to ENS,POST,192.168.2.34,null,c7e5a6c9-b1b0-4739-9132-49470306882c,6C30D0304E7A4EE795494DEB0F465B72,"6C30D0304E7A4EE795494DEB0F465B72:200",200,200,673,1806,2547,0,94,1543876004712 2018-12-03 17:26:51.884, DEBUG, (vert.x-eventloop-thread-3), Responding back to ENS,POST,192.168.2.34,null,d5cf2470-d818-45f6-ab0e-dd68599d4aa8,6C30D0304E7A4EE795494DEB0F465B72,"6C30D0304E7A4EE795494DEB0F465B72:200",200,200,673,1806,2547,0,15,1543876009322 2018-12-03 22:06:55.421, DEBUG, (vert.x-eventloop-thread-2), Responding back to ENS,POST,192.168.2.34,null,93f7f097-bda5-417a-ac67-5667b4088c84,6C30D0304E7A4EE795494DEB0F465B72,"6C30D0304E7A4EE795494DEB0F465B72:200",200,200,673,1806,12000,16,234,1543892803171 2018-12-03 22:07:00.031, DEBUG, (vert.x-eventloop-thread-0), Responding back to ENS,POST,192.168.2.34,null,d10c08a4-49cd-4240-bcc0-ba9bb81f74f0,82B15D853CC14CA3989020257158BFC1,"82B15D853CC14CA3989020257158BFC1:200",200,200,673,1815,11969,0,188,1543892807874 2018-12-04 10:31:33.786, DEBUG, (vert.x-eventloop-thread-2), Responding back to ENS,POST,192.168.2.34,null,3844719b-73c6-4b77-91d8-8a7d8b9a97c0,82B15D853CC14CA3989020257158BFC1,"82B15D853CC14CA3989020257158BFC1:200",200,200,1632,3028,2563,15,516,1543937490692
The Ews-transaction log sample filtered using the 544ef2b7.
2018-12-03 17:20:13.133 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsRequestReadHandler] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - Incoming EWS request, Path: /EWS/Exchange.asmx. Headers are 2018-12-03 17:20:13.133 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsHelper] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - Collected ENS devices: [6C30D0304E7A4EE795494DEB0F465B72] 2018-12-03 17:20:13.133 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsRequestReadHandler] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - Getting device policy for request device 6C30D0304E7A4EE795494DEB0F465B72 2018-12-03 17:20:13.133 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsComplianceCheckHandler] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - Device list: [6C30D0304E7A4EE795494DEB0F465B72] 2018-12-03 17:20:13.133 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsComplianceCheckHandler] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - Checking compliance for device 6C30D0304E7A4EE795494DEB0F465B72 2018-12-03 17:20:13.133 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsComplianceCheckHandler] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - Device 6C30D0304E7A4EE795494DEB0F465B72 is compliant 2018-12-03 17:20:13.133 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsRequestProxyHandler] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 KCD authentication is (true), upn is TUSER1@MILKYWAY.LOCAL. 2018-12-03 17:20:15.680 DEBUG (pool-7-thread-5) [c.a.s.e.h.EwsRequestProxyHandler] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - Successfully got kerberos token for UPN TUSER1@MILKYWAY.LOCAL - token length 2024 2018-12-03 17:20:15.680 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsRequestProxyHandler] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - Proxying request to EWS 2018-12-03 17:20:15.680 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsRequestProxyHandler] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - EWS client request headers: 2018-12-03 17:20:15.696 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsRequestProxyHandler] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - EWS client response headers: 2018-12-03 17:20:15.696 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsHelper] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - Response headers from SEG to ENS: X-AW-SEG-TRANSACTION-ID : 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 2018-12-03 17:20:15.696 DEBUG (vert.x-eventloop-thread-0) [c.a.s.e.h.EwsRequestProxyHandler] - 544ef2b7-9ca3-4009-b116-8a9f6513f2c7 - EWS response status 200, length 1147
The possible errors and solutions you might see during an interaction between the ENS2 and Exchange with SEGv2 as the proxy is listed as follows:
Error: 404 / https://[segURL]/EWS/Exchange.asmx is not found
If you see this error in the ENS logs, then ensure you have enabled the EWS proxy in the SEG server. If you have not enabled the EWS proxy in the SEG server then perform the following steps.
- Navigate to the folder using the File explorer.
- Select the application.properties file and edit the file.
- Select the enable.boxer.ens.ews.proxy value and update the value to enable.boxer.ens.ews.proxy=true.
- Save the file.
- Restart the VMware AirWatch Secure Email Gateway service.
Sample of the application.properties file.
####################################################################################################################################### ############################# Start - HTTP endpoint path for SEG active-sync, syncML and REST API. ################################## ####################################################################################################################################### # SEG HTTP server context path. This should be same as the context path of Email/Exchange server as Device won't know # if it's sending request to email server or SEG Proxy. This value generally don't change but we want to give # the ability to the Admin to change it, if needed in some exceptional cases. # Right now Vertx doesn't support "ignore-case" on path, and also doesn't allow mounting sub-routers on RegEx. # For now we're trying to avoid using RegEx anyway - https://groups.google.com/forum/#!topic/vertx/ck95b4juj4A activesync.context.paths=/Microsoft-Server-ActiveSync,/microsoft-server-activesync# Context path when SEG works as EWS proxy for ENS. EWS endpoint will be deactivated by default. enable.boxer.ens.ews.proxy=true ews.proxy.context.paths=/EWS,/ews # Flag used to remove unsupported www-authenticate header such as NTLM and Negotiate (in absense of certificate) from EWS response to ENS. remove.unsupported.auth.for.ews=true
Error: 401 - Please check the authentication type enabled in exchange (EWS endpoint)
If you see this error in the ENS logs, then the SEGv2 does not support the NTLM authentication. If both the Basic and NTLM authentication mechanisms are enabled for the EWS endpoint, then the SEGv2 version prior to version 2.9.0.1 cannot prefer Basic authentication over the unsupported NTLM authentication.
This results in the ENS attempting the NTLM-based authentication for requests through the SEG, that eventually causes 401 error responses as observed in the ews-transaction.log. If the user is unable to deactivate the NTLM authentication mechanism for the EWS endpoint, and is using any lower version of the SEG, then setup the KCD authentication for the ENS-SEG integration to work correctly.
If you connect directly to the EWS endpoint on the SEGv2 proxy through the https://[segURL]/EWS/Exchange.asmx URL, you might receive a 400 error message unless you connect using a permitted device.
Error: The request was aborted: Could not create SSL/TLS secure channel
In the ENS logs, if you see the following error during the registration process, then the error might be due to a cipher mismatch.
2019-12-05 15:33:40.5081|DEBUG|MailNotificationService.BusinessImpl.ExchangeRetriesHandler.SubscribeForNotificationsAsync|3ed2219d-42f2-4a2a-b857-ab7639ad1858|User Id:[af03aa8bb3cae692442ec673b207fbe5666e0762bf3ca62cbaaa61c4208cd7bd] EWSUrl used to subscribe: [https://uag.testdomain.com/ews/exchange.asmx] 2019-12-05 15:33:40.5550|WARN|MailNotificationService.BusinessImpl.SubscriptionBusiness.SubscribeV2Async|3ed2219d-42f2-4a2a-b857-ab7639ad1858|User Id:[af03aa8bb3cae692442ec673b207fbe5666e0762bf3ca62cbaaa61c4208cd7bd] Service request exception occured for userId [af03aa8bb3cae692442ec673b207fbe5666e0762bf3ca62cbaaa61c4208cd7bd], Inner exception message [The request was aborted: Could not create SSL/TLS secure channel.].
To fix the cipher mismatch error, perform the following steps:
- Run a TCP dump on the UAG or SEG. Check the reason for the handshake failure, using the following commands. See the Troubleshooting Firewall and Connection Issues section in the Deploying and Configuring VMware Unified Access Gateway guide.
/etc/vmware/gss-support/install.sh
tcpdump -i any -n -v tcp port any -w /tmp/vmware/capture.pcap
- Open the TCP dump logs using the Wireshark or any supported application. Filter the logs based on the IP source and IP destination and check for the client hello request as shown in the following log.
Use the tls.alert_message.level filter to search for the SSL error or alert in the Wireshark. Identify the source and destination IP, right click, and select .
- Right click and open the Client Hello information.
- Click the . You can see a list of cipher suites that the client ENS is sending to initiate a secure communication as shown in the following image.
- Ensure that the UAG or the SEG server has enabled the ciphers listed in the Client hello Request.
Note: To check for the enabled cipher suites in the UAG or the SEG server, you can use the SSL report. Enter your SEG or UAG URL and wait for the test to complete. When the test is complete, you might see the following result.
The following table lists all the response codes and messages in the SEG logs.
| Response Code | Message | Description |
|---|---|---|
| 204 | No Content | Indicates that the policy data is not loaded in the SEG to run the compliance check on the requesting devices. |
| 403 | Forbidden | Indicates that none of the devices listed in the ENS request headers are compliant. |
| 400 | Bad Request | Indicates that none of the devices listed in the ENS request header are found in the SEG device policy cache. |
| 5xx | Indicates the server errors. |
Troubleshooting Connection Issues to the ENS Database
When installing ENS, use the SQL authentication and not the Windows authentication to access the ENS database. This topic is applicable for ENS on-premise installation only.
Problem: In case you connect to the ENS database using the Windows authentication then you might receive the following error:
2018/11/05 19:55:40.800 EUROPA 80000005-0001-ff00-b63f-84710c7967bb [0000000-0000000] (35) Error MailNotificationService.ProviderImpl.ApiTokensDataHandler.ApiTokensAsync User Id:[ ] Error While loading the api tokens Exception [Cannot open database "ENS" requested by the login. The login failed. Login failed for user 'NT AUTHORITY\LOCAL SERVICE'.] StackTrace[ at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, DbConnectionPool pool, String accessToken, Boolean applyTransientFaultHandling)
Cause:
Connecting to the ENS database using the Windows authentication might cause this issue.
Use the SQL authentication to connect to your ENS database. In the solutions steps provided, the NT AUTHORITY\LOCAL SERVICE is the name of the user account and the database role membership for the NT AUTHORITY\LOCAL SERVICE account must have the db_owner and public enabled.
To add an SQL account to the ENS database, perform the following steps:
- Open the SQL Server Management Studio.
- Navigate to and add
NT AUTHORITY\LOCAL SERVICE. - Navigate to
- Select the ENS database and add the required permissions.
Troubleshooting SSL Errors
Use the SSL pinning certificate tool when the notifications are not delivered to the devices. This tool is only used for troubleshooting purpose and is not a mandatory step during installation. The following error message appears in the ENS logs while posting the notifications to CNS: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
Before you begin:
Download the latest certificate from the CNS Public Certificate.
The following procedure describes the steps to upload the SSL pinning certificate to the ENS.
- Click the SSLPinningCertTool shortcut on the ENS server, or navigate to the <ENS_INSTALL_DIR>\Email Notification Service\Tools\SSLPinningCertTool\SSLPinningCertTool.exe file.
- Click the Upload CNS Certificate button.
- Select the certificate to be uploaded and click Submit.
Results: If the following screen appears, then the certificate is successfully added to the ENS configuration file. Click OK to continue. After uploading the SSL pinning certificate on the ENS, the tool adds the public key of the certificate to the ENS configuration file. When the ENS posts payload to the CNS, the certificate validation is done against the newly added certificate public key.
If the following screen appears, then the certificate is added successfully to the resubscription configuration file. After uploading the SSL pinning certificate, the tool adds the public key of the certificate to the resubscription configuration file. When the resubscription mechanism posts payload to the CNS, the certificate validation is done against the newly added certificate public key.
If the certificate is already present in both the configuration files, then the following prompt message appears:
The upload pinning certificate process works as follows:
- The SSL pinning certificate tool tries to upload the certificate to the ENS configuration file only if the provided certificate is not present in the ENS configuration file. If the given certificate is already present, then the tool does not prompt any message and continues to upload the same certificate to the resubscription configuration file.
- The SSL pinning certificate tool tries to add the certificate to the resubscription configuration file only if the provided certificate is not present in the resubscription configuration file. If the given certificate is present, then the tool does not prompt any message to user.
- If the certificate is added to the resubscription configuration file, then restart the AirWatch Resubscription Mechanism service in the Services tab.
Troubleshooting AirWatch AutoDiscovery Checker Service Errors
ENS2 depends on the CNS service to deliver email notifications to the devices. If the existing CNS certificate expires then the CNS certificate rotation occurs. The autodiscovery checker service gets the latest certificate and adds the certificate in the ENS server. If the autodiscovery checker service fails to fetch the CNS certificate and add the certificate in the ENS, then ENS cannot send the notifications to devices and receives an SSL error. Perform the following steps to troubleshoot the Airwatch AutoDiscovery Checker service error.
- Ensure that the Airwatch AutoDiscovery Checker service is running correctly.
- Review the logs for the service at
\{ENS installation directory}\Email Notification Service\Servicesand ensure you are able to see the following log statements without errors: New Certificate Added Successfully.
- Review the file at
\{ENS installation directory}\Email Notification Service\Website\web.configand ensure that at least 8 pinned certificate elements are listed under the<pinnedCertificates>section.
Troubleshooting Installer Error
The following VMware AirWatch root certificate error might occur if the installer is unable to install the VMware AirWatch root certificate. To resolve this problem, ensure that the installer has the appropriate privileges to install the certificate on the server.
Troubleshooting AutoDiscoveryChecker.log File Errors
The following list describes the possible errors for the AutoDiscoveryChecker.log file:
- Error while searching for public key in an existing config file
- Error occurred while updating the config file
- Exception while obtaining the latest certificates from auto discovery
The following errors might be displayed for the AutoDiscoveryChecker.log file:
- If the following URL is not reachable
https://awtrustdiscovery.awmdm.com/autodiscovery/HostRegistry.aws?URL=cns.awmdm.com - If the error is a result of a temporary network failure, the service must attempt to connect to the endpoint again after 24 hours.
- If the ENS server is configured behind a reverse proxy, or if the outgoing traffic is going through a proxy then the auto discovery service does not go through the proxy and the firewall rules must be updated to allow the IP address
192.30.68.111for the ENS auto discovery service to be able to reach the following URLhttp://awtrustdiscovery.awmdm.com/autodiscovery/HostRegistry.aws?URL=cns.awmdm.com.
ENS2 Response Code and Error Code Details
If the Boxer application sends a request to the ENS, the ENS processes the request and sends a response with a response code and message to the Boxer application. The following table lists all the response codes and messages in the Boxer application logs and the ENS logs.
| Response Code | Message | Description |
|---|---|---|
| 14 | SubscribeAgain | If a subscription failed, then the ENS sends a subscribe again message to the Boxer application. |
| 8 | ErrorSubscribeOrUpdateDb |
When you try to add a user or device details to the database during subscription, you might receive this error . |
| 23 | UnAuthorizedRequest |
Authentication failed (API token mismatch) for the request from the Boxer application. |
| 32 | Failed (Handled Exception) | Registration failed with a handled exception. For example, the URL is not in the correct format. |
| 17 | Success |
Indicates that the registration is successful. |
| 3 | UpdateSuccess | The Boxer application receives this response when the:
|
| 4 | UpdateFail | ENS sends this response for multiple reasons. When you receive this response, verify the corresponding ENS logs and troubleshoot based on the message in the logs. |
| 5 | TokenDoesNotExists |
ENS sends this response when the device record is absent. When you send a force register, (by changing the notification sound in the Boxer application setting) a new device is created on the ENS server. |
| 6 | UserAlreadySubscribed |
ENS sends this response when a user is already subscribed on the ENS server. |
| 7 | UserSubscribed |
User subscription is successful. |
| 9 | NoRecordExists |
ENS sends this response when a user record is absent. When you send a force register, (by changing the notification sound in the Boxer application setting) a new user is created on the ENS server. |
| 15 | UserSubscribedNotUpdatedInDb |
User subscribed but failed to add device details in the database. In this case, ensure that the connection from the ENS to the database is working correctly or the user has permission to update the database. |
| 16 | FailedToGetEwsUrlFromAutoDiscovery |
Unable to determine the Exchange version after autodiscovery. |
| 21 | EmailFetchfailed |
Fetching email information from the EWS failed. |
| 24 | DeviceUnregisteredUserUnsubscribed | Unsubscribe successful and the user is unregistered. |
| 25 | DeviceUnregistered |
The device is deleted from the database. |
| 26 | DeviceNotRegistered |
The device is not registered. |
| 28 | UserSubscriptionNotFound |
User record does not exist. |
| 29 | UserRecordPresentNotSubscribed |
User records are present but not subscribed. |
| 30 | SubscribedNeedsUpdate |
User has already subscribed and must be added to the database. |
| 34 | InvalidDecryptedPayload |
Payload is encrypted with a wrong public key. |
| 35 | EWSUrlMismatch |
Unsubscribing with the wrong EWS URL. The EWSUrl for the register request and Exchange service is not the same. |
| 36 | InvalidAuthType |
Indicates the invalid authentication type. |
Troubleshooting the CNS Errors
Error: Keyset does not exist
In the ENS logs, while posting the notifications to CNS if you see the following error then it could be due to the permission issue.
2022/06/29 07:12:22.766 ENSUPGRADE 0f72abe9-d91d-4544-9608-02f38a8613a0 [000000 -0000000] (9) Error MailNotificationService.JwtTokenGenerator.JwtTokenGenerator.GenerateCnsJwtToken User Id:[ ] Exception occurred while generating jwt token [System.Security.Cryptography.CryptographicException: Keyset does not exist2022/06/29 07:12:22.766 ENSUPGRADE 0f72abe9-d91d-4544-9608-02f38a8613a0 [000000 -0000000] (9) Error MailNotificationService.JwtTokenGenerator.JwtTokenGenerator.GenerateCnsJwtToken User Id:[ ] Exception occurred while generating jwt token [System.Security.Cryptography.CryptographicException: Keyset does not exist at System.Security.Cryptography.Utils.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer) at System.Security.Cryptography.Utils.GetKeyPairHelper(CspAlgorithmType keyType, CspParameters parameters, Boolean randomKeyContainer, Int32 dwKeySize, SafeProvHandle& safeProvHandle, SafeKeyHandle& safeKeyHandle) at System.Security.Cryptography.RSACryptoServiceProvider.GetKeyPair() at System.Security.Cryptography.RSACryptoServiceProvider..ctor(Int32 dwKeySize, CspParameters parameters, Boolean useDefaultKeySize) at System.Security.Cryptography.X509Certificates.X509Certificate2.get_PrivateKey() at System.Security.Cryptography.X509Certificates.RSACertificateExtensions.GetRSAPrivateKey(X509Certificate2 certificate) at Microsoft.IdentityModel.Tokens.X509SecurityKey.get_PrivateKey() at Microsoft.IdentityModel.Tokens.X509SecurityKey.get_PrivateKeyStatus() at Microsoft.IdentityModel.Tokens.AsymmetricSignatureProvider..ctor(SecurityKey key, String algorithm, Boolean willCreateSignatures) at Microsoft.IdentityModel.Tokens.CryptoProviderFactory.CreateSignatureProvider(SecurityKey key, String algorithm, Boolean willCreateSignatures, Boolean cacheProvider) at Microsoft.IdentityModel.Tokens.CryptoProviderFactory.CreateForSigning(SecurityKey key, String algorithm, Boolean cacheProvider) at Microsoft.IdentityModel.JsonWebTokens.JwtTokenUtilities.CreateEncodedSignature(String input, SigningCredentials signingCredentials) at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.CreateJwtSecurityTokenPrivate(String issuer, String audience, ClaimsIdentity subject, Nullable`1 notBefore, Nullable`1 expires, Nullable`1 issuedAt, SigningCredentials signingCredentials, EncryptingCredentials encryptingCredentials, IDictionary`2 claimCollection, String tokenType) at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.CreateJwtSecurityToken(SecurityTokenDescriptor tokenDescriptor) at MailNotificationService.JwtTokenGenerator.JwtTokenGenerator.GenerateCnsJwtToken()]
Depending on the type of SQL login, perform the following steps to resolve the permission issue:
Login Type: SQL Server Login (user/password)
- Enter the SQL server login username and password.
- If the customer is using SQL user to communicate with the database, then by default ENS application pool identity account is set to NetworkService.
- Change the application pool identity from NetworkService to LocalSystem to resolve the issue as the local system has higher privilege to access the system level resources.
Login Type: Windows user or Active Directory (AD) user: If the customer is using AD user to communicate with the database, then by default ENS application pool identity account is set to AD user. To resolve the issue, add the AD user as a member of the Domain Admins and restart the ENS machine.
In the following example, BSB02\avinash is an AD user and the identity is set to the AD user itself.
Ensure that the AD user is a member of the Domain admins in the .
Enable Trace Level Logging for Enhanced Debugging
To help debug and troubleshoot rare issues, ENS2 version 1.5 or later supports trace level logging. To enable trace level logging, you must upgrade to ENS 1.5 or later and perform the following steps to change the log level to ActivityTrace.
- Navigate to <INSTALL_FOLDER>\Email Notification Service\WebSite\Web.config path.
- Change the value of <loggingConfiguration level="Verbose" /> to <loggingConfiguration level="ActivityTrace" />
- Navigate to <INSTALL_FOLDER>\Email Notification Service\Services\ReSubscriptionMechanism.exe.config path.
- Change the value of <loggingConfiguration level="Verbose" /> to <loggingConfiguration level="ActivityTrace" />
Result
You can view the trace level logs in the log files within 30 minutes to an hour.
To mask the PII information, you must manually replace the text with *. For more information, see the following example.
Original trace log without masking:
Trace logs after masking, that is, manually replacing text with *.
Deactivate Trace Logging
You must deactivate trace logging after you enable trace logging and debug the issue.
To deactivate trace logging, change the loggingConfiguration level value from ActivityTrace to Verbose in all the configuration files where the changes were previously made.
