VMware Telco Cloud Service Assurance user interface displays an error message

After successful login, intermittently VMware Telco Cloud Service Assurance user interface displays “Internal Server Error” message.

Root Cause

Check in keycloak logs: Caused by: org.postgresql.util.PSQLException: ERROR: cannot execute INSERT in a read-only transaction.

There was a switchover in Postgress from primary to secondary (read-only), because the monitor marked the Postgress pod unhealthy. As the Postgress connections are stateful in nature (once established they are not terminated until either of the parties close the connection). As the session termination did not happened, and the keycloak service established a connection. And, later because of the switchover it became read-only and displays an error message.

Solution

1. Stop reconciliation of Keycloak application, and edit Keycloak application using command:

   kubectl edit app keycloak

   Add attribute paused: true in the specification section as shown below:

   spec:
     deploy:
     - kapp:
         delete:
           rawOptions:
           - --apply-ignored=true
         rawOptions:
         - --diff-changes=true
     fetch:
     - imgpkgBundle:
         image: 10.225.67.173:30001/tcx/keycloak:latest
         secretRef:
           name: kubelet-pull-secret
     paused: true
     serviceAccountName: cluster-admin-sa
     syncPeriod: 1m0s
     template:
     - helmTemplate:
         path: keycloak
         valuesFrom:
         - path: keycloak/values.yaml
     - kbld:
         paths:
         - '-'
         - .imgpkg/images.yml
   status:
   

2. Edit Keycloak configmap using command:

   kubectl edit configmap keycloak-configmap-env-vars

   Add attribute KEYCLOAK_JDBC_PARAMS: target_session_attrs=read-write, as shown below:

   apiVersion: v1
   data:
     KEYCLOAK_AUTH_CACHE_OWNERS_COUNT: "3"
     KEYCLOAK_CACHE_OWNERS_COUNT: "3"
     KEYCLOAK_DATABASE_HOST: postgres.default.svc.cluster.local
     KEYCLOAK_DATABASE_NAME: keycloak
     KEYCLOAK_DATABASE_PORT: "5432"
     KEYCLOAK_ENABLE_STATISTICS: "true"
     KEYCLOAK_HTTP_PORT: "8080"
     KEYCLOAK_JDBC_PARAMS: target_session_attrs=read-write
     KEYCLOAK_JGROUPS_DISCOVERY_PROPERTIES: datasource_jndi_name=>"java:jboss/datasources/KeycloakDS",initialize_sql=>"CREATE
       TABLE IF NOT EXISTS JGROUPSPING (own_addr varchar(200) NOT NULL, cluster_name
       varchar(200) NOT NULL, created TIMESTAMP DEFAULT CURRENT_TIMESTAMP, ping_data
       BYTEA, constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name))",remove_all_data_on_view_change=>true
     KEYCLOAK_JGROUPS_DISCOVERY_PROTOCOL: org.jgroups.protocols.JDBC_PING
     KEYCLOAK_JGROUPS_TRANSPORT_STACK: tcp
     KEYCLOAK_PROXY_ADDRESS_FORWARDING: "true"
   kind: ConfigMap
   

3. Restart Keycloak pods, using command:

   kubectl delete pods keycloak-0 keycloak-1 keycloak-2