Tanzu Data Hub Documentation

Tanzu Data Hub (TDH) is a one-stop solution for all your data services needs on a Kubernetes platform.

TDH diagram

Using Tanzu Data Hub, developers, platform operators, and others create, manage, and use database services like PostgreSQL, RabbitMQ, MySQL, governed by centralized role-based access control. SRE users perform bulk upgrades of databases and other fleet management operations across organizations.

Features

Tanzu Data Hub features include:

  • Data Service LCM: Full Life Cycle Management (LCM) operations for RabbitMQ, PostgreSQL, MySQL and Redis instances.
    • Create, delete, update, backup and restore, monitoring, and CLI-based integration on a Kubernetes cluster.
  • Kubernetes:
    • Runs on Kubernetes clusters, making it easy to install and manage.
    • CNCF-compliant.
    • Supported platforms: Tanzu Kubernetes Grid service(TKGs),Tanzu Kubernetes Grid multi-cloud(TKGm), OpenShift, Tanzu Application Service (Beta).
  • Fleet Management:
    • Custom fleet management dashboards give a birds-eye view of all service, data plane and resources consumption.
    • Bulk updates capabilities for all services and data dataplane from one view
  • IDP Federation:
    • Supports federation with external identity providers via LDAP.
  • IAM and Policy Management:
    • Complete user management backed by OAuth2.0 integration.
    • Policy-based access control for both TDH and the data services it manages.
  • Monitoring:
    • Grafana and Prometheus-based monitoring, with extensive real-time data views for all data services.
  • Log Aggregation:
    • Built-in log aggregation via Elasticsearch with Kibana dashboard.
  • Backup Storage:
    • Built-in SeaweedFS to store backups of provisioned services
    • Additionally, supports external backup locations; An AWS S3 bucket can be configured to store PostgreSQL and MySQL backups
  • RBAC:
    • Role-based access control for SRE (platform engineer), Admin, Developer, Viewer, Operator, and other personas.
  • Auditing:
    • Built-in audit capabilities for key activities, accessible and exportable via dashboards.
  • CLI and API:
    • Complete CLI, API, and Swagger options for using Tanzu Data Hub programmatically or from a command-line.
  • TAS integration (Beta):
    • Integration with Tanzu Application Service for deploying and performing fleet management operations on PostgreSQL instances on TAS.

Getting Started

The high-level steps for installing and using Tanzu Data Hub are:

  1. An SRE/Platform Engineer user installs Tanzu Data Hub as described in Install Tanzu Data Hub.

  2. The SRE user logs in to Tanzu Data Hub and uses it to create one or more data planes and invite their first admin users as described in Create a Data Plane.

  3. Users are given access to Tanzu Data Hub and the database services that it manages in two ways:

    • Admins create identities manually as described in Create an Identity.
    • SREs federate identities in bulk from external identity providers via LDAP as described in Federation.

  4. Depending on their roles, users do one or more of:

    • Log in to Tanzu Data Hub to manage users and create, manage, and monitor database service instances.
    • Use the service instances.

Port Details for TDH OnPrem

TDH diagram

Following are the ports which are being consumed by TDH and its service offerings:

Plane Name Externally Exposed Ports Internally Used Ports Comments / Details to Fetch IPs
Ingress Control Plane 5671,8883,6432,9999,5432,15691,6379,26379,5551,443, (3306 to 5307), 8888, 18888, 9999 - To get the IP for Ingress of control plane perform the following command kubectl get svc traefik -n mds-infra
Ingress Data Plane 5671,8883,6432,9999,5432,15691,6379,26379,5551,443,(3306 to 5307), 6379, 9121, 8888, 18888, 9999 - To get the IP for Ingress of control plane perform the following command kubectl get svc traefik -n tdh-dp-ops
TDH Managed DNS Server 53 - IPs for the DNS Server can be found from the SRE View -> Infrastructure -> DNS Page as shown in figure TDH diagram
Control Plane & Services - 8080, 18333, 19333, 15671, 25672, 9200, 9300, 9880, 5601, 7979, 9327, 8888, 18888, 19333, 5601
Data Plane & Services - 8080, 18333, 19333, 15671, 25672, 9200, 9300, 9880, 5601, 7979, 9327, 8888, 18888, 19333, 5601

Concepts and Components

Main concepts and components underlying Tanzu Data Hub include:

  • Backups: Tanzu Data Hub supports backups via built-in SeaweedFS deployment that runs in each data plane or configure an AWS S3 bucket as backup location for each data plane.

  • Connection URL: The URL for a database instance that Tanzu Data Hub generates when it creates the instance; see Domains.

  • Control Plane: A Kubernetes cluster used by SRE users to create and manage data planes, monitor service usage and do fleet management across data planes, see what other users see, and perform other operations. SREs run control plane operations by logging in to Tanzu Data Hub GUI, the tdh-cli CLI, or an APIThe control plane runs on a Kubernetes cluster and can be accessed by a browser-based GUI, the tdh-cli CLI, or an API.

  • Data Plane: A Kubernetes cluster that hosts database services that are self-managed via Tanzu Data Hub. Installing TDH bootstraps components and operators onto this cluster, to run TDH services. Users log in to Tanzu Data Hub to provision and manage database instances and their users via a browser-based GUI, the tdh-cli CLI, or an API.

  • Dedicated Data Plane: A data plane cluster that hosts service instances which can only be used by members of a specific organization. Dedicated and Shared are the two hosting types for data planes.

  • Domains: Addresses for the Tanzu Data Hub GUI, service instance endpoints, monitoring dashboards, and other interfaces derive from a base wildcard domain configured under Certificate > Domain Name when Tanzu Data Hub is installed. For example, if the base domain is tdh.example.com, then:

    • Users would log in to Tanzu Data Hub at https://tdh-cp.tdh.example.com
    • Deployed database instance management and monitoring dashboards publish at generated URLs https://tdh.example.com.
    • The Connection URL for accessing a database instance would be at a generated URL under https://tdh.example.com that embeds the user’s username and password.

  • Hosting Type: See Shared Data Plane and Dedicated Data Plane.

  • Identity: A user account in that grants role-based access to some or all Tanzu Data Hub interfaces, such as the Tanzu Data Hub GUI, its service instances, and monitoring dashboards. The Tanzu Data Hub control plane keeps identities its own identity provider that can be federated with others for login via SSO.

    • Identities include SRE, User, Local User, and Service Account.

    • User identities have Roles and Tanzu Data Hub Policies that include Admin, Developer, Operator, Viewer, and Compliance Manager.

      • Roles govern access to capabilities within Tanzu Data Hub itself.
      • Tanzu Data Hub Policies (optional) govern access to the database service instances deployed by Tanzu Data Hub.

  • Monitoring: Tanzu Data Hub uses its native log collector to collect logs across the nodes and has embeds Grafana to let users access monitoring dashboards.

    • A link to the Monitoring Dashboard for a database service, with a selector to monitor each service instance across namespaces, appears on service instance’s details pane, under Instances:

    MonitoringDashboardRMQ

  • Auditing: Built-in audit system to track all admin and user operations on its control and data planes

  • Networking: Tanzu Data Hub supports off-platform networking via NSX.

  • Organization: A named group of users, for example Engineering or HR, who can access database services hosted on a specific set of data plane clusters. Organization members can create and access database instances hosted on data planes dedicated to their organization, or on shared data planes. Organizations are created when new users log in to Tanzu Data Hub with Create Organization User Account and enter an Organization name.

  • Provider: Cloud infrastructure that Tanzu Data Hub runs on. Currently provider is Tanzu Kubernetes Grid service on vSphere, Tanzu Kubernetes Grid multi-cloud(TKGm), OpenShift and TAS but others are planned.

  • Shared Data Plane: A data plane cluster that hosts service instances that can be used by members of all organizations. Dedicated and Shared are the two hosting types for data planes.

  • Service Manager: The management console for a service instance, linked from the Instances pane’s Overview in Tanzu Data Hub.

    RMQManagementConsole

  • System Log Details: The logs of Tanzu Data Hub internal system events

    SystemLog

check-circle-line exclamation-circle-line close-line
Scroll to top icon