Tanzu Greenplum Text enables processing mass quantities of raw text data (such as social media feeds or e-mail databases) into mission-critical information that guides business and project decisions. Tanzu Greenplum Text joins the Greenplum Database massively parallel-processing database server with Apache SolrCloud enterprise search. Tanzu Greenplum Text includes powerful text search as well as support for text analysis. Tanzu Greenplum Text supports business decision making by offering:
This chapter contains the following topics:
Tanzu Greenplum Text combines a Greenplum Database cluster with an Apache SolrCloud cluster. Greenplum Database segments and Tanzu Greenplum Text nodes can be deployed on the same hosts or on different hosts with network connectivity.
The following figure shows the process architecture of the combined Greenplum Database and ApacheSolr clusters. The figure shows four cluster nodes with four Greenplum segments and four Solr instances deployed on each. An Apache ZooKeeper service manages the SolrCloud cluster. ZooKeeper nodes are deployed on three of the four hosts. Greenplum Database users access SolrCloud services via Tanzu Greenplum Text user-defined functions installed in Greenplum databases and command-line utilities.
The figure omits the Greenplum master host, secondary master, and mirror segments for the Greenplum primary segments.
The Greenplum segments, Solr instances, and ZooKeeper nodes may all be deployed on separate hosts on the same network, depending on application and performance requirements.
The following sections describe how Tanzu Greenplum Text integrates SolrCloud with Greenplum Database and how the two clusters work together to provide parallel text search capabilities in Greenplum Database and maintain high availability.
A Greenplum Database cluster is comprised of the following components:
mdw. (Not illustrated)
smdw, acting as a warm standby for the master instance. (Not illustrated)
sdwn. A segment instance is an independent Postgres database server managing a portion of the distributed data. Each segment has a mirror (not illustrated) on another host in the cluster to provide uninterrupted service in case of a segment or segment host failure. The number of primary segments per host is determined by the hardware configuration—the number and type of processor cores, the amount of physical RAM, local storage capacity, and network capacity—as well as availability and performance requirements.
The Greenplum Database master instance, which stores no user data, coordinates the work of the segment instances. Database users log in to the master instance and submit SQL queries. The master instance creates a plan for executing the query, distributes the work to the segments, and gathers and returns the results to the user.
Apache Solr is a server providing access to Apache Lucene full-text indexes. Apache SolrCloud is a highly available, fault tolerant cluster of Apache Solr servers. The term Tanzu Greenplum Text cluster is another way to refer to a SolrCloud cluster deployed by Tanzu Greenplum Text for use with a Greenplum Database system.
A SolrCloud cluster is comprised of the following components:
Tanzu Greenplum Text provides document indexing and search capabilities for Greenplum Database with user-defined functions (UDFs) that access Solr APIs from within database queries.
Tanzu Greenplum Text UDFs perform the following tasks:
There are also Tanzu Greenplum Text UDFs and command-line utilities to configure, monitor, and manage the SolrCloud cluster, and to manage replicas, SolrCloud's high-availability mechanism. (More on replicas in the next section.)
SolrCloud distributes document indexes in slices called shards. Each shard is managed by a SolrCloud instance and ZooKeeper ensures that the shards are distributed evenly among the SolrCloud instances. The SolrCloud instances and Greenplum segments are not required to be on the same hosts.
With Tanzu Greenplum Text, the default number of shards for an index is the number of Greenplum Database segments, so that each segment operates on an equal portion of the index. Optionally, a lesser number of shards can be specified when you create a Tanzu Greenplum Text index, allowing indexing workloads to be scaled for performance requirements and resource usage.
SolrCloud provides high availability by maintaining replicas of shards and providing automatic failover if a shard fails or becomes unavailable. One replica of each shard is the lead replica and any changes to it are applied to the other replicas. The replication factor, which determines the number of replicas to maintain for each shard, is set when the index is created. Replicas may also be added or dropped later using Tanzu Greenplum Text UDFs or command-line utilities.
ZooKeeper determines the locations of shard replicas among the Solr nodes and hosts. When adding a replica using a Tanzu Greenplum Text UDF or command-line utility, a new shard can be explicitly placed on a SolrCloud instance.
Forensic financial analysts need to locate communications among corporate executives that point to financial malfeasance in their firm. The analysts use the following workflow:
Tanzu Greenplum Text works with Greenplum Database and Apache SolrCloud to store and index big data for information retrieval (query) purposes. High-level workflows include data loading and indexing, and data querying.
This topic describes the following information:
The following diagram shows the Tanzu Greenplum Text workflow for loading and indexing data.
All client interaction with the system is through the Greenplum master instance.
Load data into your Greenplum Database system.
Create a database table to hold data and then add the data to the table. Greenplum provides parallel data loading utilities and protocols that help to transform and load external data in various formats and from various sources. For details, see the Greenplum Database Administrator Guide, at http://gpdb.docs.pivotal.io.
You can also create an external index for documents you retrieve from a web server, an ftp server, Amazon S3 or other S3-compatible storage, or Hadoop.
Create and configure an empty Tanzu Greenplum Text index.
gptext.create_index() user-defined function (UDF) to create an empty Tanzu Greenplum Text index for a database table. Tanzu Greenplum Text stores configuration files for the index in ZooKeeper.
Customize the index, if desired, by editing the index configuration files with the
gptext-config command-line utility. You can customize the way document text is tokenized, filtered, and transformed before storing in the index and how query text is prepared to search the index.
Populate the index with data from the database table or external data source.
gptext.index_external() UDF to add data to the index. These UDFs work by dispatching SQL queries to execute on each Greenplum segment. The segments execute the queries and add the results to the index using Solr APIs.
Commit changes to the index.
Commit changes to the Tanzu Greenplum Text index by calling the
gptext.commit_index() UDF. Until the changes are committed, queries executed on the index cannot access any data added to the index with
gptext.index(). If needed, uncommitted changes can be rolled back. SolrCloud replicates changes committed to the lead replica to the shards' non-lead replicas.
The following diagram shows the high-level Tanzu Greenplum Text query process workflow:
A user submits a SQL query designed to search the indexed data.
A Tanzu Greenplum Text search query is a SQL
SELECT statement on a Tanzu Greenplum Text search UDF that contains full-text search expressions.
The Greenplum master dispatches the query to the Greenplum Database segments.
Each segment executes the query, using the Solr API to search its index shard.
Solr analyzes and executes the search query on the lead replica for the shard.
The Greenplum Database segments return the results of the search query to the Greenplum Database master.
The Greenplum Database master aggregates the results from all segments and returns them to the client.
Tanzu Greenplum Text enables analysis of Solr indexes with Apache MADlib, an open source library for scalable in-database analytics. MADlib provides data-parallel implementations of mathematical, statistical, and machine learning methods for structured and unstructured data. You can use Tanzu Greenplum Text to perform a variety of MADlib analyses.
Learn more about Apache MADlib at http://madlib.apache.org. A
gppkg package for MADlib is available on Tanzu network at http://network.pivotal.io.
The Apache OpenNLP toolkit provides advanced machine learning tools for tokenizing, recognizing, and tagging natural language text that you can enable for Tanzu Greenplum Text indexing and searching. See Natural Language Processing with Tanzu Greenplum Text Indexes for more information.