Runs a load job as defined in a YAML formatted control file.
**gpload.py** **-f** control\_file [**-l** log\_file] [**-h** hostname] [**-p** port] [**-U** username] [**-d** database] [**-W**] [**--gpfdist\_timeout** seconds] [**--no\_auto\_trans**] [[**-v** | **-V**] [**-q**]] [**-D**] **gpload.py** **-?** **gpload.py** **--version**
The client machine where
gpload.py is executed must have the following:
PATH. This program is installed with the Greenplum loaders package.
gpload.py is a data loading utility that acts as an interface to Greenplum Database's external table parallel loading feature. Using a load specification defined in a YAML formatted control file,
gpload.py executes a load by invoking the Greenplum parallel file server (gpfdist.exe), creating an external table definition based on the source data defined, and executing an
MERGE operation to load the source data into the target table in the database.
The operation, including any SQL commands specified in the
SQL collection of the YAML control file (see Control File Format), are performed as a single transaction to prevent inconsistent data when performing multiple, simultaneous load operations on a target table.
gpfdistparallel file distribution program to send a response. Enter a value from
30seconds (entering "
0" to disables timeouts). Note that you might need to increase this value when operating on high-traffic networks.
~/gpAdminLogs/gpload_YYYYMMDD. For more information about the log file, see Log File Format.
--no_auto_trans to disable processing the load operation as a single transaction if you are performing a single load operation on the target table.
gpload.pyprocesses each load operation as a single transaction to prevent inconsistent data when performing multiple, simultaneous operations on a target table.
$PGDATABASEor defaults to the current system user name.
$PGHOSTor defaults to
$PGPORTor defaults to 5432.
$PGUSERor defaults to the current system user name.
$PGPASSWORDor from a password file specified by
~/.pgpass. If these are not set, then
gpload.pywill prompt for a password even if
-Wis not supplied.
gpload.py control file uses the YAML 1.1 document format and then implements its own schema for defining the various steps of a Greenplum Database load operation. The control file must be a valid YAML document.
gpload.py program processes the control file document in order and uses indentation (spaces) to determine the document hierarchy and the relationships of the sections to one another. The use of white space is significant. White space should not be used simply for formatting purposes, and tabs should not be used at all.
The basic structure of a load control file is:
--- [VERSION](#cfversion): 126.96.36.199 [DATABASE](#cfdatabase): db\_name [USER](#cfuser): db\_username [HOST](#cfhost): master\_hostname [PORT](#cfport): master\_port [GPLOAD](#cfgpload): [INPUT](#cfinput): - [SOURCE](#cfsource): [LOCAL\_HOSTNAME](#cfsourcelocalname): - hostname\_or\_ip [PORT](#cfsourceport): http\_port | [PORT\_RANGE](#cfversion): [start\_port\_range, end\_port\_range] [FILE](#cfsourcefile): - /path/to/input\_file [SSL](#cfsourcessl): true | false [CERTIFICATES\_PATH](#cfsourcecertificatespath): /path/to/certificates - [FULLY\_QUALIFIED\_DOMAIN\_NAME](#fqdn): true | false - [COLUMNS](#cfcolumns): - field\_name: data\_type - [TRANSFORM](#cftransform): 'transformation' - [TRANSFORM\_CONFIG](#cftransformconfig): 'configuration-file-path' - [MAX\_LINE\_LENGTH](#cfmaxlinelength): integer - [FORMAT](#cfformat): text | csv - [DELIMITER](#cfdelimiter): 'delimiter\_character' - [ESCAPE](#cfescape): 'escape\_character' | 'OFF' - [NULL\_AS](#cfnullas): 'null\_string' - [FORCE\_NOT\_NULL](#cfforcenotnull): true | false - [QUOTE](#cfquote): 'csv\_quote\_character' - [HEADER](#cfheader): true | false - [ENCODING](#cfencoding): database\_encoding - [ERROR\_LIMIT](#cferrorlimit): integer - [LOG\_ERRORS](#cferrorlog): true | false [EXTERNAL](#cfexternal): - [SCHEMA](#cfschema): schema | '%' [OUTPUT](#cfoutput): - [TABLE](#cftable): schema.table\_name - [MODE](#cfmode): insert | update | merge - [MATCH\_COLUMNS](#cfmatchcolumns): - target\_column\_name - [UPDATE\_COLUMNS](#cfupdatecolumns): - target\_column\_name - [UPDATE\_CONDITION](#cfupdatecondition): 'boolean\_condition' - [MAPPING](#cfmapping): target\_column\_name: source\_column\_name | 'expression' [PRELOAD](#cfpreload): - [TRUNCATE](#cftruncate): true | false - [REUSE\_TABLES](#cfreusetables): true | false [SQL](#cfsql): - [BEFORE](#cfbefore): "sql\_command" - [AFTER](#cfafter): "sql\_command"
gpload.pycontrol file schema. The current version is 188.8.131.52.
$PGDATABASEif set or the current system user name. You can also specify the database on the command line using the
Optional. Specifies which database role to use to connect. If not specified, defaults to the current user or
$PGUSER if set. You can also specify the database role on the command line using the
gpload.pyis not a Greenplum Database superuser, the appropriate rights must be granted to the user for the load to be processed. See the Greenplum Database Reference Guide for more information.
$PGHOSTif set. You can also specify the master host name on the command line using the
$PGPORTif set. You can also specify the master port on the command line using the
Required. Begins the load specification section. A
GPLOAD specification must have an
INPUT and an
OUTPUT section defined.
gpload.pywill start one or more instances of the gpfdist.exe file distribution program on the current host and create the required external table definition(s) in Greenplum Database that point to the source data. Note that the host from which you run
gpload.pymust be accessible over the network by all Greenplum Database hosts (master and segments).
SOURCE : Required. The `SOURCE` block of an `INPUT` specification defines the location of a source file. An `INPUT` section can have more than one `SOURCE` block defined. Each `SOURCE` block defined corresponds to one instance of the [gpfdist.exe](gpfdistexe.html) file distribution program that will be started on the local machine. Each `SOURCE` block defined must have a `FILE` specification. For more information about using the `gpfdist` parallel file server and single and multiple `gpfdist` instances, see "Loading and Unloading Data" in the *Greenplum Database Administrator Guide*.
LOCAL\_HOSTNAME : Optional. Specifies the host name or IP address of the local machine on which `gpload.py` is running. If this machine is configured with multiple network interface cards \(NICs\), you can specify the host name or IP of each individual NIC to allow network traffic to use all NICs simultaneously. The default is to use the local machine's primary host name or IP only.
PORT : Optional. Specifies the specific port number that the [gpfdist.exe](gpfdistexe.html) file distribution program should use. You can also supply a `PORT_RANGE` to select an available port from the specified range. If both `PORT` and `PORT_RANGE` are defined, then `PORT` takes precedence. If neither `PORT` or `PORT_RANGE` are defined, the default is to select an available port between 8000 and 9000.
If multiple host names are declared in `LOCAL_HOSTNAME`, this port number is used for all hosts. This configuration is desired if you want to use all NICs to load the same file or set of files in a given directory location.
PORT\_RANGE : Optional. Can be used instead of `PORT` to supply a range of port numbers from which `gpload.py` can choose an available port for this instance of the [gpfdist.exe](gpfdistexe.html) file distribution program.
FILE : Required. Specifies the location of a file, named pipe, or directory location on the local file system that contains data to be loaded. You can declare more than one file so long as the data is of the same format in all files specified.
If the files are compressed using `gzip` or `bzip2` \(have a `.gz` or `.bz2` file extension\), the files will be uncompressed automatically \(provided that `gunzip` or `bunzip2` is in your path\).
When specifying which source files to load, you can use the wildcard character \(`*`\) or other C-style pattern matching to denote multiple files. The files specified are assumed to be relative to the current directory from which `gpload.py` is executed \(or you can declare an absolute path\).
SSL : Optional. Specifies usage of SSL encryption. If `SSL` is set to true, `gpload` starts the `gpfdist` server with the `--ssl` option and uses the `gpfdists://` protocol.
CERTIFICATES\_PATH : Required when SSL is `true`; cannot be specified when SSL is `false` or unspecified. The location specified in `CERTIFICATES_PATH` must contain the following files:
- The server certificate file, `server.crt` - The server private key file, `server.key` - The trusted certificate authorities, `root.crt`
The root directory \(`/`\) cannot be specified as `CERTIFICATES_PATH`.
FULLY\_QUALIFIED\_DOMAIN\_NAME : Optional. Specifies whether `gpload` resolve hostnames to the fully qualified domain name \(FQDN\) or the local hostname. If the value is set to `true`, names are resolved to the FQDN. If the value is set to `false`, resolution is to the local hostname. The default is `false`.
A fully qualified domain name might be required in some situations. For example, if the Greenplum Database system is in a different domain than an ETL application that is being accessed by `gpload`.
COLUMNS : Optional. Specifies the schema of the source data file\(s\) in the format of `field\_name:data\_type`. The `DELIMITER` character in the source file is what separates two data value fields \(columns\). A row is determined by a line feed character \(`0x0a`\).
If the input `COLUMNS` are not specified, then the schema of the output `TABLE` is implied, meaning that the source data must have the same column order, number of columns, and data format as the target table.
The default source-to-target mapping is based on a match of column names as defined in this section and the column names in the target `TABLE`. This default mapping can be overridden using the `MAPPING` section.
TRANSFORM : Optional. Specifies the name of the input XML transformation passed to `gploadl.py`. For more information about XML transformations, see "Loading and Unloading Data" in the *Greenplum Database Administrator Guide*.
TRANSFORM\_CONFIG : Optional. Specifies the location of the XML transformation configuration file that is specified in the `TRANSFORM` parameter, above.
MAX\_LINE\_LENGTH : Optional. An integer that specifies the maximum length of a line in the XML transformation data passed to `gpload.py`.
FORMAT : Optional. Specifies the format of the source data file\(s\) - either plain text \(`TEXT`\) or comma separated values \(`CSV`\) format. Defaults to `TEXT` if not specified. For more information about the format of the source data, see "Loading and Unloading Data" in the *Greenplum Database Administrator Guide*.
DELIMITER : Optional. Specifies a single ASCII character that separates columns within each row \(line\) of data. The default is a tab character in TEXT mode, a comma in CSV mode. You can also specify a non- printable ASCII character or a non-printable unicode character, for example: `"\x1B"` or `"\u001B"`. The escape string syntax, `E'character-code'`, is also supported for non-printable characters. The ASCII or unicode character must be enclosed in single quotes. For example: `E'\x1B'` or `E'\u001B'`.
ESCAPE : Specifies the single character that is used for C escape sequences \(such as `\n`, `\t`, `\100`, and so on\) and for escaping data characters that might otherwise be taken as row or column delimiters. Make sure to choose an escape character that is not used anywhere in your actual column data. The default escape character is a \\ \(backslash\) for text-formatted files and a `"` \(double quote\) for csv-formatted files, however it is possible to specify another character to represent an escape. It is also possible to disable escaping in text-formatted files by specifying the value `'OFF'` as the escape value. This is very useful for data such as text-formatted web log data that has many embedded backslashes that are not intended to be escapes.
NULL\_AS : Optional. Specifies the string that represents a null value. The default is `\N` \(backslash-N\) in `TEXT` mode, and an empty value with no quotations in `CSV` mode. You might prefer an empty string even in `TEXT` mode for cases where you do not want to distinguish nulls from empty strings. Any source data item that matches this string will be considered a null value.
FORCE\_NOT\_NULL : Optional. In CSV mode, processes each specified column as though it were quoted and hence not a NULL value. For the default null string in CSV mode \(nothing between two delimiters\), this causes missing values to be evaluated as zero-length strings.
QUOTE : Required when `FORMAT` is `CSV`. Specifies the quotation character for `CSV` mode. The default is double-quote \(`"`\).
HEADER : Optional. Specifies that the first line in the data file\(s\) is a header row \(contains the names of the columns\) and should not be included as data to be loaded. If using multiple data source files, all files must have a header row. The default is to assume that the input files do not have a header row.
ENCODING : Optional. Character set encoding of the source data. Specify a string constant \(such as `'SQL_ASCII'`\), an integer encoding number, or `'DEFAULT'` to use the default client encoding. If not specified, the default client encoding is used. For information about supported character sets, see the *Greenplum Database Reference Guide*.
ERROR\_LIMIT : Optional. Enables single row error isolation mode for this load operation. When enabled, input rows that have format errors will be discarded provided that the error limit count is not reached on any Greenplum Database segment instance during input processing. If the error limit is not reached, all good rows will be loaded and any error rows will either be discarded or captured as part of error log information. The default is to abort the load operation on the first error encountered. Note that single row error isolation only applies to data rows with format errors; for example, extra or missing attributes, attributes of a wrong data type, or invalid client encoding sequences. Constraint errors, such as primary key violations, will still cause the load operation to abort if encountered. For information about handling load errors, see "Loading and Unloading Data" in the *Greenplum Database Administrator Guide.*
LOG\_ERRORS : Optional when `ERROR_LIMIT` is declared. Value is either `true` or `false`. The default value is `false`. If the value is `true`, rows with formatting errors are logged internally when running in single row error isolation mode. You can examine formatting errors with the Greenplum Database built-in SQL function `gp_read_error_log('table\_name')`. If formatting errors are detected when loading data, `gpload.py` generates a warning message with the name of the table that contains the error information similar to this message.
``` timestamp|WARN|1 bad row, please use GPDB built-in function gp_read_error_log('table-name') to access the detailed error row ```
: If `LOG_ERRORS: true` is specified, `REUSE_TABLES: true` must be specified to retain the formatting errors in Greenplum Database error logs. If `REUSE_TABLES: true` is not specified, the error information is deleted after the `gpload.py` operation. Only summary information about formatting errors is returned. You can delete the formatting errors from the error logs with the Greenplum Database function `gp_truncate_error_log()`.
: For more information about handling load errors, see "Loading and Unloading Data" in the *Greenplum Database Administrator Guide*. For information about the `gp_read_error_log()` function, see the `CREATE EXTERNAL TABLE` command in the *Greenplum Database Reference Guide*
EXTERNAL : Optional. Defines the schema of the external table database objects created by `gpload.py`.
: The default is to use the Greenplum Database `search_path`.
EXTERNAL is declared. The name of the schema of the external table. If the schema does not exist, an error is returned.
% (percent character) is specified, the schema of the table name specified by
TABLE in the
OUTPUT section is used. If the table name does not specify a schema, the default schema is used.
TABLE : Required. The name of the target table to load into.
MODE : Optional. Defaults to `INSERT` if not specified. There are three available load modes:
: INSERT - Loads data into the target table using the following method:
``` INSERT INTO target\_table SELECT * FROM input\_data; ```
: UPDATE - Updates the `UPDATE_COLUMNS` of the target table where the rows have `MATCH_COLUMNS` attribute values equal to those of the input data, and the optional `UPDATE_CONDITION` is true.
: MERGE - Inserts new rows and updates the `UPDATE_COLUMNS` of existing rows where `FOOBAR` attribute values are equal to those of the input data, and the optional `MATCH_COLUMNS` is true. New rows are identified when the `MATCH_COLUMNS` value in the source data does not have a corresponding value in the existing data of the target table. In those cases, the **entire row** from the source file is inserted, not only the `MATCH` and `UPDATE` columns. If there are multiple new `MATCH_COLUMNS` values that are the same, only one new row for that value will be inserted. Use `UPDATE_CONDITION` to filter out the rows to discard.
MATCH\_COLUMNS : Required if `MODE` is `UPDATE`or `MERGE`. Specifies the column\(s\) to use as the join condition for the update. The attribute value in the specified target column\(s\) must be equal to that of the corresponding source data column\(s\) in order for the row to be updated in the target table.
UPDATE\_COLUMNS : Required if `MODE` is `UPDATE` or `MERGE`. Specifies the column\(s\) to update for the rows that meet the `MATCH_COLUMNS` criteria and the optional `UPDATE_CONDITION`.
UPDATE\_CONDITION : Optional. Specifies a Boolean condition \(similar to what you would declare in a `WHERE` clause\) that must be met in order for a row in the target table to be updated \(or inserted in the case of a `MERGE`\).
MAPPING : Optional. If a mapping is specified, it overrides the default source-to-target column mapping. The default source-to-target mapping is based on a match of column names as defined in the source `COLUMNS` section and the column names of the target `TABLE`. A mapping is specified as either:
: `target\_column\_name: source\_column\_name`
: `target\_column\_name: 'expression'`
: Where expression is any expression that you would specify in the `SELECT` list of a query, such as a constant value, a column reference, an operator invocation, a function call, and so on.
Optional. Specifies operations to run prior to the load operation. Right now the only preload operation is
gpload.pywill remove all rows in the target table prior to loading it.
gpload.pywill not drop the external table objects and staging table objects it creates. These objects will be reused for future load operations that use the same load specifications. This improves performance of trickle loads (ongoing small loads to the same target table).
LOG_ERRORS: true is specified,
REUSE_TABLES: true must be specified to retain the formatting errors in Greenplum Database error logs. If
REUSE_TABLES: true is not specified, formatting error information is deleted after the
Optional. Defines SQL commands to run before and/or after the load operation. You can specify multiple
AFTER commands. List commands in the order of desired execution.
Log files output by
gpload.py have the following format:
Where timestamp takes the form:
YYYY-MM-DD HH:MM:SS, level is one of
ERROR, and message is a normal text message.
INFO messages that may be of interest in the log files are (where # corresponds to the actual number of seconds, units of data, or failed rows):
INFO|running time: \#.\#\# seconds INFO|transferred \#.\# kB of \#.\# kB. INFO|gpload succeeded INFO|gpload succeeded with warnings INFO|gpload failed INFO|1 bad row INFO|\# bad rows
If your database object names were created using a double-quoted identifier (delimited identifier), you must specify the delimited name within single quotes in the
gpload.py control file. For example, if you create a table as follows:
CREATE TABLE "MyTable" ("MyColumn" text);
gpload.py control file would refer to the above table and column names as follows:
- COLUMNS: - '"MyColumn"': text OUTPUT: - TABLE: public.'"MyTable"'
Run a load job as defined in
gpload.py -f my_load.yml
Example load control file:
--- VERSION: 184.108.40.206 DATABASE: ops USER: gpadmin HOST: mdw-1 PORT: 5432 GPLOAD: INPUT: - SOURCE: LOCAL_HOSTNAME: - etl1-1 - etl1-2 - etl1-3 - etl1-4 PORT: 8081 FILE: - /var/load/data/* - COLUMNS: - name: text - amount: float4 - category: text - desc: text - date: date - FORMAT: text - DELIMITER: '|' - ERROR_LIMIT: 25 - LOG_ERRORS: true OUTPUT: - TABLE: payables.expenses - MODE: INSERT PRELOAD: - REUSE_TABLES: true SQL: - BEFORE: "INSERT INTO audit VALUES('start', current_timestamp)" - AFTER: "INSERT INTO audit VALUES('end', current_timestamp)"
CREATE EXTERNAL TABLE in the Greenplum Database Reference Guide