This section details the procedure to write a XML based Transaction custom collector.

The Transaction Collector allows transaction execution over the network and returns transaction KPIs. Some of the example transactions types supported are: HTTP, TCP, DNS, SQL, LDAP, and so on. The complete list of supported transaction types are covered in below sections.

To write a custom transaction collector, define templates for the following in the collector-package/templates directory:

  1. collector-manager

    1. conf

      1. collecting.xml.ftl

  2. kafka-connector

    1. conf

      1. kafka-connector.xml.ftl

  3. transaction-collector

    1. conf

      1. transaction-collector.xml.ftl

  4. property-tagging-filter

    1. conf

      1. property-tagging-filter-transaction.xml

In addition, create the following files in the collector-package directory:

  1. meta_en.properties - Metadata information for the collectors.

  2. questions.txt - Defines user configured parameters for the collectors.

  3. questions_en.properties - Defines values / description for questions.txt.

  4. spb.properties - Defines the dependent data collector modules along with version number.

  5. default.txt - Defines default values for user configurable parameters.

  6. config.json - Defines sample input for collector for testing.

  7. config_input_schema.json - Defines input JSON schema file.

Collector-Manager Configuration

To create a Transaction collector, follow the below collector-manager configuration example (collector-manager.xml.ftl):

[#ftl]
<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://www.watch4net.com/APG/Collecting"  
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.watch4net.com/APG/Collecting collecting.xsd">
   <connectors>
      <connector enabled="false" name="File"  type="File-Connector"    config="conf/file-connector.xml" />
      <connector enabled="true"  name="Kafka" type="Kafka-Connector"   config="Kafka-Connector/${module['kafka-connector'].instance}/conf/kafka-connector.xml" />
   </connectors>
   <filters>
      <filter enabled="true" name="PTF-Transaction" next="File Kafka" config="Property-Tagging-Filter/${module['property-tagging-filter'].instance}/conf/property-tagging-filter-transaction.xml" />
    </filters>
    <collectors>
      <collector enabled="true" name="TransactionCollector" next="PTF-Transaction" config="Transaction-Collector/${module['transaction-collector'].instance}/conf/transaction-collector.xml" />
    </collectors>
</config>

Transction Collector Configuration

The transaction-collector.xml file is the main configuration file of the collector. It contains information about transactions definitions and how data will be interpreted.

Collector Attributes

The file must always begins with the following section, which describes some general parameters for the collector.

<collector>
  <source>demo-transaction-collector</source>
  <max-simultaneous-transactions>10</max-simultaneous-transactions>
  <timestamp-threshold>10000</timestamp-threshold>
  <pending-transaction-soft-limit>1000</pending-transaction-soft-limit>
  <max-simultaneous-downloads-per-transaction>10</max-simultaneous-downloads-per-transaction>
  <property-refresh-frequency>43200</property-refresh-frequency>
</collector>

All elements related to the collector own configuration are nested inside this one.

  • source: The value of the source property. The source property is used to identify the collector that pushed the given metric in VMware Telco Cloud Service Assurance.

  • max-simultaneous-transactions: The global number of maximum simultaneous transactions that the collector is authorized to run. Keep in mind that each transaction instance that gets executed can also have multiple threads, so be careful when setting a high value here.

  • timestamp-threshold: Defines the threshold limit about the maximum time (in milliseconds) a transaction can live in the execution queue before being executed. If the threshold limit is reached before the execution of the transaction, the transaction will be killed, a warning message will be logged and no raw values are going to be pushed.

  • pending-transaction-soft-limit: When the number of pending transaction in the waiting queue is greater than this value, some warning messages will start appearing in the logs. This limit does not forbid transactions to run, but simply inform you when too many transactions are in the waiting queue. This can be useful to determine when your collector performance are struggling: it can be a warning that some transactions use a lot of execution time or that you are generating too many transaction instances comparing to the rythm at which they are executed.

  • max-simultaneous-downloads-per-transaction: For some transactions (such as recursive http-transaction), many files can be downloaded at the same time. This value define the maximum number of file that can be downloaded simultaneously, by one transaction. Each file that gets downloaded creates a new thread.

  • property-refresh-frequency: (Optional) The Transaction Collector implements a cache that keeps in memory the hashcode (signature) of the properties of each Raw Value that were sent to the next component (one cache entry per variable). This cache is useful in order to avoid forcing the backend to refresh uselessly the properties of a variable. However, it needs to be reset once in a while to avoid the cache to become too big in memory size. The specified value is in seconds.

Defaults for Transaction

The next block defines some default values to be use as default values if the transactions configuration does not override these properties.

<defaults>
  <group>group</group>
  <period>300000</period>
  <delay>1000</delay>
  <repeat aggregate="average">0</repeat>
  <properties>
    <property name="propname" value="Some Value Here"/>
  </properties>
  <output-directory>tmp</output-directory>
  <transaction-instances-output-to-keep>true</transaction-instances-output-to-keep>
  <http-headers>
    <property name="user-agent" value="Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.11)  Gecko/2009060308 Ubuntu/9.04 (jaunty) Firefox/3.0.11"/>
  </http-headers>
  <url-encoding>UTF-8</url-encoding>
</defaults>

  • group: The group in which the variables will be collected. This group must be defined in the backend configuration, or values will be rejected. Default value is group.

  • period: The time interval (in milliseconds) before rescheduling transaction. Setting the frequency to 1000 will schedule the transaction to run every seconds, for example.

  • delay: The start delay for the transaction (in milliseconds). This acts as a pause before executing the transaction. It can be useful when used inside a transaction sequence (also known as meta-transaction), to prevent two transactions instance from running too quickly. Default value is 0.

  • repeat: The number of times the transaction will be repeated. Example : if repeat is zero, each transaction will be executed once; if repeat is two, each transaction will be executed 3 times.

    If the repeat parameter is higher than 0, the metric values of the corresponding transaction will be determined by the aggregate attribute of the repeat parameter (Possible values : average, minimal, maximal, median and delta (max - min)). For instance, when using <repeat aggregate="average">2</repeat>, the elapsed time metric will be the average of the elapsed time of the 3 transaction executed.

  • properties: This contains some VMware TCSA properties with which any raw value will be tagged. Note that with default settings, a property name cannot be longer than 8 characters.

  • source-ip: The inteface on which the collector binds when issuing transactions. If the server has more than one interface, then specify the IP address of the interface to use. Default value is 0.0.0.0, which automatically selects one of the available interfaces. This setting may not be used by all transactions.

  • output-directory: Directory path to use for transactions that requires to store output or downloaded data. The default path, relative to the collector installation directory, is tmp. This folder will automatically be deleted from the filesystem if empty at the end of the transaction.

  • transaction-instances-output-to-keep: Value specifying the number of downloaded content folder to keep, for a particular transaction (one folder = one transaction instance). If value is set to 0, none will be kept and all the downloaded content will be cleaned up after the transaction instance. When the number of kept folder is equal or greater to the value defined by this tag for a particular transaction, the older folders are automatically deleted. Default value is 100.

  • http-request-headers: This tag defines additional HTTP header properties you need to send with the transaction. Additional headers needs to be included using the <property> tag, using a name and value attribute. This can be useful to force some global cookies, mock a user-agent etc.

  • url-encoding: Defines the encoding to use for HTTP URLs. It is highly recommended to let it to the default value, which is UTF-8.

Available Transactions

Now that we have defined the global and collector specifics parameters, we can define the transactions that we want to execute. The type of the transaction has to be defined as a specific attribute: xsi:type.

  • http-get-headers-transaction: Retreives only headers of an HTTP transaction.

  • http-transaction: Allows the execution of an HTTP transaction. Supplying the method argument, we can set the transaction to either do a GET or a POST request to a specified URL. If the recursive argument is set to true, all the content from the specified URL is download, and also all the referenced <img src=...>, <script src=...>, <link src=...> from the response.

  • sql-transaction: Allows to execute a SQL query upon a distant server.

  • tcp-connect-transaction: Allows to connect using TCP on a host with a specified port. A command can then be executed and it is also possible to wait for a response from the server. This is pretty similar to the TELNET executable (except that a ”read-timeout” has to be povided to avoid the transaction to be hanging undefinitely.

  • dns-transaction: Performs a DNS query over a resolver, for a particular host to resolve.

  • sip-transaction: Performs a SIP Invite over a SIP address.

  • ftp-transaction: Performs some basic FTP commands such as RETR (retrieving a remote file or directory), STOR (send a local file to a remote directory) or DELE (delete a remote file).

  • sftp-transaction: Performs some basic operations such as get (retrieving a file or a folder), put (sending a local file on a server) or rm (deleting a file).

  • radius-transaction: Performs an authentication request against a RADIUS server.

  • ldap-transaction: Performs a search query over a LDAP server using a search filter.

Here’s an example of a transaction as defined into the configuration file. Each and every transaction is defined by it’s own subset of parameters, some being mandatory, some being optional. They will be covered in details, however the following example shows the simplest transaction possible, with only the required parameters filled.

<transaction xsi:type="http-head-transaction">
  <id>get-example-url</id>
  <url>http://example-url.com/</url>
</transaction>

http-transaction

  • <url>: (Mandatory) The URL to hit. The URL must end with a ”/” if you are hitting a page that does not have an extension (for example, http://example-url.com/ ). This is really important in order to have the transaction to be fully and correctly executed.

  • <request-headers>: (Optional) Same thing and purpose as the defaults http-request-headers, but only for this transaction. You can use the inherit attribute to false if you don’t want your transaction to inherit from the <http-request-headers> key/value pairs.

  • <request-parameters>: (Optional) For GET request, the different key/value parameters to be appended to the URL. This field is optional, but if specified, the different values will be appended to the <url>.

    If <url> value was http://example-url.com/page-requested/, then the requested url, during the execution of the transaction would be:

    http://example-url.com/page-requested/?parameter1=value1&parameter2=value

    Note that these values can be automatically appended to the url in the <url> tag to avoid using this tag. However, don’t use the <parameters> tag if some key/value have been appended in the <url> tag.

    For POST request, the different <parameter> defined within <request-parameters> are the one passed to the requested page using POST.   

  • <url-encoding>: (Optional) The encoding format to use to encode the parameters from <request-parameters>. Default value is UTF-8.

  • <cookies-from>: (Optional) Specify the ID of the transaction from which you want to use the cookies. For example, let’s assume 2 transactions previously defined into a transaction sequence (http-get-transaction1 and http-get-transaction2). Suppose that http-get-transaction1 returned 3 cookies, you can use these cookies in http-get-transaction2 by simply forcing the <cookies-from> field to be http-get-transaction1.

<transaction xsi:type="http-transaction">
    <id>http-example</id>
    <description>Example HTTP Transaction</description>
    <url>http://localhost:8080/</url>
    <request-type method="GET" recursive="false" timeout="1000"/>
    <http-request-headers inherit="true">
        <property name="header-name" value="header-value"/>
    </http-request-headers>
    <request-parameters>
        <parameter name="param-name" value="param-value"/>
    </request-parameters>
    <url-encoding>UTF-8</url-encoding>
</transaction>

❋ Collected Metrics:

  • filecount: The total number of files that were downloaded (# of files).

  • totalsize: The total number of kb that were downloaded (kb).

  • elapsedtime: The total duration of the transaction, from the moment where it was started, up to the end of the last step (ms).

  • receivingtime: The total duration of the transaction during which data was received, from the moment when we received the first byte up to the last byte (ms).

  • averagespeed: The average speed of the transaction. Roughly, this is the totalsize divided by the elapsedtime (kb/s).

  • incomingspeed: The average speed of receiving data. Roughly, this is the totalsize divided by the receivingtime (kb/s).

  • latency: The total time required before receiving a first byte of data, since the beginning of the transaction (ms).

  • linearelapsedtime: The linear total time of download for all files. This value is a sum of all the individual download time for each files that were downloaded (ms).

  • status: The status of the request.

http-head-transaction

The HTTP Head Transaction supports the same parameters as the HTTP Transactions.

❋ Collected Metrics:

  • totalsize: The total number of kb that were downloaded (kb).

  • elapsedtime: The total duration of the transaction, from the moment where it was started, up to the end of the last step (ms).

  • averagespeed: The average speed of the transaction. Roughly, this is the totalsize divided by the elapsedtime (kb/s).

  • status: The status of the request.

sql-transaction

  • <url>: (Mandatory) The JDBC URL. Each database provider have a specific nomenclature for the URL. It is very important to add the database name against which the query is going to be executed in the URL.

  • <sql-driver>: (Mandatory) The SQL JDBC driver to be use. Note that all those driver are not shipped with the standard VMware TCSA installation, you might need to download the jar, for a specific database and use.

  • <sql-user>: (Mandatory) The username credential.

  • <sql-pass>: (Optional) The password credential. This field is mandatory but can be empty.

  • <sql-query>: (Mandatory) The query to be executed on the server.

<transaction xsi:type="sql-transaction">
    <id>mysql-example</id>
    <description>Example MySQL Transaction</description>
    <url>jdbc:mysql://127.0.0.1:3306/sales?useSSL=false</url>
    <sql-driver>com.mysql.jdbc.Driver</sql-driver>
    <sql-user>developer</sql-user>
    <sql-pass>password123</sql-pass>
    <sql-query>select COUNT(*) from history;</sql-query>
</transaction>

❋ Collected Metrics:

  • elapsedtime: The total duration of the transaction, from the moment where it was started, up to the end of the last step (ms).

  • rows: The total number of rows that were affected by the query. In case where the repeat

    attribute is greater than 0, if the number of rows is different between at least two repetitions, no rows metrics will be sent to the backend (rows).

tcp-connect-transaction

  • <connection hostname=”localhost” port=”80” timeout=”100”>: (Mandatory) The host and port on which the tcp socket should be opened. The connection timeout, in milliseconds, is optional with a default value of 1000.

  • <command>: (Optional) The command to be sent once connected.

  • <read match=”OK .*” encoding=”UTF-8” timeout=”1000”>: (Optional) When this element is set, the transaction will wait for an answer from the target host. If the server is unlikely to return some content, do not set this element, to avoid the transaction to be hanging. match is an optional regular expression against which the returned content from the server will be matched (to produce a raw value), encoding is an optional field that contains the response encoding (default is UTF-8), and the read timeout, in milliseconds, is also optional with a default value of 1000.

<transaction xsi:type="tcp-connect-transaction">
    <id>tomcat-local</id>
    <description>Example TCP Connect Transaction</description>
    <connection hostname="127.0.0.1" port="8080" timeout="2000"/>
</transaction>

❋ Collected Metrics:

  • totalsize: The total number of kb that were downloaded (kb).

  • elapsedtime: The total duration of the transaction, from the moment where it was started, up to the end of the last step (ms).

  • averagespeed: The average speed of the transaction. Roughly, this is the totalsize divided by the elapsedtime (kb/s).

  • latency: The total time required before receiving a first byte of data, since the beginning of the transaction (ms).

  • regexmatch: Tells if the expected regex matched the received output. In case where the repeat attribute is greater than 0, if the regexmatch value is different between at least two repetitions, no regexmatch metrics will be sent (no units).

dns-transaction

  • <name>: (Mandatory) The name of the website to resolve.

  • <resolver>: (Mandatory) The resolver address to use. The value must be put into the hostname attribute. Optionally, the port attribute can be use to specify a custom port number.

  • <timeout>: (Optional) The timeout, in milliseconds, after which the query will abort, if unresponding.

  • <record-type>: (Mandatory) The record type that we want to retrieve. Example: "A" record (indicated IP address of a given domain).

<transaction xsi:type="dns-transaction">
    <id>dnsgoogle</id>
    <description>Example DNS Transaction</description>
    <name>www.google.com</name>
    <resolver hostname="8.8.8.8" port="53"/>
    <timeout>5000</timeout>
    <record-type>A</record-type>
</transaction>

❋ Collected Metrics:

  • elapsedtime: The total duration of the transaction, from the moment where it was started, up to the end of the last step (ms).

  • recordcount: The number of records that were found. In case where the repeat attribute is greater than 0, if the recordcount value is different between at least two repetitions, no recordcount metrics will be sent to the backend (no units).

sip-transaction

The sip-transaction execute and INVITE Transaction (as defined in the RCF 3261), following this scenario:



  • <sip-address>: (Mandatory) The sip remote address. Format : sip:[email protected].

  • <local-port>: (Mandatory) The udp local port used to receive the responses.

  • <sip-proxy-address>: (Optional) The sip proxy server. Format: <ipaddress>:<port>/UDP.

  • <timeout>: (Optional) The timeout, in milliseconds, after which the query will abort.

  • <pause>: (Mandatory) The delay between the time the remote address send the first OK message and the BYE message. In other term, the length of the conversation.

❋ Collected Metrics:

  • tryingtime: The duration between the trying message and the ringing message (ms).

  • ringingtime: The duration between the ringing message and the first ok message (ms).

  • calllength: The total duration of the dialog, from the moment the invite message is send, up to the moment the last ok message is received (ms).

ftp-transaction

  • <connection>: (Mandatory) This tag contains the different informations needed in order to establish the connection with the remote ftp server. There are three required attributes: hostname (the hostname or IP address of the FTP server), username and password (the username and password required to connect to the FTP server). The password attribute is mandatory, but can be an empty string, if there is no password associated with the specified user.

  • <command>: (Mandatory) This tag specifies the command to be executed on the

    FTP server. The type attribute is mandatory, it describes the type of command that has to be executed. Value can be RETR (retrieve), STOR (store) or DELE (delete). The mandatory remote-target attribute specify the ressource on the remote FTP server against which the command should be executed. The local-file attribute is use with the STOR command to specify the local file to be uploaded.

<transaction xsi:type="ftp-transaction">
    <id>ftp-example</id>
    <description>Example FTP Transaction </description>
    <connection hostname="10.20.30.40" port="21" username="myuser" password="" timeout="1000" passive-mode="true"/>
    <command type="DELE" timeout="1000" remote-target="/tmp/test.log"/>
</transaction>

❋ Collected Metrics:

  • filecount: The total number of files that were downloaded (# of files).

  • totalsize: The total number of kb that were downloaded (kb).

  • elapsedtime: The total duration of the transaction, from the moment where it was started, up to the end of the last step (ms).

  • averagespeed: The average speed of the transaction. Roughly, this is the totalsize divided by the elapsedtime (kb/s).

  • latency: The total time required before receiving a first byte of data, since the beginning of the transaction (ms).

  • status: The status of the request. Refer to this page for more informations. In case where the repeat attribute is greater than 0, if more than one different status are obtained during the different requests, no status metrics will be sent to the backend (no units).

  • DigestMatch: Provide the expected MD5 hash matched the one from the specified file. In case where the repeat attribute is greater than 0, if the DigestMatch value is different between at least two repetitions, no DigestMatch metrics will be sent (no units).

  • regexmatch: Provide the expected regex matched the file content. In case where the repeat attribute is greater than 0, if the regexmatch value is different between at least two repetitions, no regexmatch metrics will be sent (no units).

sftp-transaction

  • <connection>: (Mandatory) This tag contains the different informations needed in order to establish the connection with the remote sftp server. There are two required attributes: hostname (the hostname or IP address of the FTP server), username (the username to use for the SSH connection over the server). You can also use the two following optional attributes: port (the port to connect to. Default value is 22) and the timeout which specifies the timeout time, in milliseconds, for the connection. Default value is 1000. Value of 0 means ”no timeout”.

    The authentication can be done in two different ways, either by using the password attribute which means specifying the password directly into the config, or either by specifying the private-key attribute, which specify the path to a file containing the private key to connect to the server. When using the private-key attribute, a passphrase can be specify in order to decrypt Proc- Type: 4,ENCRYPTED, DES-CBC, DES-EDE3-CBC, AES-128-CBC, AES-192-CBC and AES-256-CBC encrypted private keys.

  • <command>: (Mandatory) This tag secifies the command to be executed on the server. The type attribute is mandatory, it describes the type of command that has to be executed. Value can be get (retrieve), put (store) or rm (delete). The mandatory remote-target attribute specify the ressource on the remote FTP server against which the command should be executed. The local-file attribute is use with the command to specify the local file to be uploaded.

<transaction xsi:type="sftp-transaction">
    <id>sftp-example</id>
    <description>Example SFTP Transaction</description>
    <connection hostname="10.20.30.40" port="22" username="myuser" password="" passphrase=""/>
    <command type="put" remote-target="/opt/mykeyfile"/>
</transaction>

❋ Collected Metrics:

  • filecount: The total number of files that were downloaded (# of files).

  • totalsize: The total number of kb that were downloaded (kb).

  • elapsedtime: The total duration of the transaction, from the moment where it was started, up to the end of the last step (ms).

  • averagespeed: The average speed of the transaction. Roughly, this is the totalsize divided by the elapsedtime (kb/s).

  • latency: The total time required before receiving a first byte of data, since the beginning of the transaction (ms).

  • DigestMatch: Tells if the expected MD5 hash matched the one from the specified file. In case where the repeat attribute is greater than 0, if the DigestMatch value is different between at least two repetitions, no DigestMatch metrics will be sent (no units).

  • regexmatch: Tells if the expected regex matched the file content. In case where the repeat attribute is greater than 0, if the regexmatch value is different between at least two repetitions, no regexmatch metrics will be sent (no units).

ldap-transaction

  • <connection>: (Mandatory) This tag contains the different informations needed in order to establish the connection with the LDAP server. There are few required attributes: URL (the LDAP URL to use), authentication-type (the authentication level to be used for the connection. Possible values are none and simple).

    Optionally, the principal (the principal to use), the credentials (the credential (password) for the specified principal) and the timeout (specifies the timeout time, in milliseconds, for the connection) attributes can be specified. In order to use anonymous authentication, you can either set the authentication-type value to none or omit the principal and credentials attributes. Note that as soon as one of these two criteria is met, anonymous authentication is used.

  • <query>: (Mandatory) This tag specifies the different attributes needed in order to perform the search query over the given server. There are three mandatory attributes: base (the base node where the search should start), search-filter (the filter to be used to match the different nodes in the specified scope) and search-scope (the scope of the search. Possible values are object, one-level and subtree). Also, two optionals attributes can be use: timeout (timeout value for the search, in milliseconds (0 means no timeout)) and dereference-links (boolean value that enables/disables link dereferencing during the search).

<transaction xsi:type="ldap-transaction">
    <id>ldap-example</id>
    <description>Example LDAP Transaction</description>
    <connection url="ldap://localhost:389/" credentials="" authentication-type="none" useSSL="false"/>
    <query base="dc=example,dc=com" search-filter="uid=myuser" search-scope="object" dereference-links="false"/>
</transaction>

❋ Collected Metrics:

  • elapsedtime: The total duration of the transaction, from the moment where it was started, up to the end of the last step (ms).

  • rows: The total number of results that were returned by the search query. In case where the repeat attribute is greater than 0, if the number of rows is different between at least two repetitions, no rows metrics will be sent to the backend (rows).

radius-transaction

  • <connection>: (Mandatory) This tag contains the different informations needed in order to contact the RADIUS server. The required attributes are the following: hostname (the RADIUS server host), username (the username to use), password (the password to use), shared-secret (the shared secret used to encrypt the transmission) and authentication-type (pap or chap).

    The optional attributes are: port (the (auth, not accounting) port number to connect to. Default value is 1812), retries (the number of times the authentication request is attempted. There is a retry only after a timeout occured (if the server is down or not responding)) and timeout (timeout value for the request, in milliseconds (0 means no timeout)).

<transaction xsi:type="radius-transaction">
    <id>radious-example</id>
    <description>Example RADIUS Transaction</description>
    <connection hostname="10.20.30.40" username="myuser" password="test123" shared-secret="" port="1812" authentication-type="pap"/>
</transaction>

❋ Collected Metrics:

  • elapsedtime: The total duration of the transaction, from the moment where it was started, up to the end of the last step (ms).

  • status: The status of the authentication attempt: 100 if we received a Access-Accept packet, 0 in all other cases.