Configuration

Solution Pack related configuration files are located at the following directories:

<DCF-Install>/Collecting/Collector-Manager/<SP-Instance>(cloudify-orchestrator-collect)/conf/

<DCF-Install>/Collecting /Stream-Collector/<SP-Instance>/conf/

<DCF-Install>/Collecting /Kafka-Connector/<SP-Instance>/conf/

1. Collector Manager Configuration

   A Collecting Manager Configuration file can be found at below location:

   File: <DCF-Install>/Collecting/Collector-Manager/<SP-Instance>(cloudify-orchestrator-collect) /conf/collecting.xml

   Sample content:

   <?xml version="1.0" encoding="UTF-8"?>
   <config xmlns="http://www.watch4net.com/APG/Collecting" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://www.watch4net.com/APG/Collecting collecting.xsd ">
           <runOnce>true</runOnce>
       <connectors>
           <connector enabled="false" name="File" type="File-Connector"  config="conf/file-connector.xml" />
           <connector enabled="true" name="Kafka" type="Kafka-Connector" config="Kafka-Connector/cloudify-orchestrator-collect/conf/kafka-connector.xml" />
       </connectors>
           <collectors>
           <collector enabled="true" name="CloudifyDiscovery" next="Kafka" config="Stream-Collector/cloudify-orchestrator-collect/conf/discovery-cloudify.xml" />
            </collectors>
   </config>
   

   This is basic configuration of the SP which deals with how the processing should start.

2. Cloudify Orchestrator discovery Configuration

   The following example is a default Cloudify Orchestrator discovery configuration. This file will have all properties which are collected.

   File: <DCF-Install>/Collecting/Stream-Collector/<SP-Instance>(cloudify-orchestrator-collect)/conf/discovery-cloudify.xml

   <?xml version="1.0" encoding="UTF-8"?>
   <collector-configuration xmlns="http://www.watch4net.com/Text-Collector-Configuration" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.watch4net.com/Text-Collector-Configuration ../textCollectorConfiguration.xsd ">
       <source>cloudify-orchestrator-collect</source>
       <collecting-group>group</collecting-group>
       <default-character-encoding>UTF-8</default-character-encoding>
       <properties-refresh-periods>10m</properties-refresh-periods>
       <collecting-threads-pool-size>30</collecting-threads-pool-size>
   
           <collecting-configurations name="cloudify-node-instances-request">
           <include-contexts>conf/context-cloudify.xml</include-contexts>
            <data-retrieval-file>conf/requests/cloudify-requests.xml</data-retrieval-file>
                   <data-listeners id="CLOUDIFY-NODES" variable-id="id IPAddress" variable-id-separator="-">
                   <values context-key="CLOUDIFY_HOST">
                   <name>CLOUDIFY-NODES-@{CLOUDIFY_HOST}</name>
                   <unit>code</unit>
                   <replace value=".+"   by="0" pattern="true"/>
               </values>
                <properties context-key="node" property-name="nodeId"/>
                <properties context-key="id" property-name="Name"/>
                <properties context-key="id" property-name="id"/>
                <properties context-key="host_id" property-name="hostId"/>
                <properties context-key="version" property-name="Version"/>
                <properties context-key="ip" property-name="IPAddress"/>
                <properties context-key="deployment_id" property-name="deploymentId"/>
                <properties context-key="tenant_name" property-name="tenantName"/>
                <properties context-key="state" property-name="State"/>
                <properties context-key="CloudifyIP" property-name="cloudifyAgentIPAddress"/>
                <properties context-key="classname" property-name="ClassName"/>
                <properties context-key="classname" property-name="type"/>
                <properties context-key="network_name" property-name="networkName"/>
                <properties context-key="network_id" property-name="networkId"/>
                <hardcoded-properties key="DisplayName">@{DisplayName} [@{ip}]</hardcoded-properties>
                <hardcoded-properties key="Certification">CERTIFIED</hardcoded-properties>
                <hardcoded-properties key="context-name">@{CLOUDIFY_HOST}</hardcoded-properties>
                <dynamic-relations/>
           </data-listeners>
           </collecting-configurations>
   </collector-configuration>
   

   Supported configuration parameters are:

   Parameter	Description
   group	The collector will assign the string specified in the group element to generated raw values. This group should match one of your processing component configuration.
   threads pool-size	The thread pool-size attribute let you specify the number of concurrent polling threads which will share a customizable number of connections to smarts domain.
   Properties refresh	The refresh attribute controls how often the collector should set the refresh tag on the generated values. It can be either a time with a period (00:00/86400 will refresh properties each day at midnight) or a number of polling loop (360 will refresh properties every 360 polling loops). In both cases, properties will always be refresh on first polling loop, right after the collector startup.
   source	A source represents a cloudify orchestrator collect instance to discover.

3. Cloudify Orchestrator Requests Configuration

   The following example is a default Cloudify Orchestrator requests configuration: This file will have all cloudify orchestrator REST API configuration.

   File: <DCF-Install>/Collecting/Stream-Collector/<SP-Instance>(cloudify-orchestrator-collect) /conf/cloudify-requests.xml

   <?xml version="1.0" encoding="UTF-8"?>
   <data-retrieval-configuration xmlns="http://www.watch4net.com/Text-Parsing-Configuration" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.watch4net.com/Text-Parsing-Configuration ../textParsingConfiguration.xsd" xmlns:xi="http://www.w3.org/2001/XInclude">
     <retrieving-period>900s</retrieving-period>
     <http-request data-retry="1" data-timeout="300s" connection-timeout="120s" character-encoding="UTF-8">
          <lock name="perdevicelock-opst-a2-@{CLOUDIFY_HOST}" count="1"/>
          <credentials>
             <username>@{USERNAME}</username>
             <password>@{PASSWORD}</password>
          </credentials>
          <disable-ssl/>
          <request-groups name="RETRIEVE CLOUDIFYNODES">
               <requests options="GET" preAuth="true">
                   <url>@{protocol}://@{CLOUDIFY_HOST}:@{PORT}/api/v3.1/node-instances</url>
                   <headers key="Content-Type">application/json</headers>
                   <headers key="Tenant">default_tenant</headers>
               </requests>
          </request-groups>
          <json-to-xml-transformer>
               <xml-dataset parse-datasets-in-parallel="true">
               <datasets>
                   <xpath>/W4N/items/OBJECT</xpath>
                   <xml-reader>
                       <extractions xpath-expression="/OBJECT/id">id</extractions>
                       <extractions xpath-expression="/OBJECT/node_id">node</extractions>
                       <extractions xpath-expression="/OBJECT/runtime_properties/ip">ip</extractions>
   
                       <extractions xpath-expression="if (substring-before(/OBJECT/node_id,'_host') = 'bono') then ('P-CSCF') else if (substring-before(/OBJECT/node_id,'_host') = 'sprout') then ('I/S-CSCF') else if (substring-before(/OBJECT/node_id,'_host') = 'dime') then ('HSS/CDF') else if (substring-before(/OBJECT/node_id,'_host') = 'homer') then ('XDMS') else if (substring-before(/OBJECT/node_id,'_host') = 'vellum') then ('Vellum') else if (substring-before(/OBJECT/node_id,'_host') = 'ellis') then ('Ellis') else if (substring-before(/OBJECT/node_id,'_host') = 'bind') then ('Bind') else if (substring-before(/OBJECT/node_id,'_host') = 'proxy') then ('Proxy') else (/OBJECT/node_id)" result-type="string">DisplayName</extractions>
   
                       <extractions xpath-expression="if (substring-before(/OBJECT/node_id,'_host') = 'bono' or substring-before(/OBJECT/node_id,'_host') = 'sprout') then ('CallSessionControlFunctionsGroup:@{id}-@{ip}') else if (substring-before(/OBJECT/node_id,'_host') = 'dime') then ('HSSMirrorFunctionGroup:@{id}-@{ip}')  else if (substring-before(/OBJECT/node_id,'_host') = 'homer') then ('XMLDocumentManagementServersGroup:@{id}-@{ip}') else ('')" result-type="string">~@RPartOf</extractions>
   
                       <extractions xpath-expression="if (substring-before(/OBJECT/node_id,'_host') = 'vellum' or substring-before(/OBJECT/node_id,'_host') = 'ellis' or substring-before(/OBJECT/node_id,'_host') = 'proxy' or substring-before(/OBJECT/node_id,'_host') = 'bind') then ('IPMultiMediaSubSystem:@{id}-@{ip}') else ('')" result-type="string">~@RMemberOf</extractions>
   
                       <extractions xpath-expression="/OBJECT/host_id">host_id</extractions> 
                       <extractions xpath-expression="/OBJECT/runtime_properties/cloudify_agent/broker_ip">CloudifyIP</extractions>
                        <extractions xpath-expression="/OBJECT/runtime_properties/cloudify_agent/version">version</extractions>
                        <extractions xpath-expression="/OBJECT/deployment_id">deployment_id</extractions>
                        <extractions xpath-expression="/OBJECT/tenant_name">tenant_name</extractions>
                        <extractions xpath-expression="/OBJECT/state">state</extractions>
                        <extractions xpath-expression="/OBJECT/runtime_properties/server/meta/cloudify_management_network_name">network_name</extractions>
                        <extractions xpath-expression="/OBJECT/runtime_properties/server/meta/cloudify_management_network_id">network_id</extractions>
   
                        <extractions xpath-expression="if (substring-before(/OBJECT/node_id,'_host') = 'bono' or substring-before(/OBJECT/node_id,'_host') = 'sprout') then 'CallSessionControlFunction' else if (substring-before(/OBJECT/node_id,'_host') = 'homer') then 'XMLDocumentManagementServer' else if (substring-before(/OBJECT/node_id,'_host') = 'dime') then 'HSSMirrorFunction' else if (substring-before(/OBJECT/node_id,'_host') = 'vellum' or substring-before(/OBJECT/node_id,'_host') = 'proxy' or substring-before(/OBJECT/node_id,'_host') = 'bind' or substring-before(/OBJECT/node_id,'_host') = 'ellis') then 'VIMSApplicationService' else ('Unknown')" result-type="string">classname</extractions>
   
                        <release id="CLOUDIFY-NODES"/>
                   </xml-reader>
               </datasets>
           </xml-dataset>
         </json-to-xml-transformer>
    </http-request>
   </data-retrieval-configuration>
   

4. Kafka Connector Configuration

   Kafka server configuration used to publish data collected from Cloudify Orchestrator to Kafka is below:

   File : <DCF-Install>/Collecting/Kafka-Connector/<Sp-Instance>(cloudify-orchestrator-collect)/conf/kafka-connector.xml)

   <?xml version="1.0" encoding="UTF-8"?>
   <kafka-connector-config xmlns="http://www.watch4net.com/KafkaConnector" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.watch4net.com/KafkaConnector ../kafka-connector.xsd ">
           <kafka-producer-settings>
               <!-- Servers information -->
                   <server host="localhost" port="9092"/>
               <!-- Producer (and topic) information -->
               <producer topic-name="default-topic"
                   acks="1"
                   retries="0"
                   linger-ms="0ms"
                   buffer-memory="33554432"
                   compression-type="none"
                   batch-size="16384"
                   max-block-ms="5s"
                   max-in-flight-requests-per-connection="5"
                   max-request-size="1048576"/>
   
               <!-- All connection parameters will be attributes except "server" -->
               <connection request-timeout-ms="5s"
                   connections-max-idle-ms="9m"
                   retry-backoff-ms="100ms"
                   reconnect-backoff-ms="50ms"/>
   
               <!-- Additional properties -->
               <additional-parameters key="metadata.max.age.ms">1000</additional-parameters>
               <additional-parameters key="receive.buffer.bytes">32768</additional-parameters>
               <additional-parameters key="send.buffer.bytes">131072</additional-parameters>
   
            <additional-parameters key="sasl.mechanism">PLAIN</additional-parameters>
           <additional-parameters key="security.protocol">SASL_PLAINTEXT</additional-parameters>
           <additional-parameters key="sasl.jaas.config">org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="{E7344C553A75234C634E12E19146CE3979270044BDC2859695928B121428BAC4F4336A0F6C9992A84B18137257E6AA7F}";</additional-parameters>
   
           </kafka-producer-settings>
   
           <connector-component-behavior outputJson="true" flush-every-n-rawvalues="5000"/>
   
   </kafka-connector-config>
   

   Kafka Connector parameters:

   Kafka Connector parameters

   Parameters	Description
   server	This tag must occur at least once. host: The address of one of the kafka bootstrap servers port: The port that the bootstrap server is listening on.
   producer	This tag is optional, but may be used for templating kafka producers. Please refer to the schema for more information.
   connection	This tag is optional, but may be used for templating kafka connections. Please refer to the schema for more information.
   additional-parameters	This tag is optional, but may be used for configuring kafka options outside of the purview of the previous tags. These options include, but are not limited to SSL connection parameters.

   Configuring Kafka Connector:

   Parameters	Description
   kafka-producer-settings	This tag may be used if we want to customize how the writer writes to kafka. Please refer to the schema for more information.
   connector-component-behavior	This tag may be used if we want to control how often we flush to kafka.