* To configure you need
- install splunk on a host server (splunk-7.3.1-linux-2.6-amd64.deb)
- Install a forwarded client on the client (Splunk forwarder) that will send the data to the splunk server
(splunkforwarder-7.2.4-8a94541dcfac-linux-2.6-amd64.deb)

- Configure the Client to send data to splunk
- Open ports on splunk server


--------------------


Download Splunk forwarder
https://www.splunk.com/en_us/download/sem.html

Installation
https://docs.splunk.com/Documentation/Splunk/7.2.3/Installation/InstallonLinux#Debian_.DEB_installation


On the rundeck side you have to install the UF (Universal Forwarder),
that what he does is take the logs that you configure and send them to Splunk

In Splunk after installed, you have to configure the APIs that are the ones that will check logs

1.- Splunk forwarded in Rundeck
2.- Configure SSL (optional)
    deb -> keytool -keystore / etc / rundeck / ssl / keystore -alias rundeck -genkey -keyalg RSA -keypass adminadmin -storepass adminadmin
            keytool -importkeystore -srckeystore / etc / rundeck / ssl / keystore -destkeystore / etc / rundeck / ssl / keystore -deststoretype pkcs12
            cp keystore truststore
            vim / etc / default / rundeckd
                export RUNDECK_WITH_SSL = true
                export RDECK_HTTPS_PORT = 4443
    Set framework.properties and rundeck-config.properties for https port 4443
3.- Check Timezone (in splunk and rundeck)
    deb -> timedatectl
            sudo timedatectl set-timezone America / Santiago

4.- Install Splunk Forwarder (on rundeck server)
        sudo dpkg -i splunkforwarder-7.2.4-8a94541dcfac-linux-2.6-amd64.deb


5. Setup the `inputs.conf` and` outputs.conf` files
    5.1. Copy the example `local` directory from Splunk Server` SPLUNK_HOME / etc / apps / rundeck_app / forwarder_config` to your UF's in Rundeck Server SPLUNK_HOME / etc / apps / search directory

    5.2. Replace placeholders
      5.2.1. `inputs.conf` (vim /opt/splunk/etc/apps/search/local/inputs.conf)
      * host (to match what you specified in the app's setup page). For a cluster, use the cluster address in the host field.


Copy the folder from the splunk server to the rundeck server
# scp / opt / splunk / etc / apps / rundeck_app / forwarder_config / local user @ inspiron: / opt / splunkforwarder / etc / apps / search /

vim /opt/splunkforwarder/etc/apps/search/local/inputs.conf
host = inspiron

HERE SET EVERYTHING THAT SAYS MONITOR TO THE PATH WITH THE FILES.
/ home / rundeck / instances / 3021_cluster / server / logs

/ var / log / rundeck /

      * log paths, if these are different to the examples (which are the Rundeck defaults)

Vim replacement
:% s / var \ / log \ / rundeck / home \ / rundeck \ / instances \ / 3021_cluster \ / server \ / logs / g

:% s / 3021_cluster / 310_snapshot_fix_notifications /



Example: replace all matches in the file that have
var / log / rundeck and change it to rundeck / instances

:% s / var \ / log \ / rundeck / rundeck \ / instances / g



:% s / yourrundeck.com / inspiron / g

      * index, if using a different index to the default of `main`

      5.2.2. `outputs.conf`

vim /opt/splunkforwarder/etc/apps/search/local/outputs.conf
      * your Splunk server host / IP and receiving port

6. Restart the UF
https://docs.splunk.com/Documentation/Splunk/7.2.5/Search/Savingsearches
sudo / opt / splunkforwarder / bin / splunk stop

Add the forwarded (run on rundeck server)
/ opt / splunkforwarder / bin / splunk add forward-server servidorsplunk: 9997

Start forwarder
sudo / opt / splunkforwarder / bin / splunk start

Check connection to splunk server from forwarded (rundeck)
sudo / opt / splunkforwarder / bin / splunk list forward-server


Splunk Server
1.- Install Splunk Software
    sudo dpkg -i <splunk_enterprise>
    cd / opt / splunk / bin


2.- Start Splunk Server
    sudo ./splunk start
Enable receiver (https://docs.splunk.com/Documentation/Splunk/7.1.2/Forwarding/Enableareceiver)

    vim /opt/splunk/etc/system/local/inputs.conf
        [splunktcp: // 9997]
        disabled = 0
    restart splunk

 / opt / splunk / bin / splunk stop

 / opt / splunk / bin / splunk start

Check port 9997
    Check listening ports (Ubuntu)
        start splunk
        sudo netstat -plnt
            tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 19025 / splunkd
            tcp 0 0 0.0.0.0:9997 0.0.0.0:* LISTEN 19025 / splunkd

*** Open firewall for splunk in centos7
 firewall-cmd --get-active-zones
firewall-cmd --zone = public --add-port = 9997 / tcp --permanent

firewall-cmd --zone = public --add-port = 8088 / tcp --permanent

firewall-cmd --reload


Give log files permissions

chmod 664 / home / rundeck / instances / 3021_cluster / server / logs / *


Create tokens for rundeck
 > settings> data input

in the notification plugin Open port 8088


Open splunk in

serplorsplunk: 8000