prometheus-cadvisor-nodeExporter-grafana/instruction.md

Adding a Remote Node to Monitoring

Prerequisites

• A remote Linux server with SSH access
• Node Exporter installed on the remote server
• Port 9100 open on the remote server's firewall

Step 1: Install Node Exporter on Remote VM

SSH into your remote VM and run:

# Download and extract Node Exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
tar xvfz node_exporter-1.6.1.linux-amd64.tar.gz
cd node_exporter-1.6.1.linux-amd64/

# Create a systemd service
sudo tee /etc/systemd/system/node_exporter.service > /dev/null <<EOL
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
  --collector.filesystem.ignored-mount-points='^/(sys|proc|dev|host|etc)($$|/)' \
  --collector.diskstats.ignored-devices='^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\\d+n\\d+p)\\d+$$'

[Install]
WantedBy=multi-user.target
EOL

# Create user and set permissions
sudo useradd -rs /bin/false node_exporter
sudo cp node_exporter /usr/local/bin/
sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter

Step 2: Configure Firewall

On the remote VM, allow incoming connections on port 9100:

sudo ufw allow 9100/tcp

Step 3: Update Prometheus Configuration

Edit prometheus/prometheus.yml and add the following job:

  # Remote Node Exporters
  - job_name: 'remote-node-exporters'
    static_configs:
      - targets: 
        - 'REMOTE_VM_IP:9100'  # Replace with your remote VM's IP
        labels:
          instance: 'remote-vm'  # This will be the node name in Grafana
    
    # Optional: If you need basic auth for the remote exporter
    # basic_auth:
    #   username: 'your_username'
    #   password: 'your_password'
    
    # Optional: If your remote exporter uses HTTPS
    # scheme: https
    # tls_config:
    #   insecure_skip_verify: true

Step 4: Restart Services

Restart Prometheus to apply the changes:

docker-compose restart prometheus

Step 5: Verify

1. Check Prometheus targets at: http://localhost:9090/targets

   • Look for "remote-node-exporters" in the list
   • The target should show as "UP"

2. In Grafana, the new node will appear in the Node Exporter dashboard's instance selector.

Security Considerations

• For production, consider setting up authentication
• Use a reverse proxy like Nginx with HTTPS
• Restrict access to the Node Exporter port using firewall rules
• Consider using a VPN or SSH tunneling for secure communication