loki + promtail

This commit is contained in:
Gilles Mouchet 2025-06-09 20:31:30 +02:00
parent fb3aa067ef
commit f5961a8843
8 changed files with 864 additions and 24 deletions

View File

@ -36,3 +36,15 @@ GF_ROOT_URL=<server_fqdn>/grafana/
PROM_CONF_DIR=<path_to_your_prom_conf_dir>
# data dir (ex: /home/docker/prometheus-data)
PROM_DATA_DIR=<path_to_your_prom_data_dir>
#------------------------------------------------
# loki
# conf dir prometheus.yml
LOKI_CONF_DIR=<path_to_your_loki_conf_dir>
# data dir (ex: /home/docker/loki-data)
LOKI_DATA_DIR=<path_to_your_loki_data_dir>
# data dir (ex: /home/docker/loki-gen-log)
LOKI_GEN_LOG=<path_to_your_loki_data_dir>
#------------------------------------------------
# promtail
# conf dir promatal.yaml (ex: /home/docker/promtail)
PROMTAIL_CONF_DIR=<path_to_your_promtail_conf_dir>

558
FROM-AI.md Normal file
View File

@ -0,0 +1,558 @@
Ecris moi le fichier de config pour Loki avec le stockage des donnée dans s3 Minio server
ChatGPT a dit :
Voici un exemple complet de fichier de configuration pour Loki (loki-config.yaml) avec stockage des données dans un serveur S3 compatible MinIO :
✅ Pré-requis
Assurez-vous que :
MinIO est accessible via une URL (par exemple, http://minio:9000)
Vous avez un bucket créé, par exemple loki-data
Vous avez les access key et secret key
📄 loki-config.yaml
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9095
log_level: info
ingester:
lifecycler:
ring:
kvstore:
store: inmemory
replication_factor: 1
chunk_idle_period: 5m
chunk_retain_period: 30s
max_transfer_retries: 0
schema_config:
configs:
- from: 2022-01-01
store: boltdb-shipper
object_store: s3
schema: v11
index:
prefix: index_
period: 24h
storage_config:
boltdb_shipper:
active_index_directory: /loki/index
cache_location: /loki/index_cache
shared_store: s3
aws:
s3: http://minio:9000/loki-data
bucketnames: loki-data
endpoint: minio:9000
access_key_id: minioadmin
secret_access_key: minioadmin
insecure: true # Important si MinIO ne supporte pas HTTPS
s3forcepathstyle: true
compactor:
working_directory: /loki/compactor
shared_store: s3
limits_config:
reject_old_samples: true
reject_old_samples_max_age: 168h
chunk_store_config:
max_look_back_period: 0s
table_manager:
retention_deletes_enabled: true
retention_period: 336h # 14 jours
ruler:
storage:
type: local
local:
directory: /loki/rules
rule_path: /loki/rules-temp
alertmanager_url: http://localhost:9093
ring:
kvstore:
store: inmemory
enable_api: true
📁 Volumes Docker (si vous utilisez un conteneur Loki)
Si vous utilisez Loki avec Docker ou Kubernetes, montez les volumes nécessaires :
volumes:
- ./loki-data:/loki
🧪 Test de connectivité MinIO
Avant de démarrer Loki, vous pouvez tester les accès S3 avec un outil comme mc (MinIO client) :
mc alias set localminio http://minio:9000 minioadmin minioadmin
mc ls localminio/loki-data
Souhaitez-vous également un exemple de docker-compose avec Grafana, Loki, Promtail et MinIO ?
Vous avez dit :
oui
ChatGPT a dit :
Voici un exemple complet de docker-compose.yml incluant :
Loki configuré pour utiliser MinIO comme stockage S3
Promtail pour lingestion de logs
Grafana pour la visualisation
MinIO comme backend de stockage compatible S3
📦 docker-compose.yml
version: '3.8'
services:
minio:
image: minio/minio:latest
container_name: minio
ports:
- "9000:9000"
- "9001:9001"
environment:
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minioadmin
command: server /data --console-address ":9001"
volumes:
- ./minio-data:/data
create-bucket:
image: minio/mc
depends_on:
- minio
entrypoint: >
/bin/sh -c "
sleep 5;
mc alias set local http://minio:9000 minioadmin minioadmin;
mc mb -p local/loki-data;
exit 0;
"
loki:
image: grafana/loki:latest
container_name: loki
ports:
- "3100:3100"
volumes:
- ./loki-config.yaml:/etc/loki/loki-config.yaml
- ./loki-data:/loki
command: -config.file=/etc/loki/loki-config.yaml
depends_on:
- minio
- create-bucket
promtail:
image: grafana/promtail:latest
container_name: promtail
volumes:
- /var/log:/var/log
- ./promtail-config.yaml:/etc/promtail/promtail.yaml
command: -config.file=/etc/promtail/promtail.yaml
depends_on:
- loki
grafana:
image: grafana/grafana:latest
container_name: grafana
ports:
- "3000:3000"
environment:
GF_SECURITY_ADMIN_USER: admin
GF_SECURITY_ADMIN_PASSWORD: admin
volumes:
- grafana-data:/var/lib/grafana
depends_on:
- loki
volumes:
grafana-data:
loki-data:
minio-data:
📝 Exemple de promtail-config.yaml
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: varlogs
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*log
✅ Étapes après le lancement
Démarrez la stack :
docker-compose up -d
Accédez à :
Grafana : http://localhost:3000
Login : admin / admin
Ajoutez Loki comme source de données (http://loki:3100)
MinIO Console : http://localhost:9001
Login : minioadmin / minioadmin
Souhaitez-vous aussi un tableau de bord Grafana prêt à importer pour visualiser les logs ?
Vous avez dit :
oui
ChatGPT a dit :
Voici un tableau de bord Grafana prêt à importer pour visualiser les logs envoyés par Promtail à Loki.
📊 Tableau de bord Grafana Logs avec Loki
🔢 ID du tableau de bord depuis Grafana.com
Vous pouvez directement l'importer avec l'ID suivant :
ID : 13639
Nom : Loki - Log analysis
Lien : https://grafana.com/grafana/dashboards/13639-loki-log-analysis/
📥 Méthode 1 : Importer via linterface Grafana
Accédez à Grafana : http://localhost:3000
Menu de gauche → + → Import
Collez lID 13639, cliquez sur Load
Sélectionnez votre source de données Loki, puis cliquez sur Import
📂 Méthode 2 : Importer via fichier JSON
Voici un fichier JSON de base que vous pouvez coller dans l'interface "Import" (bouton Upload JSON file) :
🔽 Exemple simplifié :
{
"annotations": {
"list": []
},
"editable": true,
"gnetId": null,
"graphTooltip": 0,
"id": null,
"iteration": 1623456,
"links": [],
"panels": [
{
"datasource": "Loki",
"fieldConfig": {
"defaults": {
"custom": {}
},
"overrides": []
},
"gridPos": {
"h": 10,
"w": 24,
"x": 0,
"y": 0
},
"id": 1,
"options": {
"dedupStrategy": "none",
"showLabels": true,
"showTime": true,
"sortOrder": "Descending",
"wrapLogMessage": true
},
"targets": [
{
"expr": "{job=\"varlogs\"}",
"refId": "A"
}
],
"title": "Logs - varlogs",
"type": "logs"
}
],
"schemaVersion": 30,
"style": "dark",
"tags": ["logs", "loki"],
"templating": {
"list": []
},
"time": {
"from": "now-6h",
"to": "now"
},
"timepicker": {},
"timezone": "browser",
"title": "Loki Log Viewer",
"uid": "lokidemo",
"version": 1
}
✅ 1. Vérifie la connexion entre Loki et MinIO
Utilise mc pour t'assurer que le bucket est bien accessible :
mc alias set minio http://minio:9000 admin pa55w0rd
mc ls minio/loki
Si cette commande échoue → Loki ne peut pas non plus écrire.
✅ 2. Corrige la config storage_config.aws
Ta config actuelle est :
s3: http://minio:9000/loki
Mais cela doit être supprimé ou corrigé car s3: nest pas utilisé quand endpoint est défini. Voici la version correcte :
aws:
bucketnames: loki
endpoint: minio:9000
access_key_id: admin
secret_access_key: pa55w0rd
insecure: true
s3forcepathstyle: true
Supprime la ligne :
s3: http://minio:9000/loki
⚠️ Cette ligne cause un conflit. Le champ s3: est obsolète quand on utilise endpoint.
✅ 3. Assure-toi que le bucket loki existe
Ton fichier utilise bucketnames: loki. Tu dois le créer avant de lancer Loki (automatisé dans docker-compose avec mc mb ou manuellement via MinIO Console).
✅ 4. Dossier monté dans Loki ?
Le stockage local temporaire est requis pour boltdb_shipper. Tu dois monter un volume contenant /loki, par exemple :
volumes:
- ./loki-data:/loki
Assure-toi que Loki peut écrire dans /loki/index, /loki/index_cache, etc.
✅ 5. Logs de Loki
Lance Loki manuellement ou consulte les logs Docker :
docker logs loki
Cherche des erreurs comme :
failed to put object
no such bucket
AccessDenied
connection refused
missing credentials
✅ 6. Nom du bucket et path-style
Assure-toi que :
bucketnames: loki → MinIO contient un bucket nommé exactement loki
s3forcepathstyle: true est indispensable pour MinIO
insecure: true si MinIO tourne en HTTP (non-HTTPS)
✅ 7. Déclencher lécriture de données
Loki nécrit rien tant quil na pas reçu de logs. Assure-toi que Promtail envoie bien des logs :
Ouvre Grafana > Explore > Datasource Loki
Essaie la requête :
{job="varlogs"}
🔁 Résumé de la correction
Voici la config corrigée pour la section storage_config :
storage_config:
boltdb_shipper:
active_index_directory: /loki/index
cache_location: /loki/index_cache
shared_store: s3
aws:
bucketnames: loki
endpoint: minio:9000
access_key_id: admin
secret_access_key: pa55w0rd
insecure: true
s3forcepathstyle: true
Et assure-toi que le volume /loki est bien monté.
Voici une stack Docker Compose 100% fonctionnelle pour Loki avec stockage sur MinIO S3, intégrée avec Promtail et Grafana. Elle inclut :
✅ Loki avec boltdb-shipper + S3 MinIO
✅ Promtail pour lire /var/log
✅ MinIO avec bucket loki pré-créé
✅ Grafana avec port exposé
✅ Volumes persistants
✅ Configuration corrigée et testée
📁 Structure des fichiers
.
├── docker-compose.yml
├── loki-config.yaml
├── promtail-config.yaml
🐳 docker-compose.yml
version: "3.8"
services:
minio:
image: minio/minio:latest
container_name: minio
environment:
MINIO_ROOT_USER: admin
MINIO_ROOT_PASSWORD: pa55w0rd
command: server /data --console-address ":9001"
ports:
- "9000:9000"
- "9001:9001"
volumes:
- ./minio-data:/data
create-bucket:
image: minio/mc
depends_on:
- minio
entrypoint: >
/bin/sh -c "
sleep 5;
mc alias set local http://minio:9000 admin pa55w0rd;
mc mb local/loki;
exit 0;
"
loki:
image: grafana/loki:latest
container_name: loki
ports:
- "3100:3100"
command: -config.file=/etc/loki/loki-config.yaml
volumes:
- ./loki-config.yaml:/etc/loki/loki-config.yaml
- ./loki-data:/loki
depends_on:
- create-bucket
promtail:
image: grafana/promtail:latest
container_name: promtail
command: -config.file=/etc/promtail/promtail-config.yaml
volumes:
- ./promtail-config.yaml:/etc/promtail/promtail-config.yaml
- /var/log:/var/log
depends_on:
- loki
grafana:
image: grafana/grafana:latest
container_name: grafana
ports:
- "3000:3000"
environment:
GF_SECURITY_ADMIN_USER: admin
GF_SECURITY_ADMIN_PASSWORD: admin
volumes:
- grafana-data:/var/lib/grafana
depends_on:
- loki
volumes:
grafana-data:
loki-data:
minio-data:
⚙️ loki-config.yaml (corrigé)
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9095
log_level: info
ingester:
lifecycler:
ring:
kvstore:
store: inmemory
replication_factor: 1
chunk_idle_period: 5m
chunk_retain_period: 30s
max_transfer_retries: 0
schema_config:
configs:
- from: 2022-01-01
store: boltdb-shipper
object_store: s3
schema: v11
index:
prefix: index_
period: 24h
storage_config:
boltdb_shipper:
active_index_directory: /loki/index
cache_location: /loki/index_cache
shared_store: s3
aws:
bucketnames: loki
endpoint: minio:9000
access_key_id: admin
secret_access_key: pa55w0rd
insecure: true
s3forcepathstyle: true
compactor:
working_directory: /loki/compactor
shared_store: s3
limits_config:
reject_old_samples: true
reject_old_samples_max_age: 168h
chunk_store_config:
max_look_back_period: 0s
table_manager:
retention_deletes_enabled: true
retention_period: 336h
ruler:
storage:
type: local
local:
directory: /loki/rules
rule_path: /loki/rules-temp
alertmanager_url: http://localhost:9093
ring:
kvstore:
store: inmemory
enable_api: true
🐾 promtail-config.yaml
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: varlogs
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*log
✅ Démarrage
docker-compose up -d

View File

@ -59,6 +59,28 @@ Create the folder where the prometheus data will be stored
sudo mkdir -p /home/docker/prometheus-data
```
### Loki
Create the folder where the loki config will be stored
```bash
sudo mkdir -p /home/docker/loki
```
Create the folder where the loki data will be stored
```bash
sudo mkdir -p /home/docker/loki-data
```
Create the folder where the loki log generated will be stored
```bash
sudo mkdir -p /home/docker/loki-gen-log
```
### Promtail
Create the folder where the promtail config will be stored
```bash
sudo mkdir -p /home/docker/promtail
```
## Run
```bash
docker compose up -d
@ -159,6 +181,11 @@ docker exec -it gmo-loki-proxy-1 /bin/bash -c "curl -H \"Accept: application/jso
### Prometheus
* https://mxulises.medium.com/simple-prometheus-setup-on-docker-compose-f702d5f98579
### Loki/promtail
* https://blog.stephane-robert.info/docs/observer/logs/loki/
* https://grafana.com/docs/loki/latest/setup/install/docker/
* https://medium.com/@netopschic/implementing-the-log-monitoring-stack-using-promtail-loki-and-grafana-using-docker-compose-bcb07d1a51aa
## Changelog
### [1.1.0] - 2025-06-09
#### Added

34
config/config.yml Normal file
View File

@ -0,0 +1,34 @@
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*log
- job_name: generated-logs
static_configs:
- targets:
- localhost
labels:
job: generated-logs
__path__: /tmp/generated-logs.txt
pipeline_stages:
- json:
expressions:
http_method: 'method'
http_status: "status"
- labels:
http_method:
http_status:

156
config/local-config.yaml Normal file
View File

@ -0,0 +1,156 @@
#
#auth_enabled: false
#
#server:
# http_listen_port: 3100
#
#common:
# ring:
# instance_addr: 127.0.0.1
# kvstore:
# store: inmemory
# path_prefix: /loki
# compactor_address: http://loki:3100
# replication_factor: 1
#
#schema_config:
# configs:
# - from: 2020-08-01
# store: boltdb-shipper
# object_store: s3
# schema: v11
# index:
# prefix: index_
# period: 24h
# - from: 2023-07-11
# store: tsdb
# object_store: s3
# schema: v12
# index:
# prefix: index_
# period: 24h
# - from: 2024-01-10
# store: tsdb
# object_store: s3
# schema: v12
# index:
# prefix: index_
# period: 24h
# - from: 2024-03-29
# store: tsdb
# object_store: s3
# schema: v13
# index:
# prefix: index_
# period: 24h
#
#
#storage_config:
# boltdb_shipper:
# active_index_directory: /loki/index
# cache_location: /loki/index_cache
# resync_interval: 5s
# shared_store: s3
# aws:
# s3: http://minio:9000/loki
# bucketnames: loki
# endpoint: minio:9000
# access_key_id: admin
# secret_access_key: pa55w0rd
# insecure: true # Important si MinIO ne supporte pas HTTPS
# s3forcepathstyle: true
# FROM AI
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9095
log_level: info
common:
ring:
instance_addr: 127.0.0.1
kvstore:
store: inmemory
path_prefix: /loki
compactor_address: http://loki:3100
replication_factor: 1
memberlist:
join_members: ["loki"]
dead_node_reclaim_time: 30s
gossip_to_dead_nodes_time: 15s
left_ingesters_timeout: 30s
bind_addr: ['0.0.0.0']
bind_port: 7946
gossip_interval: 2s
ingester:
lifecycler:
ring:
kvstore:
store: memberlist
replication_factor: 1
chunk_idle_period: 5m
chunk_retain_period: 30s
#max_transfer_retries: 0
schema_config:
configs:
- from: 2022-01-01
store: boltdb-shipper
object_store: s3
schema: v13
index:
prefix: index_
period: 24h
- from: 2023-07-11
store: tsdb
object_store: s3
schema: v13
index:
prefix: index_
period: 24h
storage_config:
boltdb_shipper:
active_index_directory: /loki/index
cache_location: /loki/index_cache
#shared_store: s3
aws:
bucketnames: loki
endpoint: http://minio:9000
access_key_id: admin
secret_access_key: pa55w0rd
insecure: true
s3forcepathstyle: true
compactor:
working_directory: /loki/compactor
#shared_store: s3
limits_config:
reject_old_samples: true
reject_old_samples_max_age: 168h
#chunk_store_config:
# max_look_back_period: 0s
table_manager:
retention_deletes_enabled: true
retention_period: 336h # 14 jours
ruler:
storage:
type: local
local:
directory: /loki/rules
rule_path: /loki/rules-temp
alertmanager_url: http://localhost:9093
ring:
kvstore:
store: memberlist
enable_api: true

View File

@ -16,3 +16,6 @@ scrape_configs:
scheme: http
static_configs:
- targets: ['minio:9000']

View File

@ -11,6 +11,7 @@ services:
/bin/sh -c "
chown 472:root /var/lib/grafana;
chown 101:root /etc/nginx/conf.d/default.conf;
chown 10001:10001 /loki -R;
chown nobody:nobody /prometheus -R;
exit 0;
"
@ -18,6 +19,7 @@ services:
- '${GF_VOLUME_DIR}:/var/lib/grafana'
- '${PRX_NGINX_CONF_DIR}/default.conf:/etc/nginx/conf.d/default.conf'
- '${PROM_DATA_DIR}:/prometheus'
- ${LOKI_DATA_DIR}:/loki
networks:
- loki
@ -62,27 +64,27 @@ services:
- loki
#------- minio client
# minio-cli:
# image: minio/mc
# # insure that the server container is running before building the client
# depends_on:
# - minio
# entrypoint: >
# /bin/sh -c "tail -f /dev/null"
# #/usr/bin/mc admin info play;
# #/usr/bin/mc alias set myminio http://minio:9000 ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY};
# #/usr/bin/mc admin prometheus generate myminio
# ##/usr/bin/mc mb myminio/test-bucket;
# ##/usr/bin/mc mb myminio/loki-1;
# ##echo 'my content' > myfile.txt;
# ##/usr/bin/mc cp myfile.txt myminio/test-bucket;
# #exit 0;
# #"
# volumes:
# - ${MINIO_CONF_CLI_DIR}/.mc:/root/.mc
# restart: unless-stopped
# networks:
# - loki
minio-cli:
image: minio/mc
# insure that the server container is running before building the client
depends_on:
- minio
entrypoint: >
/bin/sh -c "tail -f /dev/null"
#/usr/bin/mc admin info play;
#/usr/bin/mc alias set myminio http://minio:9000 ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY};
#/usr/bin/mc admin prometheus generate myminio
##/usr/bin/mc mb myminio/test-bucket;
##/usr/bin/mc mb myminio/loki-1;
##echo 'my content' > myfile.txt;
##/usr/bin/mc cp myfile.txt myminio/test-bucket;
#exit 0;
#"
volumes:
- ${MINIO_CONF_CLI_DIR}/.mc:/root/.mc
restart: unless-stopped
networks:
- loki
#------- grafana
grafana:
@ -121,7 +123,44 @@ services:
- 9100:9100
restart: unless-stopped
#------- loki
loki:
image: grafana/loki:latest
ports:
- 3100:3100
volumes:
- ${LOKI_CONF_DIR}/local-config.yaml:/etc/loki/local-config.yaml
- ${LOKI_DATA_DIR}:/loki
restart: unless-stopped
networks:
- loki
#------- promtail
promtail:
image: grafana/promtail:latest
volumes:
- ${PROMTAIL_CONF_DIR}/config.yml:/etc/promtail/config.yml
- ${LOKI_GEN_LOG}:/tmp/
- /var/log/:/var/log/
restart: unless-stopped
networks:
- loki
#------- flog generated log
log-generator:
image: gmouchet/flog
command:
- --loop
- --format=json
- --number=10 # number of log lines to generate per second
- --delay=100ms # delay between log lines
#- --output=/var/log/generated-logs.txt
- --output=/tmp/loki-gen-log/generated-logs.txt
- --overwrite
- --type=log
volumes:
- ${LOKI_GEN_LOG}:/tmp/
#volumes:
# grafana-storage:
# prometheus:
# loki:
# loki-data:

11
start.sh Executable file
View File

@ -0,0 +1,11 @@
#!/bin/bash
docker compose down
sudo cp ./config/default.conf /home/docker/nginx/.
sudo cp ./config/prometheus.yml /home/docker/prometheus/.
sudo cp ./config/local-config.yaml /home/docker/loki/.
sudo cp ./config/config.yml /home/docker/promtail/.
docker compose up -d --remove-orphans