Beats

Elastic Beats configuration for events shipping

Beat agents are lightweight shippers from Elastic. There are multiple Beat agents that can be used to send collect and ship events from different platforms, OS or services:

  • Filebeat: agent for log files.

  • Winlogbeat: agent for Windows events.

  • Auditbeat: agent for Linux audit events.

  • Packetbeat: agent for network data.

Beats Inputs

Depending on Beat agent needed, inputs configuration can be done in various ways, here is usefull Elastic documentations:

Beats Processors

Processors configuration is valid for all Beats agent.

Sigma & Nybble configuration

Nybble use Logsource product and category fields from Sigma rules to match events against relevant rules and to split processing aming Nybble nodes.

It's therefore mandatory to configure Beats shipper to add "logsource.product" and/or "logsource.category" fields to events. (added fields need to match correspondig rules)

Extra field Beats configuration:

processors:
- add_fields:
target: logsource
fields:
product: zeek
category: network_monitoring

Timestamp fields

It's sometimes preferred to use "event.created" field as Time Filter field for Kibana index pattern creation than "@timestamp".

In some cases, "event.created" is missing from events, here is the configuration to create this field from "@timestamp" (This is how it's done by Beats behind-the-scene):

processors:
- add_locale: ~
# If "event.created" is missing, create field by copying value from @timestamp.
- convert:
when:
not:
has_fields: ["event.created"]
fail_on_error: false
ignore_missing: true
mode: copy
fields:
- {from: '@timestamp', to: event.created}

Public IP flag

It can be really useful for further investigation and rule creation to know if source/destination IP addresses are public or not and to let each source node process this information instead of doing it centrally on SIEM platform.

Beats configuration to add source/destination public IP flag:

processors:
# Add a field that indicate if a source ip address is public or not.
- if:
or:
- network:
client.address: public
- network:
client.ip: public
- network:
source.address: public
- network:
source.ip: public
fail_on_error: false
ignore_missing: true
then:
- add_fields:
target: source
fields:
ip_public: true
else:
- add_fields:
target: source
fields:
ip_public: false
# Add a field that indicate if a destination ip address is public or not.
- if:
or:
- network:
destination.address: public
- network:
destination.ip: public
- network:
server.address: public
- network:
server.ip: public
fail_on_error: false
ignore_missing: true
then:
- add_fields:
target: destination
fields:
ip_public: true
else:
- add_fields:
target: destination
fields:
ip_public: false

Duplicate the "- network:" part in if-or statement to process more fields.

Beats Outputs

Kafka output configuration is valid for all Beats agent.

Kafka output

Beats shipper must be configured to send JSON formatted events to Kafka broker(s).

Beats Kafka output configuration:

output.kafka:
# Kafka Bootstrap servers list (Network port maybe different if TLS is configured)
hosts: ["$kafka_broker_IP:9092"]
# Kafka topic where events will be produced
topic: "zeek-logs"
# Output events in non-pretty JSON format.
codec.json:
pretty: false
# To avoid events to be silently drop, require one ACK.
required_acks: 1
# The number of concurrent load-balanced Kafka output workers.
worker: 1
# The maximum number of events to bulk in a single Kafka request. The default is 2048.
bulk_max_size: 1024
# Duration to wait before sending bulk Kafka request. 0 is no delay. The default is 0.
bulk_flush_frequency: 0
# To avoid events to be drop, set max size to huge value.
max_message_bytes: 10000000
# ClientID for troubleshooting purpose
client_id: zeek-client

If TLS has been configured on Kafka brokers, add your certificates and key:

# SSL/TLS parameters
ssl:
certificate: /etc/filebeat/nybble-client.crt
key: /etc/filebeat/nybble-client.key
certificate_authorities: /etc/filebeat/nybble-ca.crt