Nybble is based on Apache Flink but need other components to ship and consume security events (logs), enrich them during the processing phase and finally store them. The following components need to be installed and managed separately from Nybble.
Beats are lightweight shippers/agents used to gather logs from files, API or Databases and send them to a configured output like Elasticsearch, Logstash, ... As for Nybble platform, the logs are sent to a Kafka broker to be consumed later.
Beats shippers parse events and normalized them using ECS (Elastic Common Schema) specification. Advanced configuration of Beats also allow to pre-process events, add or modify fields that can save many processing on SIEM side and make investigation easier.
Apache Kafka is a distributed streaming platform. Kafka is designed to build real-time streaming data pipepline, its functionnalities allow to stream data at very high velocity, ensure availability and reliability of data with horizontal scalability.
There is more information in the Configuration section, but basically the data pipeline is :
Events are produced to the Kafka broker and topics specified in Beats configuration.
Events are consumed from the Kafka broker and topics specified in Nybble configuration.
MISP is used to enrich events thought the API with RESTful searches (with JSON results). Nybble use a mapping file to map each event fields (For example ECS fields) with the corresponding MISP attributes and then create a request to query the MISP API.
After each query the result is stored in a local Redis database for faster access to the result and to avoid too many request to the MISP server. The key associated with the value (result) in Redis expire after a pre-defined time specified in Nybble configuration.
After processing events are sent to Elasticsearch for storage and investigation. Nybble use two differents indexes :
Events Index : Store all events from Beats which may have been enriched during the processing. For better search performances and easier Life Index Management one index is created per day
Alerts Index : Store all alerts triggered by Nybble. For better search performances and easier Life Index Management one index is created per day.
Elasticsearch Index Mapping file is provided with Nybble, this file contains ECS fields from all Beats shipper, Beats modules and custom fields.
Kibana is data visualization WebUI for Elasticsearch. Kibana has a SIEM app that provide an interface with pre-defined dashboards and workspaces for investigation and analysis (using ECS normalized events).
Many Kibana plugins are available like the MISP plugin or the Wazuh plugin for endpoint security.
The structure of Sigma standard also tends to be use in the Kibana interface for the "Detections" feature (still in Beta).