Logging course

Welcome and thanks for joining!

What we will cover

How logging and other observability tools helps us audit activity in IT systems.

Detect and investigate malicious activity.

Centrally collect and analyze log data from a wide range of sources.

Learn how logging helps us comply with rules and regulations.

Dip our toes into benefits in other areas, such as cost savings and increased availability.

Requires basic knowledge of...

• OS and application management
• Networking
• The Linux shell
• Docker and Docker Compose

You'll also need access to a Linux system
(any distribution), web browser and SSH client.

How we will do it

• Lectures and Q&A
• Group presentations
• Graded labs
• Continuous reflection
• Quizzes and scored tests

For detailed notes, glossary, labs and similar, see:
t.menacit.se/log.zip.

These resources should be seen as a complement to an instructor lead course, not a replacement.

Acknowledgements

Thanks to IT-Högskolan and Särimner for enabling development of the course.

Hats off to all FOSS developers and free culture contributors making it possible.

Free as in beer and speech

Is anything unclear? Got ideas for improvements? Don't fancy the images in the slides?

Create an issue or submit a pull request to
the repository on GitHub!

Let us dig in!

Vocabulary and basics

Ship captains have kept logbooks
for hundreds of years.

The airplane "black box" is another good example.

Why do we log?

• Detection of malicious activity
• Deterrence of bad behavior
• Review and optimization
• Legal/Compliance requirements

What makes a good log entry/event?

Operational logs

Enables us to understand what is
happening in a system.

Audit logs

Enables us to "reenact" events of interest.

When,
Who,
What,
Where and possibly
Why?

When?

Time and date when something occurred.

Can be used to create an event timeline across different applications and systems.

Finely tuned/synchronized clocks are vital,
solutions like the Network Time Protocol help.

Who?

Which human/computer/application caused the event?

Preferably backed by strong authentication.

What?

Explaining what activity caused the log entry to be created.

May communicate why the log event might be of interest.

(from) Where?

Information related to the location of the event causer.

Name of room/building, GPS coordinates,
phone number, IP address, device identifier...

Why?

What was the reason that caused the actor to perform an action that generated the log event?

Provided by the event causer and/or sources helping reviewers to put the log entry in a context.

Many systems, such as electronic health journals, require this to minimize/detect misuse.

Preferably readable by man and machine alike!

In practice, both may contain events of interest for security analysts.

Many systems don't differentiate between them.

How do we implement logging?

"Inspection-based"

Behavior of non-cooperating applications/systems are observed by an external application.

Network traffic sniffing, raw database queries,
syscall interception, resource consumption...

Doesn't require (costly) changes to applications/systems.

Instrumented

Applications are responsible for producing log events when activity of interest occurs.

Less guesswork and more details/context compared to inspection-based logging.

Requires that the application provides trustworthy information.

Sometimes tricky/costly to implement.

Who is looking at the logs?

Development and operations

Software developers commonly used debug logging to verify expected behavior and identify bugs.

Quality assurance teams use logs to identify performance regressions.

System administrators use logs to understand behavior of systems and implement optimizations.

Security personnel

Analysts and threat hunters in
Security Operation Centers
monitor logs to detect malicious activity.

Incident Responders dig through historical logs to build timelines of threat actor actions.

(Likely your first job in the sector)

Data analysts and scientists

Modern IT environments produce a lot of log data.

Companies employ specialists to find useful insights from logs.

Helps with A/B testing and understanding of user behavior.

Improve operations or sell to third-parties.

How do we analyze the logs?

"Pattern-based"

Search for occurrence of strings/patterns that are of interest.

May be things like known error codes or
Indicators of Compromise.

Aggregation and correlation

Not all insights can be gained by simply looking for pattern in logs.

Some require counting patterns/field values and correlation between different logs/systems.

May be things like...

• Common web server paths causing errors
• Number of failed logins per username/source IP address
• Approximated physical location of app users

"Anomaly-based"

Surface log events that haven't been seen
before or those that contain information
which is "unusual".

Relies heavily on automated detection,
more about that later!

How can we make our logs more useful?

Normalization

Massage log events from different sources
to ensure that "field names", timestamps
and similar are formatted the same way.

Makes digging and correlation easier!

Enrichment

Automation may be used to extend log events with information that may aid in analysis.

Employee position/role, system owner/purpose,
geographic IP lookups, domain/address/file occurrence in IoC lists...

Visualization

Sometimes a picture says more than
a thousand words.

Usage of visual tools graphs, charts and maps can help humans identify interesting events/patterns.

Where do we analyze logs?

Local analysis

Most applications store logs in a database or simple text files.

Shell utilities and simple scripts can be used to gain important insights.

Centralized analysis

Instead of storing/analyzing logs on the producing systems, do it centrally.

Standardized protocols and software agents can be used to collect/transfer logs over the network.

What are the benefits?

Ease correlation

Almost every network connected device can produce logs.

Manually checking everything is
time-consuming (see "expensive").

Some insights require a wider-perspective than individual systems.

Minimizes risk of tampering

Log data can be manipulated to hide malicious activity or implicate individuals.

Once hacked/modified, system logs can no longer be trusted.

Centralized logging enables us to preserve the information.

Optimize performance and cost

Not all computers are optimized for log storage/analysis.

Logs can be stored on different storage mediums depending on needs.

Retention policies can be managed in one place.

Didn't you mention
machines looking at logs?

Alerting

Scheduled searches can continuously look for known bad patterns in log events.

Once identified, automated actions can be taken or humans notified for manual analysis.

Anomaly detection

Humans are quite good at identifying things out of the ordinary.

They have neither the time nor attention span to analyze the logs from modern IT environments.

Algorithms and Machine Learning can help us, especially with centralized logging.

Centralized logging services often serve as the basis for a...

Security
Information and
Event
Management system.

(Terms are often used interchangeably)

SIEMs commonly serve as a source for...

Security
Orchestration,
Automation and
Response systems.

(SIEM + semi-automated handling)

Observability is not just logging.

Metrics are other examples.

Typically not used for security purposes.

$ curl http://server.example.com:9100/metrics | grep "errs_total"

# HELP node_network_receive_errs_total Network device statistic receive_errs.
# TYPE node_network_receive_errs_total counter
node_network_receive_errs_total{device="enp2s0"} 10185

# HELP node_network_transmit_errs_total Network device statistic transmit_errs.
# TYPE node_network_transmit_errs_total counter
node_network_transmit_errs_total{device="enp2s0"} 1249

Sounds great - what's the catch?

Logging overhead

Producing/storing/transferring log events requires CPU cycles, I/O operations, bandwidth, etc.

May require more expensive hardware and impact user experience/performance.

Storage and processing costs

We're tempted to log everything - it may contain useful information.

Storing and processing these ain't free.

Many logging systems use volume-based licensing.

Collection of sensitive data

Logs may contain
Personally Identifiable Information,
credentials and other sensitive information.

That may be of interest to malicious actors,
especially if centrailized.

Anonymization/Pseduonymization
may help, but are not a penicillin.

Legal/Compliance challenges

While some laws/compliance frameworks require us to log activity, others prevent it.

Some examples are...

• Banking/Attorney privacy protection
• EU employee monitoring laws
• Storage of credit card numbers under PCI DSS

Cost of analysis

Someone needs to analyze/act on the collected data/alerts.

Expensive and hard to recruit, especially for 24/7 operations.

Even when using managed cloud services,
some analysis is likely required due to the
shared responsibility model.

We'll dig more into these topics during the course.

Our primary focus will be security related logging.

Group exercise

Putting knowledge to use

Exercise: Sell em' logging

Participants are split into four or more groups.

Each group will be assigned an example organization that they'll pitch an aspect
of logging to.

Try involving as many use-cases as possible -
add challenges/needs to the scenario if it helps.

After presentation, send slides to
courses+log_010201@0x00.lt

Org. 1: Xample Bank & Finance

Company providing banking and payment services - both to consumers and B2B.

Pitch security and audit logging.

Org. 2: Examplezon Inc.

Company providing a web shop for all kinds of physical and virtual goods.

Pitch log analysis for UX improvements and monetary gains.

Org. 3: Exemplum Medical

Company that conducts research and manufacturing of medical devices/medicine.

Pitch security, audit and operational logging.

Org. 4: Examplx Web Services

Company providing cloud infrastructure and applications as a service.

Pitch operational logging.

Reflections exercise

What have we learned so far?

Answer the following questions

• What are your most important takeaways?
• Did you have any "Ahaaa!"-moments?
• Was anything unclear or were there specifics you didn't understand?

courses+log_010301@0x00.lt

Basics recap

Let's refresh our memory

Logging helps us...

• Debug and optimize systems
• Understand user behavior and adapt UX
• Find another venue for revenue
• Comply with rules and regulation
• Detect/Investigate malicious activity

Operational logs

Enables us to understand what is
happening in a system.

Audit logs

When, who, what, where
and (possibly) why?

Both may contain security-related events!

Centralized logging services often serve as the basis for a...

Security
Information and
Event
Management system.

Benefits of centralized logging

• Ease correlation
• Minimizes risk of tampering
• Optimize performance and cost

Beware of...

• Performance overhead
• Cost of processing, storage and analysis
• Logging of sensitive information
• Legal/Compliance challenges

Let's move on,
we got a lot to cover!

Time and clocks

A not so scary introduction

IT systems rely on time and clocks
for a wide variety of important tasks.

Authentication protocols, banking applications, industrial control systems...

Allows us to correlate events/activity in different computers and the real world.

What kind of time?

Wall time / Real time.

Keeping it simple

Most computers count number of seconds
elapsed since the first of January 1970 (UTC).

Commonly called "UNIX time"/"Epoch".

Converted into local time/calendar date
by OS/applications.

(Await the horrors of 2038!)

What is a second anyway?

Something something the sun and moon.

In the late 1800s, physicists tried
to properly define a second.

Atomic clocks measure the resonant
frequency of atoms very precisely,
ain't expensive++ these days.

Since 1968, BIPM defines it as
~9 billion frequency transitions of
Cesium 133 at -273 Celsius.

Sounds quite straightforward, doesn't it?

You're not getting away that easily.

Let's talk about time zones and dates...

Time zones

You wanna eat lunch around 12, right?

Not straight lines, quite a lot of politics involved.

Important to keep track of if we're operating internationally.

Daylight savings

Everyone Many of us love a bit of sun,
but hates being confused.

Not everyone changes at the same time.

Many plan to get rid of it, few have succeeded.

...and some of those who've done it
did it in a very annoying way.

Let's make it more exciting!

Some time zones differ by
30 or 45 minutes.

(Some places don't even want
24 hour days!)

Why not throw in
leap years and leap seconds?

These are not static things and can
change (back and forth) over time.

Not just the Gregorian calendar.

Must be remembered when performing time calculations.

Is all hope lost?

Are we doomed to live in a confusing time warp?

Could any somewhat sane person wrap their head around this?

Let's meet
Arthur David Olson and Paul Eggert.

tz database

Dataset and reference code for working with international calendar time.

Continuously updated for an ever-changing world.

Maintained by ICANN since 2011.

Challenges solved?

Time/Date representation

Many different formats exist for dates and timestamps.

Which part is the year, month and day?
What time zone are we talking about?

Some are more/less readable
by humans and machines alike,
like RFC 3339 and ISO 8601.

(please use one of these!)

Okay okay -
Time is messy but important, we get it!

The two challenges

1. All clocks show the same time
2. All clocks show the right time

In theory, if we solve the second
we should automatically solve the first.

In practice, this is tricky - just trust me for now.

Let's start with the first problem...

NTP

Network Time Protocol.

Standard for clock synchronization.
Actively developed since 1980s.

Replicates time over UDP port 123.
Uses bag of tricks to calculate
and adjust for network delay.

Mitigates clock drift/skew.

Example clients/servers

• ntpd
• NTPsec
• OpenNTPD
• chrony
• systemd-timesyncd

Some only implement Simple NTP.

Weaknesses

Plain-text protocol* vulnerable
to Man-In-The-Middle attacks.

Precision typically limited
to milliseconds.

NTS

Network Time Security.

Uses TLS and PKI to exchange key
for symmetric authenticated encryption.

Extension to NTP, like HTTPS for HTTP.

Limited software support and a bit more
resource intensive than plain NTP.

PTP

Precision Time Protocol.

Version 2 can synchronize clocks
with ~nanosecond precision.

Enabled by special handling in
Network Interface Cards
and Operating Systems.

Our clocks are in sync!

Let's focus on the second problem...

What's the correct time?

In the basement of BIPM,
atomic clocks tick to define...

Universal
Time
Coordinated.

(not a time zone, but ~matches
GMT except no daylight savings)

How does my time server know
what the correct time is?

Ask another one perhaps?

Getting reference time

• Dedicated signaling cable
• Radio broadcast
• Satellite navigation system (GNSS, like GPS)
• Locally connected atomic clock

Clocks break, radio communication can be spoofed/jammed and NTP peers may lie.

What's the solution?

Use multiple sources and calculate an average!

Kool - let's grab some time!

Using pool.ntp.org

Used as default by many operating systems
and IoT appliances.

Run by volunteers, anyone* can join and contribute!

Region specific aliases, like "se.pool.ntp.org", can be used in attempts to find servers nearby.

Cloudflare and NIST provide
good alternatives/complements.

Using ntp.se

Also known as the
Swedish Distributed Time Service.

Funded by PTS and operated by Netnod.

Provides highly accurate time via
Anycast from several redundant sites
spread over Sweden.

Relies on an open-source FPGA-based
for NTP and NTS. Offers PTP.

Wanna geek out on time?

Join the annual
Netnod Tech Meeting!

Questions and/or thoughts?

Network traffic logging

Capturing/Sniffing of network traffic
allows us to implement
inspection-based logging.

Freely available tools like
tcpdump and WireShark/TShark are
easy to use on your own computer/server.

$ tcpdump -v -n -A -i "enp2s0" -- port 80              

tcpdump: listening on enp2s0,
link-type EN10MB (Ethernet),
snapshot length 262144 bytes          

[...]
11:16:57.522596 IP (tos 0x0, ttl 64, id 33090,
offset 0, flags [DF], proto TCP (6), length 127)
198.18.100.3.45924 > 93.184.216.34.80: Flags [P.],
[...]
  GET / HTTP/1.1
  Host: example.com
  User-Agent: curl/7.81.0 

Most enterprise-grade switches supports
configuration of a "span/mirror/tap" port.

Just connect a computer and start sniffing.

(Works even if other hosts are owned!)

Public-cloud features like
Azure Virtual Network TAP and
AWS Traffic Mirroring
provide similar capabilities.

Enables fairly low-friction
logging implementations!

Sounds good! What's the problem?

Resource intensive

Require many CPU cycles and lots of storage I/O operations to handle gigabits of traffic.

You'll also need quite a bit of disk space to store all that data.

(~43GB for each hour of 100Mbit/s)

Prevalence of encryption

These days, most interesting network traffic is encrypted.

Interception boxes exist to decrypt, inspect and re-encrypt data streams.

These interceptors require tricky and risky configuration changes on all networked systems.

Full network capture and storage may
be reasonable in highly sensitive
environments with limited traffic.

Partly alleviate the costs by using
solutions like tc and (e)BPF filters
to minimize processing/storage.

Any alternatives?

NIDS

Network Intrusion Detection System.

Look for suspicious network traffic using IoCs.

Functionality typically provided by
enterprise-grade firewalls, dedicated appliances
and open-source software (Snort, Suricata,
Zeek/Bro IDS, etc.)

Doesn't require storage of (all) traffic.

IPS

Often extended to act as an
Intrusion Prevention System.

Don't just detect attacks, block them.

Sounds great, but introduces some
availability risks.

Shared problems

Requires lots of computing resources.

Limited by wide-spread use of encryption.

Mayhaps we don't need to store all traffic,
but only metadata?

Let me introduce
Network flow logging.

The basics

Limit collection and storage to information
about network communication and not its content.

Many solutions define a flow as the
same peers, protocol and
(when applicable) port.

Argus

Highly configurable open-source software.

Only requires a span/mirror/tap port and
commodity server hardware.

NetFlow

Proprietary protocol developed by Cisco
for traffic meta-data logging.

Typically implemented in hardware,
resulting in low overhead.

Routers logging traffic (called "exporters")
send flow information to a "collector"
over the network.

May optionally use sampling to minimize
transfer/processing/storage costs.

IPFIX

Formally/Openly standardized by IETF as

Internet
Protocol
Flow
Information
eXport.

More or less the same as NetFlow.

If you wanna play with it but don't have
an "enterprise-grade" switch, try softflowd.

That's neat, but how is this useful?

Operational benefits

Understand how systems really
communicate with each other.

Can we decommission this server or does anything still seem to be using it?

Is traffic really reaching the web server or is it blocked somewhere during the path?

Which database server was the application using during a performance incident?

Security benefits

Detect and investigate suspicious traffic patterns.

A laptop has been infected with malware, did it try to communicate with anything?

Has any device performed port scanning in our network?

Has any of our systems communicated with a known bad host?

Conclusions

Complements host-based logging.

Makes more or less sense depending on your network architecture/security model.

Remember that you may need permission
to capture/log network traffic.

Questions or thoughts?

Log analysis with Coreutils

UNIX-like systems have historically produces tons of text files containing log events.

Take a peak in "/var/log".

Many different tools exist to tame them.

Meet GNU Coreutils

The GNU Core Utilities are the basic file,
shell and text manipulation utilities of
the GNU operating system.

These are the core utilities which are
expected to exist on every operating system.

Let's play around with some of them!

cat

$ cat fruits.txt

apple
banana

tac

$ tac fruits.txt

banana
apple

Introducing grep

Almost part of Coreutils - I'm cheating a bit.

Only output lines matching pattern:

$ cat favourite_countries.txt

1. Iceland
2. Kazakhstan
3. Greece
4. Turkmenistan

$ cat favourite_countries.txt | grep stan

2. Kazakhstan
4. Turkmenistan 

Case-insensitive

$ cat logins.log | grep -i admin

18:49 - User "Administrator" logged in

Multiple patterns

$ cat logins.log | grep -e Admin -e root

08:22 - Failed login for user "root"
18:49 - User "Administrator" logged in

Inverted/Excluding match

$ cat berries.txt

Raspberry
Tomato
Cloudberry

$ cat berries.txt | grep -v Tomato

Raspberry
Cloudberry

Including and excluding patterns

$ cat berries.txt

Raspberry
Tomato
Cloudberry

$ cat berries.txt | grep berry | grep -v Ras

Cloudberry

Matching files

$ grep -l password /etc/my_app/*.conf

/etc/my_app/cache.xml
/etc/my_app/db.conf

Files without matches

$ grep -L "Completed" /var/backup/*.log

/var/backup/mail-5_20230904.log
/var/backup/mail-5_20230905.log
/var/backup/www-2_20230904.log

Include line before match

$ cat auth.log | grep -B 1 root

08:11 Successful login for:
root@127.0.0.1

Include line after match

$ cat auth.log | grep -A 1 "login for"

08:11 Successful login for:
root@127.0.0.1
08:12 Failed login for:
backup@66.96.149.32

Wanna know how many matches you got?

wc got you covered!

$ cat logins.log | grep "Error" | wc -l

9001

Perhaps you're only interested in parts of the matching lines?

Meet cut

Split lines for every occurrence of delimiter character into fields.

Extract second space-delimited field

$ cat friends.txt

Eddard "Ned" Stark
Jon "Bastard" Snow

$ cat friends.txt | cut -d " " -f 2

"Ned"
"Bastard"

Extract third comma-separated field

$ cat taxi_rides.log

Date,From,Destination,Cost
0930,Cityterminalen,Granö,1959
1005,Sickla,Liljeholmen,201

$ cat taxi_rides.log | cut -d "," -f 3

Destination
Granö
Liljeholmen

Extract field three and all before

$ cat taxi_rides.log | cut -d "," -f -3

Date,From,Destination
0930,Cityterminalen,Granö
1005,Sickla,Liljeholmen

Extract field three and all after

$ cat taxi_rides.log | cut -d "," -f 3-

Destination,Cost
Granö,1959
Liljeholmen,201

Advanced field selection

$ cat numbers.txt

one two three four five six seven eight nine ten

$ cat numbers.txt | cut -d " " -f 1,3-5,8-

one three four five eight nine ten

Need to clean up your input/output?

tr may be able to help!

Replace occurrences of character

$ cat names.txt

Jöel
Jönas

$ cat names.txt | tr ö o

Joel
Jonas

Change casing of letters

$ cat methods.txt

get
post

$ cat methods.txt | tr "[[:lower:]]" "[[:upper:]]"

GET
POST

Delete specified characters

$ cat username.txt

__--bogdan--__

$ cat username.txt | tr -d "_-"

bogdan

Remove repeating character

$ cat friends.txt

Eddard  "Ned"      Stark
Jon     "Bastard"  Snow

$ cat friends.txt | tr -s " "

Eddard "Ned" Stark
Jon "Bastard" Snow

Let's make things a bit more interesting by performing basic aggregation.

sort and uniq are common couple for the task!

Counting unique occurrences

$ cat logins.txt

root
root
bob
root
backup
backup
backup
root
bob

$ cat logins.txt | sort | uniq -c | sort

2 bob
3 backup
4 root

Let's combine them!

$ cat auth.log

Invalid password for user >Foobar< from 10.1.1.3:4121
Untrusted key for user >Admin< from 10.1.1.3:5124
Invalid password for user >Foobar< from 127.0.0.1:3155

$ cat auth.log \
  | grep -i -e "Invalid password" -e "Untrusted key" \
  | cut -d " " -f 5 \
  | tr -d "><" \
  | sort | uniq -c | sort

1 Admin
2 Foobar

Many other great tools exist for
working with text filtering.

awk is amazing, but uses its own custom programming language.

sed is another, but heavily relies on
regular expressions which we'll cover later in the course!

As previously mentioned, out-of-sync clocks are a common problem.

Let's see how we can modify and correct timestamps in logs.

What is date?

Command-line tool for working with
calendar time.

Uses the tz database under the hood.

Useful for manual and automated
time/date conversion.

Convert to different TZ

$ date -u --date "18:47 CET"

Wed Nov  1 05:47:00 PM UTC 2023

Convert to sane format

$ date -u --date "18:47 CET" --rfc-3339=s

2023-11-01 17:47:00+00:00

Manually correct time skew

$ date -u --date "09:50 UTC - 1 hour - 5 minutes"

Wed Nov  1 08:45:00 AM UTC 2023

Output custom time format

$ date -u --date "09:50:41 UTC" "+%H_%M (==%s)"

09_50 (==1699005041)

Advanced time expressions

$ date --date "tuesday next week 13:30 PST"

Tue Nov  7 10:30:00 PM CET 2023

Great, but how do we fix a log file?

Let's combine for-loops, cut and date!

Looping in bash

$ cat fruits.txt

apple
banana

$ IFS=$'\n'
$ for LINE in $(cat fruits.txt); do
    $ echo "It's an ${LINE}"
$ done

It's an apple
It's an banana

Putting it all together

$ cat clock_skewed_log.txt

08:14=User "root" logged in
08:15=User "anna" logged out

$ IFS=$'\n'
$ for LINE in $(cat clock_skewed_log.txt); do
  $ TIMESTAMP="$(echo "${LINE}" | cut -d = -f 1)"
  $ MESSAGE="$(echo "${LINE}" | cut -d = -f 2-)"
  $ FIXED_TIMESTAMP="$(date -u --date "${TIMESTAMP} UTC + 45 minutes" "+%H:%M")"
  $ echo "${FIXED_TIMESTAMP}=${MESSAGE}"
$ done

08:59=User "root" logged in
09:00=User "anna" logged out

Wanna store the output to a file?

Just use basic redirection:

$ cat auth.log | grep "Failed to" > failed.txt

To prevent overwriting the output file,
use ">>" instead.

Or even better, with tee:

$ cat auth.log | grep "Failed to" | tee failed.txt

13:37 - Failed to authenticate "boba"
13:38 - Failed to authenticate "fatty"

$ cat failed.txt

13:37 - Failed to authenticate "boba"
13:38 - Failed to authenticate "fatty"

To prevent overwriting the output file,
add the "-a" option to tee.

Wrapping it up

Learning the ins and outs of Coreutils are a worth-while investment.

When in doubt,
use the man and info commands.

grep -F == grep --fixed-strings

With time, you'll grow your own toolbox for efficiently working with data filtering and analysis.

Lab: Analysis with Coreutils

Lab description

Graded exercise to use GNU Coreutils and grep for analyzing logs and extracting in-sights.

For detailed instructions, see:
"resources/labs/coreutils/README.md".

Remember to download the latest version of
the resources archive! ("log.zip")

Course recap

Let's refresh our memory

Time and clocks

Very messy, but important - especially for enabling log correlation!

Whenever possible, normalize time zone configuration (preferably UTC).

NTP helps us keep our clocks in sync.

NTS prevents MITM attacks and
PTP improves precision.

Network traffic logging

Capture all traffic flowing across the network
using tap/mirror/span functionality in switches.

Easy to implement inspection-based logging.

Requires lots computing resources and storage,
encrypted traffic is a challenge.

NIDS are a middle-ground that just looks for
suspicious traffic using IoCs/rulesets.

Flow logging

Only log and store traffic metadata.

Network flow ~=
same peers, protocol and
(when applicable) port.

Routers and networking gear provides
HW-based support for NetFlow/IPFIX.

(In many cases extremely) useful for
both NOC and SOC.

UNIX-like systems have historically populated
/var/log with a bunch of text files.

GNU Coreutils provides several useful tools
for text data filtration/extraction.

• cut: Split/filter lines into distinct fields
• wc: Count lines/bytes of input data
• uniq: Basic data aggregation
• tr: Various clean-up tasks
• date: Voodoo-magic with date time

And let's not forget GNU grep!
("sed" is not a part of Coreutils)

Choo choo - let's move on!

Centralized logging

A somewhat gentle introduction

What does it take to implement and operate
a centralized logging solution?

Architectural choices and their pros/cons.

We won't talk about specific
products/projects for now.

Let's begin with the soft stuff!

Ingestion

How many MB/GB/TB do we need
to process and store per day?

Events Per Second
is another very useful metric.

May require quite a bit of
research and guesstimation.

Influences HW requirements and
cluster architecture.

Availability requirements

Must the system always be available for ingestion of logs or can we handle buffering?

Are we required to store logs for a specific amount of time? Can we afford to lose some?

Is it acceptable if analysis/alerting capabilities aren't always working?

Who will use it?

Security analysts, operations personnel, developers, marketing, data scientists...

May affect the need for capabilities like visualization, reporting and machine learning.

Managed VS Self-hosted

Should we operate the solution ourselves or rely on a managed service?

Do we have the expertise, time and interest required?

Are there any legal/contractual/policy considerations?

Is there a good fit available?

Support needs

Do you have risk appetite to go through this alone?

Potential to save quite a bit of money.

Not just about the vendor - are consultants/experts available nearby?

Access control

Should all logs be accessible to every analyst?

Do we need to support multiple tenants?

Take a second to meditate upon your needs for vertical and horizontal access control.

Primary motivations

Remember to be honest with yourselves.
Are you doing this just to tick a box?

Influences performance and feature requirements.

Common fee/licensing schemes

• None: Do what you want, yay!
• Volume: Amount ingested and/or stored
• Events: Number of ingested events
• Features: Enable/Disable functionality
• Per-seat: Number of users/analysts

Enough of this - let's have a look at
some of the technical considerations,
shall we?

Pull VS Push

Should our logging solution
reach out and collect logs?

Majority of solutions require
log producers/intermediaries to
send/deliver the data (push-based).

Generally simpler to implement.

(We'll get back to agents/protocols)

Where do we parse logs?

Should the producing systems massage logs
to extract relevant data and make them
machine-readable?

Distributes the load and can save bandwidth.

Introduces friction during implementation,
software/management requirements
and processing overhead.

When do we parse logs?

Index-time VS Search-time.

Index-time parsing

Interesting data is parsed/extracted
before storage.

The heavy-lifting is done once,
improves search performance/cost.

Requires knowledge of log format
beforehand and a bit extra disk space.

Search-time parsing

Interesting data is parsed/extracted
during each query.

Enables low-friction ingestion and
less storage space.

Increases query time/cost.

While logging solutions tend to focus on one of the approaches, many supports a hybrid solution.

Retention / Rotation strategies

Time-based

Keep logs around for X amount of days.

Volume-based

Store X gigabytes of events and delete the oldest ones if that's not enough.

Capacity-based

Cram as many events as we can fit into
X% of total disk space.

In practice, we tend to combine these strategies.

Store sensitive logs for at least two weeks,
but longer if possible.

Do we want a development system or DDoS attack to overwrite our authentication logs?

Storage tiers

Not all log events are equally interesting.

We can utilize different storage tiers
to optimize cost and performance.

Let's talk about
hot, warm, cold and frozen storage
in somewhat general terms.

Hot and warm storage

Log data frequently accessed during
manual queries and automated analysis.

Backed by a fast storage medium such as
SSDs in RAID configuration.

Multiple replicas can also be used to
improve query performance.

Typically capacity-based retention.

Cold and frozen storage

Log data that is rarely/never analyzed.

Migration typically happens automatically.

Backed by high-capacity storage mediums,
such as HDDs, cloud object stores and tape.

Useful for compliance and incident response.

Scaling beyond a cluster

Find it hard to cram all events and
users into one log system?

Consider using multiple independent
servers/clusters.

Not just for performance reasons -
aids autonomy/decentralization and
simplifies access control.

Let's look at some solutions!

Selective forwarding

Specific event types and/or sources
are replicated to other log system.

Enables usage of different logging
applications based on needs/budget.

Can help with optimization of
retention and bandwidth usage.

The downsides

Hard to determine what might be of
interest before shit hits the fan.

Should forwarded events also be
kept around locally?

Can require quite a bit of coordination.

Federated searching

Some logging solutions support
"cross-cluster"/"federated" queries.

Enables us to decentralize
collection/retention/access control,
while allowing centralized alerting
and analysis.

Distributes query load and eliminates
need of unnecessary data duplication.

Ain't perfect either

Analysis is gonna be painful if data
ain't normalized.

No cross-solution standard/protocol,
often requires versions to be in sync.

Let's summarize

There are many different paths to
choose from - make sure to known
your needs and wants.

Learning how to architect/operate
(and not merely use) logging solutions
can be a great career move.

Laws and standards

Staying compliant with logging

As previously mentioned,
much regulation requires us
to log security/privacy related
activity, either directly or indirectly.

We'll limit our scope to
IT related laws/compliance standards.

For more details, check out
our "threat intelligence" course.

GDPR

General Data Protection Regulation.

Restricts how PII and be stored/processed.

Requires logging when PII is accessed by employees/third-parties.

Especially tricky/costly considering the wide definition of personal data.

(Who watches the watchers?)

NIS(2)

Directive on security of
Network and Information Systems.

Puts security-related requirements on
operators of critical infrastructure/services.

Following the baseline and reporting requirements
without logging is near impossible.

DORA

Digital Operational Resilience Act.

Attempt to harmonize IT security regulation for
financial-sector companies within the EU.

Puts requirements on monitoring, logging and incident reporting.

Similar rules exist in most finance/banking regulation.

LEK and Data Retention Directive

Lagen om Elektronisk Kommunikation.

Swedish regulation declaring metadata logging
requirements for Internet/telephony providers,
among other things.

The Data Retention Directive tried to unify
similar regulation within the EU, but was
declared unlawful by the Court of Justice.

Logs must be stored for
two, six or ten months depending on content.
Longer retention may be illegal.

PCI DSS

Payment Card Industry
Data Security Standard.

Requirement #10:

Track and monitor all access to
network resources and cardholder data

Scope managed by using an isolated
Card Data Environment.

ISO 27001

Requires regular review of
security-related events from
sensitive systems.

NTP or equivalent solution for
accurate time is mandatory.

For details, check out
A.12.4: "Logging and monitoring".

(If you can buy/borrow the standard!)

When not to log

Privacy laws and labor unions protects
certain employee activity on corporate
IT systems, regardless of ownership.

Be especially considerate of
legitimate interest when working
with logs from sources like
End-point Detection and Respone tools,
mail servers, HTTP(S) proxies and
SSL/TLS interceptors.

Ensure that signed and understood
usage polices are in place to CYA.

Wrapping up

Dig deeper into the subject
in future courses.

Questions or thoughts?

Protecting log data

CIA and privacy

The basics

You're logs may contain juicy information
that is of interest to adversaries.

Protection of their integrity is
desirable and may be a (legal) requirement.

Besides regular system hardening,
what else can we do?

Data classification

Not all log events are created equal.

Proper tagging/classification of
ingested sources aids enforcement of...

• Access control
• Retention policies
• Backups

Preferably done by producer, if possible -
make sure guidelines are available.

Append-only / Write-once storage

Certain storage mediums, like tape,
optical disks and special-purpose HDDs can
prevent data manipulation after write.

"Write Once Read Many".

A resonable compromise could be to
utilize external object storage like
AWS S3 (with strict access permissions)
or a hash-chain protected database
such as immudb.

Anonymization / Pseudonymization

The best way to handle storage
of PII is to not.

Anonymization removes/replaces information
that could be tied to an invidual/entity.

Pseudonymization works similarly,
except that a "lookup table" exist to
revert the process if needed.

Preferably done by producer, if possible.

Data scrubbing

Logs may contain secrets like passwords,
API keys/tokens and credit card numbers,
especially from applications configured
in "debugging" mode.

By configuring "search-and-replace"
rules for known patterns, this sensitive
information may be automatically removed.

As previously mentioned, preferably
performed by the log producer.

(Reducing verbosity reduce retention cost)

Availability

Can you access your logs during an attack
or outage to investigate the incident?

What dependencies exist on networking
infrastructure, centralized storage,
authentication providers, etc?

May be a decent reason for outsourcing
hosting to a third-party.

Conclusions

Course recap

Let's refresh our memory

Centralized logging requirements

• Ingestion amount (volume/EPS)
• Availability requirements
• Use-cases and intended end-users
• Hosting and sovereignty
• Support/Competence needs
• Security and access control

Collection and parsing

Most solutions available utilize "push-based"
collection and centralized parsing.

Index-time parsing helps query performance,
but increases onboarding and storage costs*.

Search-time parsing adds a per-query cost
but increases flexibility and
lowers storage costs.

Retention and storage tiers

Storing log data using
time-based, volume-based
or capacity-based retention strategies.

Optimizing cost/performance using
hot, warm, cold, frozen storage tiers.

Selective forwarding or
federated/cross-cluster querying
enables us to analyze logs from
multiple independent solutions.

Helps us decentralize management,
scale better and embrace autonomy.

(not without some issues/caveats!)

Many laws and compliance frameworks require us to log and monitor sensitive activity.

Some also prevents/restricts logging.

Protecting log data

Some example approaches are...

• Confidentiality: Hardening, pseudonymization
• Integrity: Forwarding, append-only storage
• Availability: Replication, offline backups

Heavily dependent on properly
categorizing log sources.

Let's continue, shall we?

Regular expressions

Data filtering and extraction

As humans with a bit of technical know-how,
mentally parsing this event is easy:

13:36 - "Johan" logged in from 192.0.121.195
13:38 - "Sanna" logged in from 192.0.121.203

Mayhaps you wanna extract fields,
normalize log formats or filter events?

How can we make computers do the same
without lots of sh, grep, cut and tr?

Introducing regular expression

Language for matching/extracting
patterns in text/data.

Also known as re, regex and regexp.

Exists in several different flavors,
we'll shall focus on the widely used
Extended Regular Expressions and
Perl Compatible Regular Expressions.

Used by almost all logging software
for advanced field extraction/validation.

Let's take it for a spin!

Simple string matching

$ cat auth.log | pcregrep 'login failed'

13:49 - Admin login failed using password
13:49 - Katey login failed using password
13:51 - Admin login failed using key
13:53 - Admin login failed using TOTP

Specifying variations

$ cat auth.log | pcregrep \
  'login failed using (password|TOTP)'

13:49 - Admin login failed using password
13:49 - Katey login failed using password
13:53 - Admin login failed using TOTP

$ cat auth.log | pcregrep \
  '^\d\d:\d\d - [A-Z][a-z]+ login .+ using (TOTP|key)$'

13:51 - Admin login failed using key
13:53 - Admin login failed using TOTP
13:53 - Admin login succeeded using TOTP

^\d\d:\d\d →
Line begins with two digits, a comma and two more digits.

[A-Z][a-z]+ →
Word begins with a uppercase letter, followed by one or
more lowercase letters.

.+ →
Match one or more of any character.

(TOTP|key)$ →
Line ending with "TOTP" or "key".

A note about wildcards

. →
Matches one character of any kind.

.* →
Matches any character zero or more times.

.+ →
Matches any character one or more times.

Named capture groups

$ cat auth.log | pcregrep --only-matching=2 \
  '(?<time>.+) - (?<user>.+) login (?<result>.+) using (?<method>.+)'

Admin
Katey
Admin
Admin
Admin

(Typically turned into log field names, like "time" and "method")

Repetition ranges and negation

$ cat auth.log

23:51 backup logged in
9:52 janne logged in
11:52 monitor logged in

$ cat auth.log | pcregrep --only-matching=2 \
  '^(?<time>\d{1,2}:\d{1,2}) (?<user>(?!backup|monitor).+) logged in$'

janne

Advanced features

• Multi-line matches
• "Lookahead" / "Lookbehind"
• UTF-8 character ranges
  ...

Support/Implementations differ between
flavors of regular expression.

Any alternatives?

Nothing that has taken off, but the
Simple Regex Language is a kool attempt:

/^(?:[0-9]|[a-z]|[\._%\+-])+(?:@)(?:[0-9]|[a-z]|[\.-])+(?:\.)[a-z]{2,}$/i

begin with any of (digit, letter, one of "._%+-") once or more,
literally "@",
any of (digit, letter, one of ".-") once or more,
literally ".",
letter at least 2 times,
must end, case insensitive

Regardless of its imperfections,
mastering regex is a very
worthwhile investment.

Scary at first, but a fundamental
skill for developers and
log/data analysts.

A good resource is Deeecode's
"Simplified Regular Expressions" course

Just remember to also include
negative test cases.

RegEx exercise

Let's try it out!

Why re-invent the wheel?

Pop open your web browser and visit
RegexOne (https://regexone.com)!

Feeling brave? Checkout the
"Practice Problems" section.

Log formats and protocols

Pros/Cons of common approaches

An ideal log format should enable
analysis by humans and machines alike.

We also want to collect/transfer logs
for centralized analysis/storage.

...all while being mindful of security risks,
performance impact and storage costs.

How do we achieve this?

Let's begin by discussing
pros/cons of log formats!

Keeping it simple

Log events delimited by a new line:

13:36 - "Johan" logged in from 192.0.121.195
13:38 - "Sanna" logged in from 192.0.121.203
16:20 - Invalid key from 127.0.0.1 for "Bob"

(Important to remove/escape new line characters
in your log messages - I'm looking at you, Java!)

We can try to parse it using some regular expressions:

^(?<time>\d\d:\d\d) - "(?<user>.+)" logged in from (?<ip>\d+\.\d+\.\d+\.\d+)$

Ain't all that trivial

Most log files contain multiple
different event types - in this
case, perhaps runtime errors?

We could write a bunch more
complex regular expressions,
but it soon gets out of hand.

We're assuming that the format
is stable/won't change - should
developers really guarantee this?

Simple key-values (KV)

time=13:36 type=login user=Johan ip=192.0.121.195 success=yes
time=13:38 type=login user=Sanna ip=192.0.121.203 success=yes
time=16:20 type=login user=Bob ip=127.0.0.1 success=no

Clear and easily parsable fields.

Requires many bytes just to describe
the log structure (don't make Greta cry).

Need to handle escaping/quoting of
special characters like spaces.

CSV

Let's try Comma Separated Values
with a "header" line instead!

LoginTime,User,DurationSeconds,Commands
09:42,Joe,139,apk-update apk-upgrade ps
12:52,Tim,300,dmesg top kill top reboot
22:19,Adam,36,top dmesg

Requires many bytes just to describe
the log structure (wasteful).

Need to keep the header around
and handle escaping/quoting of
delimiter character.

(Confusingly, commas aren't always
used as field delimiter)

Does this field contain a
string, number, boolean or list?

JSON

JavaScript Object Notation got you covered-ish!

[
  {
    "LoginTime": "09:42",
    "User": "Joe",
    "DurationSeconds": 139,
    "Commands": ["apk-update", "apk-upgrade", "ps -faux"]
  },
  {
    "LoginTime": "12:52",
    "User": "Tim",
    "DurationSeconds": 300,
    "Commands": ["dmesg", "top", "kill", "top", "reboot"]
  },
  [...]
]

NDJSON

Newline Delimited JavaScript Object Notation.
is probably a better fit for log events.

{"LoginTime": "09:42", "User": "Joe", "DurationSeconds": 139, [...]}
{"LoginTime": "12:52", "User": "Tim", "DurationSeconds": 300, [...]}
{"LoginTime": "22:19", "User": "Adam", "DurationSeconds": 36, [...]}

Filtering (ND)JSON with jq

$ cat logins_log.ndjson | jq -r '.
  | select( 
    .User != "monitor"
    and .DurationSeconds > 60
    and (.Commands | index("dmesg"))
  ) | .User'
  
Tim

(Consider adding "jq" to your
list of "things to learn"!)

Swapping problems

Does this field contain a
string, number or list?

Requires many bytes just to describe
the log structure (you made her cry).

Besides data type, JSON doesn't
tell us what the field contains.

The Graylog Extended Log Format
and Elastic Common Schema try to
solve the latter problem for JSON.

Many network/security products supports
the Common Event Format,
which relies on an odd mix of
CSV and key-values.

XML

eXtensible Markup Language.

Requires an honorable mention.

Similar to JSON, but with more
complexity/bells and whistles.

Used for log storage by Windows
and other enterprise software.

Hold on a second, we still haven't solved the
"byte wasting problem"!

Let's talk about binary logging formats.

Store and/or transfer key-values.

Requires external schema/lookup table
to (de)serialize/parse data.

Fabricated/Mock example:

• Byte 1 to 7: UNIX timestamp
• Byte 8: Event type (Firewall reject)
• Byte 9 to 12: Source IP address
• Byte 13 to 14: Destination port
• Byte 15 to 18: Destination IP address
• Byte 19 to 20: FW rule identifier

The pros

$ echo "\
2023-11-07T10:53:16+00:00 \
FW_BLOCK 10.13.37.142 3389 \
192.168.119.231 #146" \
| wc --bytes

74

Can save you a lot of storage/bandwidth
and many CPU cycles. Greta is happy!

The cons

Very unreadable for humans without a
schema and translation layer.

Limited support in off-the-shelf
centralized logging solutions.

If you wanna learn more, check out the
"Protocol Buffers Documentation" website.

Alright, we've chosen a log format.

How do we get these events to the
centralized logging server?

syslog

In the beginning (since 80s),
there was syslog.

Local service and network protocol
for log collection/transfer.

Port 514/UDP (and TCP, sometimes).

Still dominant method for sending
logs from network equipment and
embedded appliances.

Message content

• Timestamp
• Hostname/IP address
• Facility: 0-23 (11 == FTP daemon)
• Severity: 0-7 (0 == emergency)
• Process ID
• Message

+ perhaps more, depending on which
flavor/standard is followed...

What's the problem?

Loosely defined protocol.

In practice, this result in:

• Bad support for traffic encryption
• Insufficient authentication capabilities
• Tight message size restrictions
• Limited signaling capabilities

Furthermore, it doesn't really specify
how the message part is formatted.

Logging over HTTPS

Well understood/supported protocol.

Supports authentication, encryption,
compression and large messages.

Quite a bit of overhead/bloat for
the logging use-case.

Doesn't define a message format.

GELF

The Graylog Extended Log Format
doesn't just define a message structure.

Supports transfer via...

• UDP
• TCP
• TCP + TLS
• HTTP
• HTTPS

Hasn't yet taken over as a
syslog replacement.

For better or worse, most logging solutions
use their own custom agents/protocols.

More about those later.

Wrapping it up

Log events should have a
clearly defined structure.

Each format has its own
benefits and trade-offs.

Insufficient standardization and
a wide range of user requirements
results in custom agents/protocols
for log transmission over networks.

Windows logging

Like any other platform worth its name,
Microsoft Windows produces logs.

The way it does this is however slightly different.

Event logging subsystem

Provides APIs for structured (key-values)
operating system and application logging.

Handles event retention/rotation.

Stores logs on disk in a custom binary format, exposed to consumers as XML.

Supports log forwarding.

Tries to solve many of the same problems as syslog daemons on Linux/UNIX.

Let's have a look, shall we?

Let's generate an event

> eventcreate.exe \
  /L APPLICATION /SO MyMockApp \
  /ID 123 /T WARNING \
  /D "Look at me ma - I'm logging!"

What does the OS log?

Primarily configured using the "audit policy".

Most capabilities are turned off by default, enabling them all makes system unusable.

Requires customization depending on
your use-case.

Have a good look at
Microsoft's "Audit Policy Recommendations"
and CIS's Windows benchmark
for guidance/inspiration.

That's neat, but not too exciting.

Let's have a look at the
File Integrity Monitoring
capabilities provided by "Object auditing".

Log forwarding/collection

Windows Event Forwarding and
Windows Event Collector.

Built-in capability, slightly
tricky to configure.

Supports "source initiated" (push)
and "collector initiated" (pull).

Events are transferred using
authenticated/encrypted HTTP(S).

Often used in combination with
a third-party logging agent.

Conclusions

Capable, but quite different.

Structured logging, but often
requiring a schema/lookup table.

Highly configurable, for better
and/or worse.

For configuration guidance,
check out CIS benchmark.

Extending Windows auditing

Sysmon and PowerShell logging

Windows provides many knobs for
adjusting logging besides the
"audit policy".

Many third-party tools/products
exist to extend the capabilities
even further.

We'll look at two examples -
Sysmon and PowerShell audit logging.

Meet Sysmon

Released in 2014 as a part
of Microsoft "Sysinternals".

Hooks into Windows kernel
to intercept activity, just like many
End-point Detection and Response tools do.

Gratis, but very capable!

No official support provided by
Microsoft, sadly.

Most of Sysmon's super powers are
not enabled by default.

Granular configuration options
resulting in complexity.

Community resources like
"sysmon-modular" are
here to help!

Still not all that convinced?

Let's mock an investigation!

But what was happening
inside PowerShell?!

PowerShell is loved by
admins and hackers alike.

Utilizes Windows APIs instead
of spawning processes that are
easily picked up by tools like Sysmon.

Supports extensive audit logging,
but it needs to be enabled.

Log queries from PowerShell

> Get-WinEvent \
  -LogName "Microsoft-Windows-Sysmon/Operational" \
  | where {$_.Id -eq 22} | Select-Object -ExpandProperty Message

Dns query:
UtcTime: 2023-11-12 15:14:14.430
ProcessGuid: "{944e8c49-ebbf-6550-340b-000000000700}"
QueryName: windows.metasploit.com
QueryResults: type: 5 download2.rapid7.com.edgekey.net;
              type: 5 ::ffff:92.123.206.32;
Image: C:\Windows\System32\WindowsPowerShell\v1\powershell.exe
User: "LAB-LOG-WIN-1\Administrator"
[...]

Yo dawg, I heard you like logs...

Closing thoughts

While many third-party tools exist to aid
threat detection and incident response on
MS Windows, we can get far for free with
official tools and a bit of configuration.

Logging platform overview

Comparison of common log solutions

Many products and services are available
for centralized logging.

Let's take a look at some of them
and their pros/cons/trivia.

ArcSight

Released by Micro Focus in
early 2000s.

Defined the SIEM product category.

Artifacts such as the
Common Event Format
still lives on.

Splunk

SIEM and data analytics platform.

Provides search-time parsing and
agent configuration management.

Relatively easy to scale.

Used to be the king, but was
also priced accordingly.

Provided as on-prem or SaaS.

Acquired by Cisco in 2023.

Elastic / "ELK" stack

Elasticsearch, Logstash, Kibana
+ Beats (logging agents).

Index-time parsing backed by
very capable search engine.

Resource hungry, but quite easy to scale.

Support and plugins for "enterprise"
features (like auth and TLS) available
from Elastic.

Closed-sourced in 2021, resulting in
"OpenSearch" fork. Became open again in 2024.

Graylog

Logging solutions with
batteries included.

Uses Elasticsearch OpenSearch
under the hood.

Less capable than the Elastic stack,
but easier to setup/manage.

Closed-sourced in 2021.

Wazuh

Builds upon the ELK stack OpenSearch and
a fork of the HIDS "OSSEC" to provide
an open source solution for...

*drum roll*

eXtended Detection and Response.

Freely available, but developed
by a company that sells SaaS,
consulting and support.

Loki

FOSS logging solution built
by Grafana Labs.

Trying to take a fresh approach
and learn from previous mistake.

Easier and cheaper to operate.

Less flexible, but good enough
for most operational logging
use-cases. A SIEM? Not yet.

Also available as SaaS.

Microsoft Sentinel

Security-focused solution for
centralized logging.

Provides lots of features OoB,
things usually in the realm
of SOC operators/analysts.

Only provided as cloud service.

Many options exist with
pros and cons.

Our focus moving forward
will be OpenSearch.

Course recap

Let's refresh our memory

Regular expressions

Language for pattern matching and
data extraction.

Defacto standard for massaging
unstructured log data.

Practice your skills using sites
like RegexOne!

Common formats

We want structured data to enable
better querying/filtering.

Key-values, CSV, (ND)JSON and XML
are commonly used formats.

Usage of binary storage/transfer of
logs decrease overhead/cost.

The Common Event Format
and Elastic Common Schema
exist to provide standardized naming
and data types for fields in logs.

Transfer methods and protocols

Syslog is a common, but flawed, protocol
for sending log events over a network.

Graylog Extended Log Format
aims replace syslog and supports several
different transfer mechanisms,
such as UDP and TCP + TLS.

HTTP(S) is a popular option due to its
wide support, but introduces overhead.

Logging on Windows

The kernel, system services and
most applications rely on the
Windows event log subsystem.

Log data is structured and stored on
disk in a custom binary XML format.

Windows Event Forwarding uses
HTTP or HTTPS for transferring events to
a Windows Event Collector.

Tweaked through audit policies and
can be extended using tools like Sysmon.

Thoughts and/or questions?

OpenSearch introduction

The OpenSearch project aims to
provide an open-source software suite for
data processing, analysis and visualization.

Popular choice for centralized logging.

Platform used in coming labs and presentations.

History / Background

In the beginning, there was
Elasticsearch, Kibana and Logstash,
which formed the open-source "ELK stack".

Loved by devops teams, security analysts
and data scientists alike.

The company leading development, Elastic,
made money by selling proprietary plugins
(called "X-Pack") and support/services.

The community developed freely available
plugins that matched many of Elastic's
proprietary features.

The "Open Distro" project packaged
these together with the open-source
ELK-components to provide a fully
usable out-of-the-box experience.

Elastic didn't like this and were mad
at the big cloud providers for selling
"ELK as a Service" without willingly
"giving them their fair share".

In 2021, Elastic closed-sourced
Elasticsearch and Kibana.

This made the community and
companies basing their services
on the ELK stack a bit grumpy.

The OpenSearch project was formed
to provide an open-source fork.

Developed by the community, supported
by the OpenSearch Software Foundation.

(Elastic open sourced it again in 2024)

What does our implementation
of the stack look like?

OpenSearch

Fork of Elastic's "Elasticsearch".

It's a search engine, powered by
Apache Lucene, but you can
think of it as a database.

Users can submit arbitrary JSON objects
to an index. Once stored, they are
called "documents" and become
available for queries/analysis.

Ehmm, perhaps a demonstration
might be reasonable?

$ curl \
  "https://teacher:hunter_2@search-engine.logs.labs.teaching.sh/" \
  --request GET

{
  "name" : "893e26a2db10",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "dVw9XlAYQk2laTvVSv7LpA",
  "version" : {
    [...]
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

$ BASE_URL="https://teacher:hunter_2@search-engine.logs.labs.teaching.sh"
$ curl "${BASE_URL}/" --request GET

{
  "name" : "893e26a2db10",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "dVw9XlAYQk2laTvVSv7LpA",
  "version" : {
    [...]
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

{
  "name": "Test Examplesson",
  "kool": false,
  "age": 42
}

$ curl "${BASE_URL}/myindex/_doc?pretty" \
  --request POST \
  --header "Content-Type: application/json" \
  --data @example-document.json

{
  "_index" : "myindex",
  "_id" : "RXE06IsBQrucVyA52bmU",
  "_version" : 1,
  "result" : "created",
  [...]
}

$ curl \
  "${BASE_URL}/myindex/_search?pretty" \
  --request GET

{
  [...]
  "hits" : {
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "myindex",
        "_id" : "h3Ey6IsBQrucVyA5xLjN",
        "_score" : 1.0,
        "_source" : {
          "name" : "Test Examplesson",
          "kool" : false,
          "age" : 42
        }
        [...]

Why is it kool?

Amazing analytics capabilities
and quite easy to scale!

Data storage

Documents are grouped in indices
(plural of "index").

Documents in an index are
stored in one or more shards.

Shards are spread over
one or more nodes.

Node clustering

OpenSearch is "cluster-first".

Adding more nodes and shards can improve
capacity, performance and availability.

Can be scaled down to save
money/electricity.

Node types can be mixed and matched to
implement storage/processing tiers.

Searching capabilities

Swiss army knife of data analysis.

Full-text queries and advanced aggregation.

Includes plugins out-of-the-box for
machine learning, correlation, etc.

Detailed field mapping

Besides the JSON data types, such as strings,
integers and arrays, several others are supported
to aid the search engine find relevant results.

{
  "mappings": {
    "properties": {
      "source.ip": {
        "type": "ip"
      },
      "source.geo.location": {
        "type": "geo_point"
      }
    }
  }
}

OpenSearch is a complex beast.

Takes quite a bit of expertise
to manage and use.

Don't feel bad if you're
getting confused.

We'll focus the use-case of
centralized logging.

Let's move on, shall we?

Next up is data ingestion!

Logstash

Helps us to build log processing pipelines!

Supports wide range of "inputs", "filters"
and "outputs" to enable centralized logging
in heterogeneous IT environments.

Development led by Elastic,
available under an open-source license.

What is a pipeline?

Pipelines consist of three stages:

1. Input: Specify how to receive/collect events
2. Filter: Manipulate, enrich and drop events
3. Output: Do something with processed events

The filter stage effectively acts as a
script being executed for each log event.

A Logstash instance can run one or more
processing pipelines simultaneously.

Input stage

A single pipeline can have one or more
inputs configured, such as:

• HTTP(S)
• Syslog
• Netflow / IPFIX
• S3 object storage
• Redis / Kafka
• Scheduled command execution
• Twitter X feed

Filter stage

Conditionally execute zero or more
filter plugins depending on event
content or external factors.

Can look a lot like a script with
if-else conditions and "function calls"
to plugins for manipulating/enriching
log events.

Output stage

A single pipeline can have one or more
outputs configured, such as:

• File
• Elasticsearch / OpenSearch
• Syslog / GELF / Logstash
• Email / IRC / Slack

Conditional statements can be used to
control which output is used based on
event content.

Still a bit confused?

Let's peak at a real example
from the lab environment!

input {
  # JSON objects POSTed via HTTP
  http {
    type => "json_events"
    port => 8080
    codec => json
  }

  # Events from logging agents
  beats {
    type => "local_events"
    port => 5044
  }
  
[...]

filter {

  # Parse NGINX web server logs
  if [event][dataset] == "nginx.access" {
    # Tag added to all web server access logs,
    # regardless of server software used
    mutate {
      add_tag => [
        "web_server_access"
      ]
    }

    # Use Grok (regular expressions on steroids)
    # to extract fields from event.
    grok {
      match => {
        "message" => "%{IPORHOST:[source][ip]} - \
        %{DATA:user} \[%{HTTPDATE:time}\] [...]"

[...]

[...]

  # Parse time from request and replace existing
  # timestamp, as the former is based on when
  # the log event was picked up by the shipping
  # agent and not when the request actually
  # happened
  date {
    match => ["time", "dd/MMM/YYYY:HH:mm:ss Z"]
    target => "@timestamp"
    remove_field => "time"
  }

[...]

[...]

  # Tag and normalize pre-structured IIS web server logs
  if [event][provider] == "Microsoft-Windows-IIS-Logging" {
    # Normalize field names by creating copies and tag as
    # web server log
    mutate {
      add_tag => [
        "web_server_access"
      ]
  
      # We're making a copy of the fields instead of changing
      # their names, as some already existing queries may
      # rely upon them
      copy => {
        "[winlog][event_data][c-ip]" => "[source][ip]"
        "[winlog][event_data][csUser-Agent]" => "raw_user_agent"
        "[winlog][event_data][sc-status]" => "response_code"
      }
  
[...]

[...]

  # If event is a web server access log, parse
  # user agent string and extract information,
  # such as browser and operating system version
  if ("web_server_access" in [tags]) {
    useragent {
      source => "raw_user_agent"
      target => "user_agent"
    }
  }
  
  # If event contains a field called "source.ip,
  # lookup  IP address in GeoIP database to find
  # information about its approximate location
  if [source][ip] {
    geoip {
      source => "[source][ip]"
    }
  }

[...]

[...]

  # The remaing parts of the filter section
  # are just used to control in which index
  # the event will be stored in
  if ("web_server_access" in [tags]) {
    mutate {
      add_field => {
        "index_suffix" => "web_servers"
      }
    }
    	
  } else if [type] == "json_events" {
    mutate {
      add_field => {
        "index_suffix" => "json" 
      }
    }

[...]

output {
  opensearch {
    hosts => ["https://opensearch:9200"]

    # Use variable specified during event
    # filtering and date expression to
    # control which index is used for data
    # storage. The date expression will
    # result in new indexes being created
    # each day, making rotation/retention
    # easier to control
    index => "logs-%{index_suffix}-%{+YYYY.MM.dd}"

    user => "logger"
    password => "G0d="
    ssl => true
    ssl_certificate_verification => false
  }
}

While extremely flexible, Logstash is
quite complex and resource hungry.

Alternatives exist, such as
OpenSearch Data Prepper
and Fluentd.

How do we get our logs to Logstash?

Drop em' Beats!

Beats

Family of light-weight log agents.

Filebeat, Winlogbeat, Auditbeat,
Metricbeat, Packetbeat, Heartbeat...

Development led by Elastic,
available under an open-source license.

We'll talk more about these later!

I'm tired of staring at text
in the terminal!

Let's have a look at
OpenSearch Dashboards.

OpenSearch Dashboards

Fork of Elastic's "Kibana".

Web application that exposes analytics
capabilities and provides several data
visualization features.

The main interface humans use for
interaction with OpenSearch.

We've just dipped our toes so far.

During the rest of the course we'll
continue exploring!

OpenSearch basics

Let's search that data!

If you have an OpenSearch instance running,
chances are that you wanna make some searches.

We'll look at some common use-cases and how
its searching super-powers can help us.

(We'll start with general usage before
getting into logging-specific considerations)

{
  "cve_identifier": "CVE-2023-20273",
  "description": "Management interface code injection",
  "cvss_score": 7.2,
  "included_in_kev": true,
  "category": "Remote code execution",
  "date_published": "2023-10-25",
  "date_updated": "2023-11-15",
  "affected_software": [
    "Cisco IOS",
    "Cisco IOS XE"
  ]
}

$ for CVE_FILE in CVE-*; do
  curl "${BASE_URL}/myvulns/_doc/${CVE_FILE}?pretty" \
  --request PUT \
  --header "Content-Type: application/json" \
  --data @${CVE_FILE}
done

$ curl "${BASE_URL}/myvulns/_search?pretty" --request GET

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 8,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    
[...]

    
[...]

    "hits" : [
      {
        "_index" : "myvulns",
        "_id" : "CVE-2021-1675",
        "_score" : 1.0,
        "_source" : {
          "cve_identifier" : "CVE-2021-1675",
          "description" : "Code injection in print spooler",
          "cvss_score" : 9.3,
          "included_in_kev" : true,
          "category" : "Remote code execution",
          "date_published" : "2021-06-08",
          "date_updated" : "2022-08-01",
          "affected_software" : [
            "Microsoft Windows"
          ]
        }
[...]

$ curl \
  "${BASE_URL}/myvulns/_doc/CVE-2023-36036?pretty" --request GET

{
  "_index" : "myvulns",
  "_id" : "CVE-2023-36036",
  "_version" : 1,
  "_seq_no" : 4,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "cve_identifier" : "CVE-2023-36036",
    "description" : "Flaw in Windows Cloud files driver",
    "cvss_score" : 7.8,
    "included_in_kev" : true,
    "category" : "Privilege escalation",
    "date_published" : "2023-11-14",
    "date_updated" : "2023-11-14",
    "affected_software" : [
      "Microsoft Windows"
    ]
  }
}

$ curl \
  "${BASE_URL}/myvulns/_doc/CVE-2023-36036/_update" \
  --request POST \
  --header "Content-Type: application/json" \
  --data '{"doc": {"cvss_score": 7.5}}'

{
  "_index" : "myvulns",
  "_type" : "_doc"
  "_id" : "CVE-2023-36036",
  "_version" : 2,
  "result" : "updated",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 2,
  "_primary_term" : 1
}

Let's get searching with
Lucene Query Language!

(AKA "Query DSL")

{
  "query": {
    "range": {
      "date_updated": {
        "gte": "2023-11-01"
      }
    }
  }
}

$ curl \
  "${BASE_URL}/myvulns/_search?pretty" \
  --request GET \
  --header "Content-Type: application/json" \
  --data @query.json

[...]

"hits" : {
  "total" : {
    "value" : 3,
    "relation" : "eq"
  },
  "max_score" : 1.0,
  "hits" : [
    {
      "_index" : "myvulns",
      "_id" : "CVE-2023-20273",
      "_score" : 1.0,
      "_source" : {
        "cve_identifier" : "CVE-2023-20273",
        "description" : "Management interface code injection",
        "cvss_score" : 7.2,
        "included_in_kev" : true,
        "category" : "Remote code execution",
        "date_published" : "2023-10-25",
        "date_updated" : "2023-11-15",
        "affected_software" : [
          "Cisco IOS",
          "Cisco IOS XE"
        ]
      }
    },

[...]

{
  "query": {
    "match": {
      "affected_software": {
        "query": "windows"
      }
    }
  }
}

[...]

"hits" : {
  "total" : {
    "value" : 2,
    "relation" : "eq"
  },
  "max_score" : 1.5532583,
  "hits" : [
    {
      "_index" : "myvulns",
      "_id" : "CVE-2021-1675",
      "_score" : 1.5532583,
      "_source" : {
        [...]
        "affected_software" : [
          "Microsoft Windows"
        ]
      }
    },
      
[...]

{
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "date_published": {
              "gte": "2018",
              "lte": "2022"
            }
          }
        }
      ],
      "should": [
        {
          "term": {
            "included_in_kev": {
              "value": true
            }
          }
        },
        {
          "range": {
            "cvss_score": {
              "gte": 5.5
            }
          }
        }
      ]
    }
  }
}

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 2.3254223,

[...]

[...]

"hits" : [
  {
    "_index" : "myvulns",
    "_id" : "CVE-2021-1675",
    "_score" : 2.3254223,
    "_source" : {
      "cve_identifier" : "CVE-2021-1675",
      "description" : "Code injection in print spooler",
      "cvss_score" : 9.3,
      "included_in_kev" : true,
      "category" : "Remote code execution",
      "date_published" : "2021-06-08",
      "date_updated" : "2022-08-01",
      "affected_software" : [
        "Microsoft Windows"
      ]
    }
  },

[...]

[...]

  {
    "_index" : "myvulns",
    "_id" : "CVE-2019-0233",
    "_score" : 1.0,
    "_source" : {
      "cve_identifier" : "CVE-2019-0233",
      "description" : "Faulty validation in file upload",
      "cvss_score" : 5.0,
      "included_in_kev" : false,
      "category" : "Denial of service",
      "date_published" : "2020-09-14",
      "date_updated" : "2022-04-18",
      "affected_software" : [
        "Apache Struts",
        "Oracle MySQL Enterprise Monitor",
        "Oracle Financial Services Data Hub"
      ]
    }
  }

[...]

Searches in OpenSearch are
"queries", "aggregations"
or a combination of both.

Queries return matching documents.

Aggregations returns statistics
about document fields.

They can be combined to filter
data for statistical analysis.

{
  "aggs": {
    "vulnerability_categories": {
      "terms": {
        "field": "category"
      }
    }
  }
}

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "Text fields are not optimised for operations
                    that require per-document field data like
                    aggregations and sorting, so these operations
                    are disabled by default. Please use a
                    keyword field instead. [...]

$ curl "${BASE_URL}/myvulns/_mapping?pretty" --request GET

{
  "myvulns" : {
    "mappings" : {
      "properties" : {
        [...]
      
        "cvss_score" : {
           "type" : "float"
        },
        "date_published" : {
          "type" : "date"
        },
        "date_updated" : {
          "type" : "date"
        },
        "category" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
        }
[...]

{
  "aggs": {
    "vulnerability_categories": {
      "terms": {
        "field": "category.keyword"
      }
    }
  }
}

[...]

"aggregations" : {
   "vulnerability_categories" : {
     "doc_count_error_upper_bound" : 0,
     "sum_other_doc_count" : 0,
     "buckets" : [
       {
         "key" : "Remote code execution",
         "doc_count" : 3
       },
       {
         "key" : "Privilege escalation",
         "doc_count" : 2
       },
       {
         "key" : "Authentication bypass",
         "doc_count" : 1
       },
       {
         "key" : "Cross-site scripting",
         "doc_count" : 1
       },
       {
         "key" : "Denial of service",
         "doc_count" : 1
       }
     ]

[...]

{
  "query": {
    "match": {
      "affected_software": {
        "query": "Juniper Junos"
      }
    }
  },
  "aggs": {
    "vulnerability_categories": {
      "terms": {
        "field": "category.keyword"
      }
    }
  }
}

[...]

"aggregations" : {
  "vulnerability_categories" : {
    "doc_count_error_upper_bound" : 0,
    "sum_other_doc_count" : 0,
    "buckets" : [
      {
        "key" : "Authentication bypass",
        "doc_count" : 1
      },
      {
        "key" : "Remote code execution",
        "doc_count" : 1
      }
    ]
    
[...]

Besides Lucene Query Language,
OpenSearch provides plugins for other ways
to express your searches/aggregations.

Depending on your preferences, you can use
Dashboard Query Language,
Pipe Processing Language or
Structured Query Language.

Under the hood, these get translate to
LQL with varying degrees of success.

(Sigma is also an option!)

Are you missing some of that eye candy?

Let's do some searches and visualizations
in OpenStack Dashboards!

Wrapping up

Don't be scared, you'll have plenty
of time to get friendly with OpenSearch.

In your day-to-day digging, DQL is
likely the best choice. For advanced
queries and aggregations, learning
LQL is a worthwhile investment.

Having a hard time finding documentation
and guides? Try Googling for Elasticsearch.

Lab: OpenSearch

Data filtering and visualization

Lab description

Graded exercise to use Logstash for data extraction and OpenSearch Dashboards
for visualization.

For detailed instructions, see:
"resources/labs/opensearch/README.md".

Remember to download the latest version of
the resources archive! ("log.zip")

Enriching logs

Aiding our data analysis

"Enrichment" is the process of improving
the value of our logs.

Often this means providing useful context
for analysts and machines alike.

We've already played around with adding
GeoIP information.

Let's look at some more examples and
how to implement them in OpenSearch.

What about that source/dest?

• IP reputation
• IP type (residential, cloud, proxy, etc.)
• Current host patch level
• Vulnerability scan and/or Shodan results
• All kinds of CMDB data!

Let's not forget humans!

• Role description
• Employment location / Timezone
• Occurrence in data leaks
• Contact information

Enrichment can be performed during
ingestion or at search-time.

Like with field parsing, both have
their pros/cons.

Current relevance VS Historic accuracy.

"This IP address is used by evildoers now"
VS
"This IP address was used by evildoers then".

Useful filter plugins

• GeoIP and user agent
• DNS (forward/reverse lookups)
• "Translate"
• JDBC and Memcached
• HTTP client (for APIs)

...and as always, "ruby"!

Why may DNS be interesting?

# Forward lookup
$ host suspicious.example.com

suspicious.example.com has address 93.184.215.14
suspiciousexample.com has IPv6 address
2606:2800:21f:cb07:6820:80da:af6b:8b2c

# Reverse lookup
$ host 93.184.215.14

14.215.184.93.in-addr.arpa domain name pointer
suspicious.example.com.

Erghh - less talk, more examples!

/var/ioc/evil_ip.csv ("key-value")

157.245.96.121,Observed in logs during 2025 Xmplify incident
185.120.19.98,Associated with Explum spear phishing campaign
194.61.40.74,Have been trying to brutforce our VPN for years!

Logstash filter pipeline

[...]

if [source][ip] {
  translate {
    source => "[source][ip]"
    target => "ip_related_to_incident"
    dictionary_path => "/var/ioc/evil_ip.csv"
  }
}

[...]

[...]

"must": [
  {
    "match_phrase": {
      "tags.keyword": "web_server_access"
    }
  },
  {
    "exists": {
      "field": "ip_related_to_incident"
    }
  }
]

[...]

[...]

"hits" : [
  {
    "_index" : "logs-web_servers-2025.10.19",
    "_id" : "6C0B74sB7PKVx7m-L2xx",
    "_score" : 1.0048822,
    "_source" : {
      "url" : "/internal/nuke_control.aspx",
      "ip_related_to_incident" : "Associated with Explum spear phishing campaign",
      "source" : {
        "ip" : "185.120.19.98",
      [...]

While OpenSearch relies heavily on
parsing/enrichment during ingestion,
there are some neat things we can do
at search-time.

{
  "known_evil_ip_addresses": [
    "34.76.96.55",
    "198.235.24.39",
    "157.245.96.121",
    "143.198.117.36"
  ],
  "scripted_http_clients": [
    "curl",
    "Go-http-client",
    "Python Requests",
    "Nmap Network Scanner"
  ]
}

$ curl \
  "${BASE_URL}/mylookupdata/_doc/ioc" \
  --request PUT --data @ioc.json \
  --header 'Content-Type: application/json'

{
  "query": {
    "bool": {
      "must": [
        {
          "match_phrase": {
            "tags.keyword": "web_server_access"
          }
        },
        {
          "terms": {
            "source.ip": {
              "index": "mylookupdata",
              "id": "ioc",
              "path": "known_evil_ip_addresses"
            }
          }
        }
      ],
      "must_not": [
        {
          "match": {
            "raw_user_agent": {
              "query": "CensysInspect"
            }
          }
        }
      ],
      "should": [
        {
          "terms": {
            "user_agent.name": {
              "index": "mylookupdata",
              "id": "ioc",
              "path": "scripted_http_clients"
            }
          }
        }
      ]
    }
  }
}

[...]

   "must": [
     {
       "match_phrase": {
         "tags.keyword": "web_server_access"
       }
     },
     {
       "terms": {
         "source.ip": {
           "index": "mylookupdata",
           "id": "ioc",
           "path": "known_evil_ip_addresses"
         }
       }
     }
   ],

[...]

[...]

  "must_not": [
    {
      "match": {
        "raw_user_agent": {
          "query": "CensysInspect"
        }
      }
    }
  ],

[...]

[...]

  "should": [
    {
      "terms": {
        "user_agent.name": {
          "index": "mylookupdata",
          "id": "ioc",
          "path": "scripted_http_clients"
        }
      }
    }
  ]

[...]

[...]

  "hits" : {
    "total" : {
      "value" : 28,
      "relation" : "eq"
    },
    "max_score" : 2.0053382,
    "hits" : [
      {
        "_index" : "logs-web_servers-2025.10.27",
        "_id" : "53JE6osBQrucVyA5EqK1",
        "_score" : 2.0053382,
        "_source" : {
          "request_method": "GET"
          "request_path" : "/admin.php",
          "raw_user_agent" : "curl/8.1.2",
          "source" : {
            "ip" : "143.198.117.36",
            "geo" : {
              "country_iso_code" : "US",
              "continent_code" : "NA",
              "country_name" : "United States"
            }

[...]