Kryesh

What are the disks and how full is the pool?

The biggest thing imo is to have the program do something you want/need, without that motivation becomes difficult and having a problem to solve gives you something to design for. I've been working away at my own side project to learn rust for a while building a log search server Crystalline

I ended up using Poem for the HTTP server, Leptos + Tailwind/DaisyUI for the frontend, and Tantivy for the main data storage

I'm currently using the fluentbit http output plugin, fluentbit can act as an otel collector with an input plugin which could then be routed to the http output plugin. Long term I'll probably look at adding it but there's other features that take priority in the app itself such as scheduled searching and notifications/alerting

Applications like metrics because they're good for doing statistics so you can figure out things like "is this endpoint slow" or "how much traffic is there"

Security teams like logs because they answer questions like "who logged in to this host between these times?" Or "when did we receive a weird looking http request", basically any time you want to find specific details about a single event logs are typically better; and threat hunting does a lot of analysis on specific one time events.

Logs are also helpful when troubleshooting, metrics can tell you there's a problem but in my experience you'll often need logs to actually find out what the problem is so you can fix it.

Thanks! definitely aiming for a stupid easy installation/management for the app itself; but in my experience getting a wide range of supported log sources is no small feat. I've been using fluentbit to handle collection from different sources and using the following has been working well for me:

• docker 'journald' log driver
• fluentbit 'systemd' input
• fluentbit 'http' output like the one in the readme

with that setup you can search for container logs by name which works great with compose:

or process logs from an nginx container like this to see traffic from external hosts:

I'll add a more complete example to the docs, but if you look in the repo there's a complete example for receiving and ingesting syslog that you can run with just "docker compose up"

Oh I wasn't using it as a full recursive resolver - just reading the resolv.conf set by docker and sending requests

More good points, thank you! for trust-dns-resolver that's a relic from a previous iteration that had polling external sources and needed to resolve dns records. Since i haven't gotten around to re-implementing that feature it should be removed. As for why - I actually needed to bring my own resolver since the docker container is a scratch image containing only some base directories and the server binary so there isn't any OS etc to lean on for things like dns; means that the whole image is ~15.5MB which is nice and negates a whole class of vulnerabilities.

Understood that your actual point is to document this stuff and not answer the trivia question though

Thanks! it's definitely got a way to go before it's remotely competitive with any of the enterprise solutions out there, but you make a good point about having comparisons so I'll look at adding it.

I'm basically building it to have a KQL/LogScale/Splunk/Sumologic style search experience while being trivial to deploy (relative to others at least...) since I miss having that kind of search tooling when not at work; but I don't want to pay for or maintain that kind of thing in a lab context. It creates a Tantivy index per day for log storage (with scoring and postings disabled for space savings).

In the end my main goal of the project was as a vehicle to get better at programming with, and if I get a tool I can use for my lab then that's great too lol.