Users of systemd-based Linux distributions are frustrated by journald's bloated binary log format, excessive disk consumption, and difficult-to-use interfaces. LogSculpt replaces journald with a lightweight, human‑readable, compression‑aware log collector that stores logs in a minimal on‑disk format and offers a fast, web‑based query UI, dramatically reducing disk usage while improving debugging speed.
I recently looked into disk usage of journald and was also shocked. My next step towards peace of mind is https://www.devuan.org/os/init-freedomWill try it out as next distro for my Debian system, longtime experience with Void Linux (runit) on another box is great.
systemd-journald has one of the most deranged log file formats I have ever dealt with, and one of the worse user interfaces, too.I am not again binary logs, or logs in a database. It's just yet another time I deal with good ideas implemented horribly, horribly badly when it comes to systemd.
The two I most often see in Ubuntu's dmesg are:audit - appears to be some sort of AppArmor logging?br[] - bridge interface docker uses consistently rebuilds itself? May be related to docker compose networking.
I'm probably the main person responsible for making journald usable at all.But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon.Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog.My impression back then was the on-disk format dispersed the information too much within the same file, and those individual datums being written at discontiguous offsets were quite small, far smaller than an
This issue report feels like it ought to be accompanied by a fix. If you think you can do better than journald's existing format, propose a new one with tests to prove it. GenAI makes this much easier than it used to be.
>This issue report feels like it ought to be accompanied by a fix.systemd is a stewarded FOSS, which means there's a team behind it, who are getting paid, and develop this software with release cycles, backwards compatibility guarantees, architectural decisions, and such.These people know better. I usually only prepare fixes for FOSS one-man-show which have little to no maintenance, otherwise I prefer professionals to handle it. Sometimes "suggestion" PR is worse than a triaged issue IMO.
Uh, who said anything about writing 122880 rows every time you do a single insert into DuckDB? There's a WAL. Consolidation happens in big chunks. (And it's not like journald log rotation is somehow better than WAL consolidation.)
I would much rather that they had used an existing database file format. Sqlite3 is robust and already present in the default installation of most Linux distributions. Querying system logs with SQL would be cool and likely faster than using the sd_journal API with all it's weird quirks.I don't much like that it's possible for a log entry to have multiple fields with the same name, neither will journalctl reveal their existence without passing additional flags. Seems a bit insane to me.
> This is a huge issue for btrfs and for COW FS in general, because they have massive write amplification for small and frequent writesHave you considered using a different fstype like XFS for this? btrfs is good for homedirs, but I wouldn't necessarily use it for other filesystems (/usr, /var, etc.)
SQLite here is okay, but DuckDB or LevelDB would be better. Either way, no need to invent a new storage format.
No, not at all. Parquet is great for building static content incrementally, but it’s not great for this: the aim is durable writes (it’s a log system after all), but with parquet you need large row group batches. Worst case (low log volumes and a time-based flush) you’d end up with loads of tiny row groups.You also need metadata in the file footer, so you can’t query it until the file is “done”. When is that?
If it ever worked like that, then gradual accretion of (mis)features and misguided enhancements pretty clearly broke it. Based on my years and years and years of reading about and using the output of the Systemd Project, there's really clearly no Linus Torvalds on the project to hold the line on software quality.
What is the real-world use for these features? Who is this built for?Modern drives will read data at 500MB/s, sometimes even more. Your log files are approaching tens if not hundreds of gigabytes before a sequential read stops being a viable option. Tinies modicum of partitioning by date and source basically makes it a complete nothingburger.
journald is awful for many reasons, but what makes it worse is that everything running on your machine thinks it has any rights to dump all the logs it wants unprompted. Open a file picker and kio will decide it's a good idea to spam tens or hundreds of thousands of entries into it a day, listing every single file you have in a directory with some log such as "No node found for item that was just removed" and that has zero impact to the user whatsoever. You almost need to keep a script tracking all the journal floods for every new service to make sure it's not treating your system log as its d
Someone should implement a new operating system that can efficiently handle text processing.It could have some simple tools that let you generate reports, display them on screen, and compose tools for that sort of thing in a natural way.We could call it UNIX.
Or, you could write a plain text file.Yes, that means the FS will sometimes punch nulls towards the tail of the log. However, it is the lowest latency / write amplification way to get stuff on disk (other than a blocked compression format, which would be a small change to syslog), so if the text file gets holes punched in it, the journalctl file would be truncated before the hole anyway in practice.If you really care about nulls in logs for ideological reasons, you could write a few lines of code that finds the first stream of nulls in the text file, then truncates there.In practice, no one w
> Anyone with any sort of scale where you actually need indexing immediately drops journald and uses loki or elasticsearch instead. Journald is not even remotely a contender in that space.That I agree with. I don't personally use journalctl much these days, particularly now that practically everything's a container and all their logs are getting shipped off-host for indexing. But I get why, 14 years ago, it was considered a good idea.
You: Nobody's talking about an obligation here. Also you: If you think you can do better than journald's existing format, propose a new one with tests to prove it. The fact that you're powerless to enforce an obligation doesn't change the fact that you're talking about creating an obligation.> I'm not entirely sure this is a bug so much as an inefficiency report.Performance bugs absolutely are bugs... especially when they're in a long-running corporation-backed project that presents itself as the project atop which all Linux systems should be built.
The journald schema is surprisingly wide and has a bunch of boilerplate, and one of the goals is to keep the on-disk size under control (and an efficient format directly reduces write amplification). And you kind of want a format that allows a reader to just read the file without blocking concurrent writes. And the ability to use third-party tools to easily read the format is quite nice.SQLite gets the last one but misses on the first two (although WAL and the improved read-only support in 3.20+ mostly gets #1). DuckDB might be decent except that you would need to connect through the daemon
SQLite has the problem of a malicious reader being able to hold up writers. Maybe that's fine in most cases, but in a system log, I don't think that's acceptable. IMHO, options are to indirect through a daemon anyway (e.g. using Quack) or do a lot of engineering to make it possible for an unprivileged reader to open() the log file and read it in such a way that it can't interfere with privileged writers.Your LSM compaction strategy is going to have to solve the same problem anyway, isn't it? DuckDB is an LSM compaction strategy of this form, already done.
"Don't come to me with problems, come to me with fixes." has always been a thing. You shouldn't take it too literally or personally. For "fixes" read alternatives or suggestions. e.g. hook in a senior engineer that you know is intimate with the system.
Dont like SSD wear for no reason. I wont look at those logs anyway on my personal gaming pc so its all useless.
journald does do a rewrite of the log file on rotation, so you're paying that IO anyway even if you ignore the dumb hash table updates.https://github.com/systemd/systemd/blob/8f4cd7de43d1e6e94687...WAL writeback is at least principled and efficient. It works out to being equivalent to the custom Parquet-rotation things others mention, but already implemented and working.So, yes, WAL for logs, because LSM is the design everyone converges on.
I can confirm this, used it for many years, 3 keywords: efficient, reliable, useful.all 3 missing on journald,in my experience i saw it inefficient also on configuration level, unreliable because of loosing loglines on crash or reboot and not useful since to look at logs i need 3 commands, verbose parameters and 5 google search to find them.syslog experience? very efficient also on heavy load production instances, never lost a log, pipe grep and jq and you have the info you need.so what I experienced is that a default linux install was shipping a rock solid logging system by default, reliable
So, yes, journald does in fact do bulk copies of log files on rotate. I'd expect someone familiar with journald's storage layer to know that. btrfs is hardly some fringe FS and its COW-flag behavior is documented and well-known.
WHen I provisioned a mid-range dedicated box recently, I went FreeBSD almost entirely so as to not have to touch or deal with anything related to systemd, the worst the thing to ever happen to linux.
Recommended execution roadmap for "LogSculpt: Linux Log Stream Optimizer"
Examine the 26 harvested raw posts to map specific feature complaints, workflow workarounds, and user friction points.
Build a minimalist solution focused exclusively on solving "Users of systemd-based Linux distributions are frustrated by journald's bloated binary log format, excessive d..." without feature bloat.
Directly engage users in subreddits and developer forums who expressed frustration to offer early access beta invites.
Introduce structured subscription pricing matching market urgency score (75%).
Developers today trade rapid coding for sluggish execution, especially when using GC-heavy languages that suffer with heavy allocation. HasteRun delivers a SaaS compiler that instantly transforms high‑level code into highly optimized machine binaries, eliminating GC overhead and giving teams the speed of scripting with near‑C performance.
Designers struggle with clunky rectangle selects and manipulation in both photo editors and CAD suites, causing wasted time and frustration. SnapSelect offers an AI‑driven, cross‑platform selection layer that replaces drag‑and‑drop with gesture‑based, context‑aware selection, dramatically simplifying workflow in both 2D and 3D design environments.
Heavy-duty applications and virtualization layers are introducing crippling latency and CPU bottlenecks on Apple Silicon hardware. LatencyZero optimizes system-level resource scheduling to eliminate software slowdowns and restore native-speed execution.
Developers building event‑driven, serverless stacks are frustrated by webhook‑only integrations that force them to maintain extra infrastructure. EventSync converts those webhooks into a simple, pull‑based /events endpoint, letting teams retrieve change events on demand with minimal setup and zero webhook maintenance.