Goofed Home

Woke up today to the homeserver being unresponsive. Couldn’t SSH, no video out when I connected a monitor, and even the reset button didn’t do anything. Had to hold the power button to shut it down.

/var/log/syslog doesn’t show anything interesting other than the issue happened at just after 4am. Log

2026-02-27T03:55:01.481794-08:00 blackbox CRON[1743418]: (www-data) CMD (/usr/bin/php8.3 /mnt/MONSTERDRIVE/pixelfeddata/pixelfed/artisan schedule:run >> /dev/null 2>&1)
2026-02-27T04:00:00.198504-08:00 blackbox smartd[2126]: Device: /dev/sdd [SAT], CHECK POWER STATUS spins up disk (0x81 -> 0xff)
2026-02-27T04:00:00.291853-08:00 blackbox systemd[1]: Starting sysstat-collect.service - system activity accounting tool...
2026-02-27T04:00:00.298344-08:00 blackbox systemd[1]: sysstat-collect.service: Deactivated successfully.
2026-02-27T04:00:00.298523-08:00 blackbox systemd[1]: Finished sysstat-collect.service - system activity accounting tool.
2026-02-27T04:00:00.299608-08:00 blackbox kernel: kauditd_printk_skb: 8 callbacks suppressed
2026-02-27T04:00:00.299613-08:00 blackbox kernel: audit: type=1130 audit(1772193600.298:798916): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=sysstat-collect comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
2026-02-27T04:00:00.299615-08:00 blackbox kernel: audit: type=1131 audit(1772193600.298:798917): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=sysstat-collect comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
2026-02-27T04:00:01.923610-08:00 blackbox kernel: audit: type=1101 audit(1772193601.922:798918): pid=1744810 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='op=PAM:accounting grantors=pam_permit acct="www-data" exe="/usr/sbin/cron" hostname=? addr=? terminal=cron res=success'
2026-02-27T04:00:01.923614-08:00 blackbox kernel: audit: type=1103 audit(1772193601.922:798919): pid=1744810 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='op=PAM:setcred grantors=pam_permit,pam_cap acct="www-data" exe="/usr/sbin/cron" hostname=? addr=? terminal=cron res=success'
2026-02-27T04:00:01.923615-08:00 blackbox kernel: audit: type=1006 audit(1772193601.922:798920): pid=1744810 uid=0 subj=unconfined old-auid=4294967295 auid=33 tty=(none) old-ses=4294967295 ses=50544 res=1
2026-02-27T04:00:01.923615-08:00 blackbox kernel: audit: type=1300 audit(1772193601.922:798920): arch=c000003e syscall=1 success=yes exit=2 a0=7 a1=7fff81d75200 a2=2 a3=0 items=0 ppid=2654 pid=1744810 auid=33 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=50544 comm="cron" exe="/usr/sbin/cron" subj=unconfined key=(null)
2026-02-27T04:00:01.923616-08:00 blackbox kernel: audit: type=1327 audit(1772193601.922:798920): proctitle=2F7573722F7362696E2F43524F4E002D66002D50
2026-02-27T04:00:01.924259-08:00 blackbox CRON[1744811]: (www-data) CMD (/usr/bin/php8.3 /mnt/MONSTERDRIVE/pixelfeddata/pixelfed/artisan schedule:run >> /dev/null 2>&1)
2026-02-27T04:00:01.924614-08:00 blackbox kernel: audit: type=1105 audit(1772193601.923:798921): pid=1744810 uid=0 auid=33 ses=50544 subj=unconfined msg='op=PAM:session_open grantors=pam_loginuid,pam_env,pam_env,pam_permit,pam_umask,pam_unix,pam_limits acct="www-data" exe="/usr/sbin/cron" hostname=? addr=? terminal=cron res=success'
2026-02-27T04:00:01.925610-08:00 blackbox kernel: audit: type=1110 audit(1772193601.924:798922): pid=1744811 uid=0 auid=33 ses=50544 subj=unconfined msg='op=PAM:setcred grantors=pam_permit,pam_cap acct="www-data" exe="/usr/sbin/cron" hostname=? addr=? terminal=cron res=success'
2026-02-27T04:00:02.357616-08:00 blackbox kernel: audit: type=1104 audit(1772193602.356:798923): pid=1744810 uid=0 auid=33 ses=50544 subj=unconfined msg='op=PAM:setcred grantors=pam_permit acct="www-data" exe="/usr/sbin/cron" hostname=? addr=? terminal=cron res=success'
2026-02-27T09:23:35.786375-08:00 blackbox systemd-modules-load[904]: Inserted module 'dm_multipath'

Would something like this be a direct hardware failure? Like a power supply hiccup or something? It happening at 4am coincides with my electric car starting to charge, but the server is on a dedicated 20A circuit and behind a battery backup. I also don’t see any power issues on my Sense monitor at that time though it has limited resolution.

Mainboard is a Supermicro H13SAE-MF and I’m using ECC RAM.

I’ve been running this hardware for over a year and never had this issue, but I’m running out of places to look.

Might be time to finally get IPMI working.

So you’re saying there’s a market for lead lined PC cases? 🤔

That might introduce more issues than help. If high speed particles impact your shielding, you might get a “particle shower” from the impact on your electronics. Radiation Hardening is part of the design of the chips - mainly creating less dense structures with bigger transistors, because they don’t flip as easily as the very small gates on a H200. That’s also the reason why most space based computers have the processing power of a system around 2005.

I just installed NetBird on a VPS using the Self-Hosting Quickstart Guide. Now I want to connect the VPS using Netbird to another client. When I also use Docker to register the VPS as a Netbird peer the whole network gets messed up because now the server and the client try to manage the network. So how am I supposed to register the VPS as a netbird peer to connect it to other peers?

I’m not quite sure what you are asking but I run the Netbird management containers containers on a server and also run a native client alongside them to have the server itself also perform as a peer. Is that what you are looking to do above?

Yes, but I also want to run the client in a container and the docs recommend to run the container using network_mode: host. And I suspect this creates a conflict in networks. So I want to have Netbird server, Netbird client and Nginx Proxy Manager all in containers share the same network.

Hey there, it’s me again with my cursed project. Last time is said “i basically reinvented Kubernetes”. But the voices won and I legit did.

Last time it was a cursed novelty. A random script made by some autistic dude with too much time on its hand.

Now it’s become its own project, with ecosystem and overpriced .io domain. For no reason other than : It’s cursed, but it works beautifully.

Every Kind is handled by its distinct code. Everything is pluggable, nothing is hardcoded. The next layer of hell is for someone else to write Docker Swarm extensions. Won’t be me.

I am, again, very sorry. Sorry for releasing this thing into the world as a complete, working, product.

And sorry for keeping spamming it. I will stop, i promises (the voices will never)

Ξέρεις την ελληνικά γλώσσα;

No, no hablo Griego.

Not in there: - https://github.com/dannymcc/bluehood (alpha) - https://github.com/p2r3/convert

Yep. I got an email from them yesterday. My lil box is going from just under 4 USD to 5 USD per month.

Yes but the RSS feed for non-subscribers is just the announcement of the post. I still have to go to the site to read the whole newsletter, which is fine.

A domain name I was interested in expired in January this year. It was previously registered at Squarespace.com.

Why is it still unavailable to purchase despite being more than a month since its expiry?

Not sure if relevant but I checked the expiry date at: whatsmydns.net/domain-expiration

If it’s not a cooldown period as that other guy said, you may contact the scammer new owner, he will ask for a billion dollars and it’s up to you whether that domain was important enough. Consider finding a new one right now if you can.

Basically many domain providers will hold onto domains for a little while after it expires.

Some like namecheap also advertise the domain names to peddle-man companies that will somehow buy temporary access to the domain after your extortion recall window expires.

To continue the namecheap example, when your namecheap domain expires, it gives you a lapse window where you can pay like double the cost of the domain renewal to reclaim it. If you don’t reclaim it during that window they give it to a middleman whom will somehow buy a 2 or 3 months domain lease for it. They will put it on a “site for sale” broker page and will charge yo easily 100x what you paid for the domain if you wanted it back.

suggestions to improve appreciated!

i download it at beginning of the week, convert to ical, a bit inconvinient, thinking of writing a script to automate it

Still the same, a starter home page.

Here are some cool examples I was looking at:

https://github.com/zardoy/minecraft-web-client — Minecraft in your browser, complete with connections to servers.

https://github.com/inolen/quakejs — quake 3 in your browser, has multiplayer as well.

Any other good examples? or good lists?

probably also selfhosted.

Here is a link do selfhosting it: https://github.com/TeamHypersomnia/Hypersomnia/blob/master/README_SERVER.md#docker-setup

I know wikis have been discussed here before, but I wanted to add my two cents after shopping around for a wiki at work and for personal use.

Obsidian

Pros

• plain text storage format
• great at gathering disorganized thoughts without imposing a rigid structure

Cons

• closed source
• many features that arguably define a wiki are either absent or paywalled, like easy sharing, collaboration, and versioning

Mediawiki

Pros

• it’s the wiki. Everyone’s used and possibly edited a Wikipedia page.
• version history
• close to Obsidian in terms of “write now, organize later”
• Probably the nicest-looking FOSS wiki platform out of the box
• a lot of the features that Obsidian paywalls are built in, like multi user support and version history

Cons

• Articles not stored in plain text
• Has its own markup. Granted Mediawiki predates Markdown but the table syntax is horrendous. The Mediawiki help page on the matter actually tries to dissuade you from using tables and notes that the markup is ugly.
• Extensions are annoying to install
• Absolutely zero access control. You can even edit other people’s user pages. There’s no way to hide sections of a wiki from the public or from particular groups of users.
• It tries to be all things to everyone. While this makes it versatile, it also means doing a particular thing probably requires knowledge of CSS or Mediawiki’s own templeting syntax. Sometimes I just want to have an info box that doesn’t clutter the source code of a page.

Dokuwiki

Pros

• Access control finally!
• Plain text files
• Easy to create namespaces, which Mediawiki also has but doesn’t want you to go crazy making your own.
• While it’s not Markdown, the markup is nicer than Mediawiki IMO. The table syntax at least is miles better

Cons

• Uglier than sin. Yes even many of the templates (themes) on offer aren’t much better. The Bootstrap 3 template seems particularly popular, and while it’s a marked improvement in most areas, like a lot of frontends that use those bootswatch pallets there are dusty corners that don’t work, like black text on a black background.
• Some stuff like tags and moving pages have to be achieved via plugins. Seriously you can’t even rename a page?
• Mutilates article titles. Makes everything lowercase and replaces non alphanumeric chars with underscores (or something else configurable).

Bookstack

Pros

• It looks good I guess. Haven’t spent much time with it.
• Yay markdown!
• Also has access control

Cons

• Also not plain text
• remember earlier when I talked about “write now, organize later”? Bookstack holds a gun to your head and forces you to use its shelf>book>chapter>page organization system. I know some people thrive under this limitation, but I don’t.

Other wikis I’ve tried but not to the same extent

Wiki.js

IDK, I don’t know much about this one, but don’t like the workflow of making new pages.

Gollum

Really simple, which is both good and bad.

An Otter Wiki (the article seems to be part of the name)

A lot like Gollum. Doesn’t indicate when you link to a nonexistent page. No support for article tags.

Pepperminty wiki

Looks cool but it’s abandoned

Tiddlywiki

Steep learning curve but pretty versatile. It’s a single HTML file so you can host it on something like Neocities. Really rudimentary search functions

A little bit of both. I ran a private wiki for writers to collaborate on for a project. I was doing other tech stuff for the team so it was my job to deal with it. Keeping it updated was a chore and actually using it was finicky.

For example, there was an issue we ran into where we wanted a dynamic table that pulled from other pages. Think of a shopkeeper inventory or something similar where each item was another page. Displaying an item worked fine the first time you pulled it, but if you updated the item’s page it wouldn’t push that to any page it’s displayed on. We ran into issues like this constantly. Some solutions worked, others didn’t.

After a year or so we migrated to something else. It’s free and it’s great that it exists, but it just has a roughness to it that we didn’t have the resources to deal with.

UPDATE:

I see Bookstack mentioned a lot, so I decided to try installing it. I took the better part of a day and I still can’t get it working. Pity since it looks a lot nicer than Dokuwiki and has access control unlike Mediawiki.

An informative YT channel I found. I’m sure many people here might already know, but I found it helpful and it makes the comm a good resource for newer folks looking to get a handle on what all these tools do and how they will use them in their selfhosting.

Cgroups is not a really a security feature (from what I understand). It is about controlling process priority, hierarchy, and resources limiting (among other things).

With respect, I think you misunderstand what gvisor does and containerization in general. cgroups2 is the isolation mechanism used by most modern Linux containers, including docker and lxc both. It is similar to the jail concept in BSD, and loosely to chroot. It limits child process access to files, devices, memory, and is the basis for how subprocesses are secured against accessing host resources without the permission to do so.

Gvisor adds more layers of control over this system by adding a syscall control plane to prevent a container from accessing functions in the host’s kernel that might not be protected by cgroups2 policy. This lessens the security risk of the host running a cutting-edge or custom kernel with more predictable results, but it comes with caveats.

Gvisor is not a universally “better” option, especially for homelab, where environment workloads vary a lot. Gvisor comes with an IO performance penalty, incompatibility with selinux, and its very strength can prevent containers from accessing newer syscalls on a cutting edge host kernel.

My original comment was that ultimately, there is no blanket answer for “how secure is my virtualization stack”, because such a decision should be made on a case-by-case basis. And any choice made by a homelabber or anyone else should involve some understanding of the differences between each type.

Yes, I understand what GVisor does. Cgroups2 are for isolation of system resources, bit arent even the main sandbox feature used for isolation by Docker. I am pretty sure namespaces significantly more important for these containers’ security.

GVisor helps with one of the main risks in a container setup which is the shared kernel by hosts and guests. I understand it comes with a performance penalty (and I didnt know it was incompatible with SELinux), but that does change my original point that GVisor is a security improvement to default Docker. I understand there is more nuance, even when I wrote my original comment I understood (just like any other security feature) it cant be used in every scenario. I was being intentionally general, and in my second comment I was pretty specific about what it protects against: Kernel vulnerabilities and privilege escalation.

I researched cgroups2 more and I still dont understand why you brought it up in the first place. Cgroups2 and gvisor provide very different security benefits. Cgroups help to keep a system available (lessening the risk DoS attacks) by controlling access to some system resources (io, devices, cpu, memory) and grouping processes of a similar type. It seems rather optimized to solve resource control on a container host. I mentioned gvisor because it is mostly just a drop-in replacement container runtime which doesnt need setup to be used.s

Now for a different container runtime which provides significantly more features (than gvisor) with less downsides (if configured correctly for a specific workload), Sydbox provides syd-oci which id an application kernel runtime which uses a permission config file to create a sandbox, isolating using namespaces, seccomp, landlock, and more. It can sandbox in many different categories (often times leveraging multiple features to provide a multilayer sandbox), you can see the categories at the syd manpage. The biggest downside is that you must really understand what your container application needs otherwise it will prevent it from running. It is a “secure by-default” sandbox which can be softened through config.

I have a 56 TB local Unraid NAS that is parity protected against single drive failure, and while I think a single drive failing and being parity recovered covers data loss 95% of the time, I’m always concerned about two drives failing or a site-/system-wide disaster that takes out the whole NAS.

For other larger local hosters who are smarter and more prepared, what do you do? Do you sync it off site? How do you deal with cost and bandwidth needs if so? What other backup strategies do you use?

(Sorry if this standard scenario has been discussed - searching didn’t turn up anything.)

So being encrypted before transmission and at rest isn’t enough simply because someone at backblaze can send the encrypted files out to you on a HDD……..

lol

Nice ragebait.