irmadlad

I've read 'The Home Lab Handbook: Building and Managing Your Own IT Lab from Scratch' which I would recommend to anyone just starting out in selfhosting and homelabing. Relative to that, I found a 'course' online (https://linuxupskillchallenge.org/#table-of-contents) that would also be useful for new arrivals.

Anyone reading any good HomeLab & Selfhosting books lately?

For the past 3 or so months I've been noticing entries in Suricata that concern me. Maybe they are benign, but figured I'd throw this out there and see if anyone has/is experiencing this.

There is a pattern to these entries. All of them are listed as 'PROTOCOL-ICMP Destination Unreachable Network Unreachable'. But it's like there is a cron that fires this off once every hour and 5 +/- minutes.

12/13/2025 16:55:02
12/13/2025 15:50:01
12/13/2025 14:45:01
12/13/2025 13:40:01
12/13/2025 12:35:01
12/13/2025 11:30:01
12/13/2025 10:25:02
12/13/2025 09:20:01
12/13/2025 08:15:01
12/13/2025 07:10:01

These ip ranges are usually from China, Romania, and Singapore. The biggest 'offender' being China:

203.119.27.1 was found in our database!
This IP was reported 11 times. Confidence of Abuse is 1%:
ISP 	China Internet Network Information Center
Usage Type 	Data Center/Web Hosting/Transit
ASN 	AS24406
Hostname(s) 	c.dns.cn
Domain Name 	cnnic.cn
Country 	🇨🇳 China
City 	Shanghai, Shanghai 

Thing is, these ip's are usually what I consider 'clean'. Not a lot of abuse reports. On the surface, I know what 'PROTOCOL-ICMP Destination Unreachable Network Unreachable' means. Pretty self explanatory. What I'm trying to figure out is the why part.

I have gone through my logs, monitored for any calls to these ip's from inside the network, and I come up empty. Nothing within my network, whether server or other devices, is requesting data from these ip's. I have no cron set to do such on a hour and 5 minute interval.

So I'm left wondering, is this normal network chatter? Perhaps scraping attempts? Or perhaps breach attempts. So, I sit at the feet of the network experts to be schooled and see if I have something misconfiguration, or if it's nothing to be worried about, or what the devil is going on.

ETA: Suricata is running in conjunction with pFsense as part of a standalone firewall. ETA2: Also running the evil Cloudflare Tunnel/Zero Trust.

Looking for a self hosted, web search trends monitor. I have looked at Plausible Analytics, OpenSearch, Matomo, and some other website analytics platforms, but I'm not necessarily wanting to monitor a specific website(s). Rather, I want to monitor what people are searching for on the internet.

Is such a thing possible?

LOL Not really, but boy it has been a day. Started at 7:00 am and I finally resolved (?) the issue. In fact I've got through every last bit of my network, and at this point in the evening, I actually don't have a solid reason why the issue was present. Something in my VPN settings glitched, or something got triggered on pFsense and got hung up....something, something with Tailscale. It wasn't CLoudflare this time. LOL

You ever do so much to a problem that when you 'fix' it, you have no real idea what the fix truly was? You ever have a problem and find all the shit you cobbled together in the name of 'just get it running and back online'? I did, and decided that I would fix that shit too. It took all flippin' day.

You guys that do this for a living....I salute you! jebus crispies!

ETA: 8 bells and all's well today.

I think this is how you cross post something. If not, be patient with the old head. I was told you guys might enjoy this glimpse into a simpler time.

Not sure if this is allowed, but I'm sure the mods will let me know.

Memories. LOL Things were so simple way back then it seems.

Did anyone host a BBS? I found this article kind of funny about 'fixing' WIndows by C:\FORMAT C:. Some things don't change. LOL

If you just updated Docker and are having issues with Portainer not connecting to the environment, check this:

Issue: Failed loading environment The environment named <your specific environment> is unreachable

Fix (credit xman601):

Stop Docker

sudo systemctl stop docker

Make sure you have the Docker official repository added for your Ubuntu version. In my case Ubuntu Jammy

sudo apt update
sudo apt install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg


echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  jammy stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update

Install the specific Docker version

sudo apt install docker-ce=5:28.5.2-1~ubuntu.24.04~noble \
                 docker-ce-cli=5:28.5.2-1~ubuntu.24.04~noble \
                 containerd.io

or if you are on 22:

sudo apt install docker-ce=5:28.5.2-1~ubuntu.22.04~jammy \
                 docker-ce-cli=5:28.5.2-1~ubuntu.22.04~jammy \
                 containerd.io

Prevent Ubuntu from automatically upgrading Docker:

sudo apt-mark hold docker-ce docker-ce-cli

Start Docker again

sudo systemctl start docker
sudo systemctl enable docker

Verify the version

docker --version

You should see:

Docker version 28.5.2, build ...

Once this issue has been fixed you can run the following to update docker back

sudo apt-mark unhold docker-ce docker-ce-cli

Referance: https://github.com/portainer/portainer/issues/12925

It should go without saying, you should do your due diligence reviewing and confirming any code snippets found on the internet before you drop them in on a production environment.

So, a buddy of mine dropped off a box of 18 Wyse 3040 & 5010 thin clients. I believe they all run W10 embedded, but doing some research, I think I can also run a lightweight Linux OS like maybe Tiny Core. The 5010 can run SuSE Linux Desktop 11, ThinOS, or ThinOS PCOIP acording to Dell.

So, the burning question I have today is 'If you were gifted a box of 18 Wyse 3040 & 5010 thin clients, what would you do with them'? I want something I can incorporate into my already established homelab.

Inundate me with ideas!

I'm almost embarrassed to ask this question, but it's been bugging me for years. I've read the documentation, searched online. Perhaps my search-fu is lacking.

In ntopng there is a panel called Traffic Classification. One of the classifications is 'fun'. Exactly how is this classification derived, and what is classified as 'fun'?

I know this isn't explicitly selfhosting related, but the Venn diagrams do overlap. I've been searching, but for the most part, what I find are 2 year old, defunct subs.

So, right out the gate, I realize Watchtower has been abandoned. I get that. Still, it's a great app. 95% of the time, it works every time.

The issue I have is as follows:

INFO[93549] Found new searxng/searxng:latest image (5613a99caf5d) 
INFO[93565] Found new prom/prometheus:latest image (b5239da13ab0) 
INFO[93584] Found new ghcr.io/karakeep-app/karakeep:latest image (c5fcb1a653c6) 
INFO[93592] Stopping /CADVISOR-PROMETHEUS (342680d2a684) with SIGTERM 
INFO[93593] Creating /CADVISOR-PROMETHEUS                
INFO[93594] Stopping /SEARXNG (0842827f17a9) with SIGTERM 
INFO[93597] Creating /SEARXNG                            
ERRO[93597] Error response from daemon: the container-wide MAC address must match the endpoint-specific MAC address for the main network, or be left empty 
INFO[93597] Removing image 9e24e9a9806c                  
INFO[93597] Session done                                  Failed=1 Scanned=38 Updated=2 notify=no

It doesn't do this everytime, and it's not one particular container. It seems random. For the example of SearxNG, I have not specified a MAC address.

docker run -d \
  --name searxng \
  -p 8989:8080 \
  -v /srv/docker/searxng/data:/etc/searxng \
  -e SEARXNG_BASE_URL="https://" \
  -e SEARXNG_INSTANCE_NAME="SearxNG" \
  -e SEARXNG_CONTACT_INFO="mail.com" \
  -e SEARXNG_LANGUAGE="en-US" \
  -e SEARXNG_AUTOCOMPLETE="duckduckgo" \
  -e SEARXNG_THEME="simple" \
  -e SEARXNG_OUTGOING_METHOD="default" \
  -e SEARXNG_ENABLE_METRICS="true" \
  -e SEARXNG_ENABLE_CAPTCHA="false" \
  -e SEARXNG_ENABLE_INFINITE_SCROLL="true" \
  -e SEARXNG_ENABLE_PIWIK_ANALYTICS="false" \
  -e SEARXNG_ENABLE_ADVANCED_SEARCH="true" \
  -e SEARXNG_ENABLE_PRIVATE_RESULTS="true" \
  -e SEARXNG_ENABLE_TORIFICATION="false" \
  -e SEARXNG_ENABLE_HTTPS_EVERYWHERE="true" \
  -e SEARXNG_ENABLE_PROXY="false" \
  -e SEARXNG_ENABLE_PLUGINS="true" \
  searxng/searxng

In fact, I've never specified a specific MAC for any of my containers, and just let Docker do what it do. I'm just wondering if anyone has run up on this issue, and what did you do to remediate it, besides use another updater app? I like Watchtower, but only 95% of the time. LOL

It's not a huge issue spinning the container back up. Takes less than 30 sec. However, that's not how it's supposed to work, so I seek input.

ETA: Just adding this for anyone who is testing the Watchtower fork or are thinking about it. Day 2, and everything is running just fine. I have not had any of the issues that I had with OG Watchtower:

time="2025-07-01T17:28:21Z" level=info msg="Found new netdata/netdata:edge image (7cbd870f51b0)"
time="2025-07-01T17:28:42Z" level=info msg="Found new influxdb:2 image (b4355848b856)"
time="2025-07-01T17:28:54Z" level=info msg="Found new ghcr.io/karakeep-app/karakeep:latest image (c5fcb1a653c6)"
time="2025-07-01T17:29:03Z" level=info msg="Found new redis:latest image (ed3a2af6d0d4)"
time="2025-07-01T17:29:04Z" level=info msg="Stopping /CADVISOR-REDIS (84a28cf8904d) with SIGTERM"
time="2025-07-01T17:29:06Z" level=info msg="Stopping /ROOT-INFLUXDB2-1 (a14e51a1b7db) with SIGTERM"
time="2025-07-01T17:29:07Z" level=info msg="Stopping /NETDATA (bde1241c0073) with SIGTERM"
time="2025-07-01T17:29:09Z" level=info msg="Creating /NETDATA"
time="2025-07-01T17:29:09Z" level=info msg="Creating /ROOT-INFLUXDB2-1"
time="2025-07-01T17:29:10Z" level=info msg="Creating /CADVISOR-REDIS"
time="2025-07-01T17:29:10Z" level=info msg="Session done" Failed=0 Scanned=38 Updated=4 notify=no
time="2025-07-02T17:28:07Z" level=info msg="Found new openlistteam/openlist:latest image (0926eda3de9a)"
time="2025-07-02T17:28:23Z" level=info msg="Found new portainer/portainer-ce:latest image (71de3839351a)"
time="2025-07-02T17:28:32Z" level=info msg="Found new ghcr.io/karakeep-app/karakeep:latest image (c5fcb1a653c6)"
time="2025-07-02T17:28:57Z" level=info msg="Found new bigbeartechworld/big-bear-btop:latest image (737e9af25da3)"
time="2025-07-02T17:29:01Z" level=info msg="Stopping /BTOP-PLUS-PLUS (751299b8b022) with SIGTERM"
time="2025-07-02T17:29:04Z" level=info msg="Stopping /PORTAINER (aa754cb14780) with SIGTERM"
time="2025-07-02T17:29:05Z" level=info msg="Stopping /OPENLIST (2ca56a872944) with SIGTERM"
time="2025-07-02T17:29:06Z" level=info msg="Creating /OPENLIST"
time="2025-07-02T17:29:06Z" level=info msg="Creating /PORTAINER"
time="2025-07-02T17:29:07Z" level=info msg="Creating /BTOP-PLUS-PLUS"
time="2025-07-02T17:29:08Z" level=info msg="Session done" Failed=0 Scanned=38 Updated=4 notify=no

So, recently I spun up cAdvisor to provide some metrics for the Grafana dashboard. I created both the docker-compose.yml and prometheus.yml thusly:

prometheus.yml:

scrape_configs:
- job_name: cadvisor
  scrape_interval: 5s
  static_configs:
  - targets:
    - cadvisor:8080

docker-compose.yml

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
    - 9090:9090
    command:
    - --config.file=/etc/prometheus/prometheus.yml
    volumes:
    - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    depends_on:
    - cadvisor
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    ports:
    - 8080:8080
    volumes:
    - /:/rootfs:ro
    - /var/run:/var/run:rw
    - /sys:/sys:ro
    - /var/lib/docker/:/var/lib/docker:ro
    depends_on:
    - redis
  redis:
    image: redis:latest
    container_name: redis
    ports:

- 6379:6379

Placed them both in /tmp/cadvisor/ and ran docker compose up. All well and good, got some metrics to feed Grafana and all would seem jippity jippity.

Next day I notice Prometheus is off line. Hmm, check everything out. Logs complaining of a missing prometheus.yml. On a hunch I recreated the above prometheus.yml and placed it back in /tmp/cadvisor/, restart Prometheus, and it fires right up no runs, no drips, no errors. Before I uploaded the new prometheus.yml, I notice that there is a directory now named prometheus.yml in /tmp/cadvisor/, which is empty. Deleted it.

Next day, same scenario. Missing prometheus.yml, directory called prometheus.yml in /tmp/cadvisor/. I thought well, if it's getting deleted, change the permissions, and continued my daily affairs.

Today, same exact scenario. So, wtf, over? Run some commands:

stat /tmp/cadvisor/prometheus.yml
sudo lsof /tmp/cadvisor/prometheus.yml
grep "delete" /var/log/syslog

I can see that the file IS being deleted, but I cannot seem to trace down what is deleting it. It's like there is a cron job that fires off every day at a certain time and deletes prometheus.yml, and in it's place, creates a directory called prometheus.yml effectively taking Prometheus offline. I have no such cron job tho.

Any ideas? Suggestions? Ancient wizardry? Any mystical incantations or tomes to consult?