this post was submitted on 21 Jul 2024
97 points (96.2% liked)

Selfhosted

44306 readers
1 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS
 

Sorry but I can't think of another word for it right now. This is mostly just venting but also if anyone has a better way to do it I wouldn't hate to hear it.

I'm trying to set up a home server for all of our family photos. We're on our way to de-googling, and part of the impetus for the change is that our Google Drive is almost full.We have a few hundred gigs of photos between us. The problem with trying to download your data from Google is that it will only allow you to do so in a reasonable way through Google takeout. First you have to order it. Then you have to wait anywhere from a few hours to a day or two for Google to "prepare" the download. Then you have one week before the takeout "expires." That's one week to the minute from the time of the initial request.

I don't have some kind of fancy California internet, I just have normal home internet and there is just no way to download a 50gig (or 2 gig) file in one go - there are always intrruptions that require restarting the download. But if you try to download the files too many times, Google will give you another error and you have to start over and request a new takeout. Google doesn't let you download the entire archive either, you have to select each file part individually.

I can't tell you how many weeks it's been that I've tried to download all of the files before they expire, or google gives me another error.

top 50 comments
sorted by: hot top controversial new old
[–] [email protected] 31 points 9 months ago (1 children)

There's no financial incentive for them to make is easy to leave Google. Takeout only exists to comply with regulations (e.g. digital markets act), and as usual, they're doing the bare minimum to not get sued.

[–] [email protected] 3 points 9 months ago* (last edited 9 months ago) (1 children)

Or why is Google Takeout as good as it is? It's got no business being as useful as it is in a profit-maximizing corpo. 😂 It can be way worse while still technically compliant. Or expect Takeout to get worse over time as Google looks into undermaximized profit streams.

[–] [email protected] 8 points 9 months ago* (last edited 9 months ago) (1 children)

Probably because the individual engineers working on Takeout care about doing a good job, even though the higher-ups would prefer something half-assed. I work for a major tech company and I've been in that same situation before, e.g. when I was working on GDPR compliance. I read the GDPR and tried hard to comply with the spirit of the law, but it was abundantly clear everyone above me hadn't read it and only cared about doing the bare minimum.

load more comments (1 replies)
[–] [email protected] 16 points 9 months ago (1 children)

Google takeout is the best gdpr compliant platform of all the big tech giants. Amazon for example lets you wait until the very last day they legally can.

Also they do minimal processing like with the metadata (as others commented) as it is probably how they internally store it and that's what they need to deliver. The simple fact that you can select what you want to request and not having to download everything about you makes it good in my eyes.

I actually see good faith compliance with the gdpr in the Plattform

load more comments (1 replies)
[–] [email protected] 15 points 9 months ago (1 children)

Not sure if somebody mentioned, but you can export to one drive. So you can get a 1TB account for a free trial or for a single month and export everything there as simple files, no large zips. Then with the app download to the computer and then cancel one drive.

Pretend to be in California/EU and then ask full removal of all your data on both Microsoft and google

[–] [email protected] 3 points 9 months ago

This route may be the answer. I didn't have success so far in setting up a download manager that offered any real improvements over the browser. I wanted to avoid my photos being on two corporate services, but as you say, in theory everything is delete-able.

[–] [email protected] 15 points 9 months ago* (last edited 9 months ago) (4 children)

Honestly I thought you were going to bitch about them separating your metadata from the photos and you then having to remerge them with a special tool to get them to work with any other program.

[–] [email protected] 12 points 9 months ago

omg they WHAT

[–] [email protected] 9 points 9 months ago

I'm not really looking forward to that step either

[–] [email protected] 7 points 9 months ago

Lmao I am both amused and horrified that I had somehow never come across this datapoint before

[–] [email protected] 6 points 9 months ago (1 children)

immich has a great guide to move a takeout from google into immich

[–] [email protected] 3 points 9 months ago* (last edited 9 months ago) (1 children)
[–] [email protected] 6 points 9 months ago (1 children)
[–] [email protected] 4 points 9 months ago (1 children)

Thank you! The goal is to set up immich. It's my first real foray into self hosting, and it seems close enough to feature parity with Google that the family will go for it. I ran a test with my local photos and it works great, so this is the next step.

load more comments (1 replies)
[–] [email protected] 14 points 9 months ago

Because Google don’t want you to export your photos. They want you to depend on them 100%.

[–] [email protected] 8 points 9 months ago (3 children)

I know it’s not ideal, but if you can afford it, you could rent a VPS in a cloud provider for a week or two, and do the download from Google Takeout on that, and then use sync or similar to copy the files to your own server.

[–] [email protected] 4 points 9 months ago (2 children)

I don't know how to do any of that but I know it will help to know anyway. I'll look into it. Thanks

[–] [email protected] 4 points 9 months ago

Be completely dumb and install a desktop OS like Ubuntu Desktop. Then remote into it, and use the browser just as normal to download the stuff on it. We'll help you with moving the data off it to your local afterwards. Critically the machine has to have as much storage as needed to store all of your download.

[–] [email protected] 3 points 9 months ago

Instead of having to do an Operating system setup with a cloud provider, maybe another cloud backup service would work. Something like Backblaze can receive your Google files. Then you can download from Backblaze at your leisure.

https://help.goodsync.com/hc/en-us/articles/115003419711-Backblaze-B2

Or use the filters by date to limit the amount of takeout data that's created? Then repeat with different filters for the next chunk.

load more comments (2 replies)
[–] [email protected] 6 points 9 months ago

I have fancy California Internet and the downloads are surprisingly slow and kept slowing down and turning off. It was such a pain to get my data out of takeout.

[–] [email protected] 6 points 9 months ago

The word you're looking for is "petty."

[–] [email protected] 6 points 9 months ago

Try this then do them one at the time. You have to start the download in your browser first, but you can click "pause" and leave the browser open as it downloads to your server

[–] [email protected] 5 points 9 months ago

It's called: vendor lock-in.

[–] [email protected] 5 points 9 months ago (4 children)

There was an option to split the download into archives of customizable size IIRC

load more comments (4 replies)
[–] [email protected] 4 points 9 months ago* (last edited 9 months ago)

Google takeout is there so they are technically compliant with rules that say you must be able to download your personal data, but they make it so inconvenient to use that practically it's almost impossible to download it. Google photos isn't a backup service so much as a way for Google to hold your photos hostage until you start paying for higher amounts of storage. And by the time you need that storage, Google takeout download has become impractical.

[–] [email protected] 4 points 9 months ago (1 children)

You could try using rclone's Google Photos backend. It's a command line tool, sort of like rsync but for cloud storage. https://rclone.org/

[–] [email protected] 10 points 9 months ago (2 children)

Looked promising until

When Images are downloaded this strips EXIF location (according to the docs and my tests). This is a limitation of the Google Photos API and is covered by bug #112096115.

The current google API does not allow photos to be downloaded at original resolution. This is very important if you are, for example, relying on "Google Photos" as a backup of your photos. You will not be able to use rclone to redownload original images. You could use 'google takeout' to recover the original photos as a last resort

[–] [email protected] 3 points 9 months ago

Oh dang, sorry about that. I've used rclone with great results (slurping content out of Dropbox, Google Drive, etc.), but I never actually tried the Google Photos backend.

load more comments (1 replies)
[–] [email protected] 4 points 9 months ago* (last edited 9 months ago) (1 children)

I think this is a bit unfair. Most Google Takeout requests are fulfilled in seconds or minutes. Obviously collating 100GB of photos into a zip takes time.

And it's not googles fault you have internet issues: even a fairly modest 20Mbps internet connection can do 50GB in 6h. If you have outages that's on your ISP not Google. As others have said, have it download to a VPS or Dropbox etc then sync it from there. Or call your ISP and tell them to sort your line out, I've had 100℅ uptime on my VDSL copper line for over 2 years.

I was able to use Google Takeout and my relatively modest 50Mbps connection to successfully Takeout 200GB of data in a couple of days.

[–] [email protected] 3 points 9 months ago (3 children)

What download manager did you use? I've tried with whatever's built into Firefox on two different networks and similar results. The downloads freeze every so often and I have to restart them (it picks up where it left off). Sometimes it just won't reconnect, which I'm guessing is a timeout issue with Google, although I really have no idea.

I don't ever have to manage downloads of this size, so sorry if it's an obvious question

load more comments (3 replies)
[–] [email protected] 4 points 9 months ago (6 children)

It doesn't have an option to split it?

When I did my Google takeout to delete all my pics from Google photos there was an option to split in like "one zip every 2gb"

load more comments (6 replies)
[–] [email protected] 3 points 9 months ago

Not really helping you here. But when I started using Google Photos, I still manually downloaded files from my phone to local storage. I did this mainly to ensure I have the original copies of my photos and not some compressed image. Turns out that was a wise move as exporting photos from Google is a pretty damned awful experience.

[–] [email protected] 3 points 9 months ago (1 children)

Im surprised that feature exist tbh. It worked fine for my 20GB splited into 2GB archives if I remember correctly

[–] [email protected] 2 points 9 months ago

I used it for my music collection not that long ago and had no issues. The family's photo library is an order of magnitude larger, so is putting me up against some of the limitations I didn't run into before

[–] [email protected] 3 points 9 months ago (1 children)

A 50GB download takes less than 12h on a 10Mbps internet. And I had a 10Mbps link 10 years ago in a third world country, so maybe check your options with your ISP. 50GB really should not be a problem nowadays.

[–] [email protected] 4 points 9 months ago (7 children)

It's not the speed - it's the interruptions. If I could guarantee an uninterrupted download for 12 hours, then I could do it over the course of 3-4 days. I'm looking into some of the download management tools that people here have suggested.

[–] [email protected] 3 points 9 months ago

that might work; I don't know if you live in a remote area, but I'd also consider a coffee shop, library, university, or hotel lobby with wifi. You might be able to download it within an hour.

load more comments (6 replies)
[–] [email protected] 2 points 9 months ago

It sucked when I closed my accounts years ago. I had to do it manually for the most part.

load more comments
view more: next ›