kamstrup

Targeting vulnerable people based on metadata with any form of commercial intent, is morally and ethically highly questionable! A vulnerable person is by definition extremely susceptible to exploitation. Assuming that companies are gonna act out of philanthropy and goodness of their hearts seems a bit naiive.

Can't divulge too many details, but one example was when we had 2 options for solving a problem: 1. The "easy" way, storing a bunch small blobs to s3 as a job was running on an embedded device, or 2. The slightly tricky, implement streaming of said data on the device (not as easy as it sounds).

We went with option 1, the easy one, because it was deemed faster bang for the buck. I did some basic math showing that the bandwidth required upload the high number of blobs to s3 within our time budget was not possible on our uplink.

After we spend a month failing on 1., it was clear that we hit the predicted problem. Eventuelly we implement option 2.

Being comfortable with basic back-of-the-envelope math can be a huge benefit. (Full disclosure: i am a math major who is now a programmer)

Over my career I have several examples of projects that have saved weeks worth of dev time because someone could predict the result with some basic calculations. I also have several examples where I have shown people some basic math showing that their idea is never gonna work, they don't listen and do it anyway, and I see them 1 month later and the project failed in the way i predicted.

A popular (and wise) saying is that "Weeks of work can save you hours of meetings". I think the same is true for basic math. "Weeks of coding can save you minutes of calculation".

You can definitely be a successful programmer career without great math skills. Math is a tool that can help you be more effective.

Must include CHANGELOG...

The changelog:

• misc fixes
• pls work
• fixe a typo

Interesting observation! The most simple explanation would be that it is memory claimed by the Go runtime during parsing of the incoming bson from Mongo. You can try calling runtime.GC() 3 times after ingest and see if it changes your memory. Go does not free memory to the OS immediately, but this should do it.

2 other options, a bit more speculative:

Go maps have been known to have a bit of overhead in particular for small maps. Even when calling make() with the correct capacity. That doesn't fit well with the memory profile you posted well, as I didn't see any map container memory in there...

More probable might be that map keys are duplicated. So if you have 100 maps with the key "hello" you have 100 copies of the string "hello" in memory. Ideally all 100 maps qould share the same string instance. This often happens when parsing data from an incoming stream. You can either try to manually dedup the stringa, see if the mongo driver has the option, or use the new 'unique' package in Go 1.23