melezhik

I mean if it happens when two different repos would run builds simultaneously. Usually average team may have few repos . If there is no builds at all , I think it’s even less by RAM - as all the dependencies are pretty much in binary complied form - so , say 2 GB would be enough . As backend is pure go ( echo framework ) and is extremely low footprint and as for the CI runner is just a dispatcher background process gets run by very efficient rakupp and just idling . All the git operations are also very lightweight - native smart http / CGI protocol is in use under the hood if you need details

Ok, what requirements you think is lightweight ? We talk about team work anyway , where concurrent builds run …

Even though you’d prefer do everything in cli which is arguable alternative , what about cicd part? Teams need that one

Hey. My too cents here. I am building DSCI - self hosted git with ci embedded . Suitable for small teams with limited VPS hosting, features:

• general programming languages for ci ( not YAML )

• super simple install and maintaining- everything is a single binary ( golang ) , ci jobs run on localhost/podman/docker

• no need extra runners / k8s / devops teams - everything is in dev hands through regular programming code - super simple

• ci SDK is available for bash/python/perl/raku/php/powershell/golang

It’s still in early development stage , but users can already play with it - here is demo stand - http://dsci.sparrowhub.io/

Documentation is available here - http://deadsimpleci.sparrowhub.io/ or GitHub - https://github.com/melezhik/DSCI

See installation page on podman specific setup … Pretty much the same as docker

But now when you have a YAML program as the first level abstraction how do you handle results between those multi language calls ? In DSCI this is achieved via states and normal functions , and everting is just a function on general purpose programming language, in YAML you need all these magic ( awkward ) YAML syntax to mimic all those things ( pass parameters , handle global variables , process user input , handle returned parameters , etc )

If you use tools gluing them into YAML - you get YAML bloated with time . When you say those tools already having Python - excellent I would like to use those Python libs or SDK directly in my Python code instead of juggling those tools as cli or code blocks inside YAML

UPDATE: and yeah , re-read again - I guess the most of automation is done today via “CI” pipelines even when those are not meant to be CI only, like you said … anyways the rest I have said stands true for me … don’t bake your code into YAML )

The issues you had just proves that YAML based CI approach always leads to troubles with time. And yeah, I have been there, code generators for YAML. Hundreds of lines for YAML pipelines, etc ))

It allows the CI engine to determine which jobs to start, what their steps are etc.

Yep, like a said , I don't mind to have such a configuration inside YAML, but this should NOT be pipeline code itself )

... put those in separate scripts and call them from the YAML.

This is exactly what I try to avoid, cause:

1. many people (in my experience ) even don't bother refactoring YAML spaghetti code to separate scripts and we end up unmaintainable codebase

2. and even if one has to do such a refactoring what is the point of using YAML at all ?

All I need just a collection of tasks/jobs written on languages of choice and I don't need YAML "programming" language at all )

PS And btw I don't mind having a minimal amount of YAML as configuration layer and this is what is presented in DSCI, but only minimal ))

it's just because I think in real world we have a lot of tasks where state is required or extremely beneficial, some examples on top of my head:

• creation of virtual machines with dynamic IP addresses
• creation of bug tracking system tickets with unique ticket IDs
• looking up in databases where fetched records have unique IDs

etc

Ok, try to do it on GH actions, share states between tasks/jobs for example:

tasks/task_one/task.py

#!/usr/bin/python3

update_state({
  'out1' : 'out1 value',
  'out2' : 'out2 value'
})

tasks/task_two/task.py

#!/usr/bin/python3

dict = get_state()
print(dict["out1"])
print(dict["out2"])

Or share states between jobs:

jobs/job1/task.py

#!/usr/bin/python3

update_state({
  'out1' : 'out1 value',
  'out2' : 'out2 value'
})

jobs/job2/task.py

#!/usr/bin/python3

dict = config()

print(dict["_dsci_"]["job1"]["out1"])
print(dict["_dsci_"]["job1"]["out2"])

I can't imagine how much boilerplate code (if this ever possible ) one needs to write to achieve that on YAML based pipelines (GH Actions/ etc)

And don't tell me about jobs artifacts ))

UPDATE:

Another good example is to run tasks conditionally, yes using old good if:

#!/usr/bin/python3

if some_condition(foo, bar): 
    run_task(
       'task1', {
          'foo' : 'foo value',
          'bar' : 'bar value'
       }
    )

The same could be quite awkward in YAML based code

Only as high level glue , main logic is just normal programming languages - see here https://github.com/melezhik/DSCI/tree/main/examples