Interoperability between self-hosted services
Plenty of us are using Docker, Podman, Incus, chroot jails, etc to isolate services.
It has become good practice and it makes setting up yet another service, usually, so convenient.
Some services like YunoHost, StartOS, Cloudron and others try to facilitate the process.
What I haven't seen though is a way to facilitate interoperability BETWEEN services we self-host. Sure there are plugins for each service, e.g. npmjs.com/package/peertube-plu… to provide XMPP chat for PeerTube, or anecdotal discussions e.g. github.com/jitsi/jitsi-meet/is… to embed PeerTube on Jitsi Meet.
So... how do YOU do it? How do you make on self-hosted service with another? Do you check after each one you install in the plugin category? Do you write your own plugins or extensions? Do you have a design pattern (e.g. Swagger API discovery with token generation per service, "cheat" via sockets, use a dedicate new service or even host) which you repeat?
I do ask because I bet most of you have a moment like this :
- Hey how about we start this new project together?
- Yes, let's change the World!
- OK let's write manifesto.md
- Where are we going to host it?
- Hmmm we could use my Cryptpad instance...
- OK but I don't get notification on my GMail, could we use GoogleDocs instead?
So... I feel like FLOSS self-hosting is honestly on-par functionality-wise with proprietary solutions. I might be bias but it's rare when I think "Damn... that's cool, shame I can't have it at home". I can nearly always (in fact I have a hard time thinking of an example) self-host functional equivalent solutions myself. The ONE thing that I feel is often missing is integration which relies on interoperability.
How do YOU it?
PS: this isn't about ntfy, PeerTube, HA or any specific service to a specific problem, it's about HOW to facilitate, when one wants to, already great services work together.
Please support Peertube (and/or other) embedding
The functionality to play YouTube videos work great. It would be even better if Jitsi Meet also supported playing videos from a PeerTube instance (or even allow to paste an embed code).highvoltage (GitHub)
Tolookah
in reply to utopiah • • •My process typically goes:
ooh, shiny new service!
If it has a docker compose, I'll read up a little more on it, then set it up.
At that point, I'll cry a bit in having to change my nginx config, because it's new and unsupported, figure out what's actually needed, then realize I didn't set it up with authentik or even check if it was supported.
Usually I'll dig around and someone got something close, so I'll set it up that way, and if it works, great! If not, dig through the documents.
Any further interoperability is luck based.
(I'm currently trying to get element-call working before I open it to my family, the whole matrix stack is leaving me scarred )
utopiah
in reply to Tolookah • • •Yes I can relate to the process.
Unfortunately I can relate to that, hence the question here :D
themachine
in reply to utopiah • • •utopiah
in reply to themachine • • •same
... well that's the part I'm challenging. I was thinking like this but I'm wondering if that could be improved.
PS: I use ntfy and like it, that was just an example.
irmadlad
in reply to utopiah • • •Do you have a specific use case for two containers that you want to talk to each other? There are some Docker containers that will cozy up, but as you point out, there are some that don't. Maybe it would be worth the time to consult with the dev teams of said containers, and see if they have any suggestions or ways to go about it. It would seem to me that would be your best point to deviate from. Couldn't hurt. I would create a defined road map of what you are trying to accomplish before hand, and run it by the dev teams. Just give them a little time to respond. They are real people with real lives too.
As I'm thinking about it, I wonder if your solution might be automation? Something like n8n might allow cross container exchanges. Of course, those data points have to be present for you to tap them tho.
utopiah
in reply to irmadlad • • •Sure, for example once a Jitsi Meet meeting ends (more than 1 person in a room in, everybody gone), save the chat log to CopyParty e.g. WebDAV push to /meetingname_date.txt would be enough to be useful. It's something we tend to do manually on a regular basis.
Yes no rush and I can code so I would be able to test before suggesting anything.
I don't touch AI but I do think conventions, e.g. not "just" an API but SWAGGER, specific filesystem on mountpoints, etc could facilitate this.
irmadlad
in reply to utopiah • • •Automation doesn't always mean AI. The app I mentioned, n8n, has two versions: with and without. The plain n8n app is very capable of doing a ton of stuff.
utopiah
in reply to irmadlad • • •Sorry if I'm a bit slow but what does it actually do? I skimmed through "automations" earlier this morning and I mostly found paid-for GenAI related stuff.
irmadlad
in reply to utopiah • • •You're not slow...sheesh.
The version I'm using does not include AI, basically because I don't have the equipment to run AI 100% local, and n8n AI makes you tap public AI outlets. Both versions are available. The community version of n8n is open-source.
A workflow:
::: spoiler spoiler
:::
There are over 400 different integrations: n8n.io/integrations Of course, the AI stuff you can skip. And when you get to the point of proficiency, you can build your own templates and workflows.
For instance, here is a workflow designed to automatically convert media: noted.lol/self-running-video-e…
Another example is pulling logs: noted.lol/system-log-dashboard…
There are literally endless things you can integrate and build with n8n.
n8n.io/
noted.lol/tag/n8n/
Building a Simple System Log Dashboard and Alert Workflow with n8n
Jeremy (Noted)dfense
in reply to utopiah • • •As an example, what I do with n8n to hook services together:
- We use Zello in firefighting to document the incident with short verbal notes or take pictures. Voice gets transcribed, sent to MQTT for dashboards, taken by Telegraf to influx. Pictures and mp3s get downloaded from zello, put into minio, linked back with the metadata and also sent to influx. Finally all is displayed in chronological order on a Dashboard. Zello, Minio, MQTT, influx all glued together with n8n.
- Another one is the appsmith app for equipment maintenance. We need reports and appsmith is bad at that. n8n queries the Source pgsql, formats it to JSON, sends it to an online service to generate a PDF (with a template), downloads the PDF and puts it into Nextcloud.
There is more, but just as two examples. Quite easy in n8n, because many integrations are preconfigured (e.g. Nextcloud), but also plain REST API if necessary (zello).
All without AI....
Edit: Forgot one, Rocket Chat to keep everybody updated on a bunch of stuff, e.g. new maintenance report available, message on the day of a training, based on caldav. Also all with n8n.
yaroto98
in reply to utopiah • • •like this
TVA likes this.
utopiah
in reply to yaroto98 • • •Well I do have Home Assistant, been running it for years, but HA is solely for ... well home assisting (or IoT). HA as integrations but let's say I want to use HA with ... any of my other services, e.g. CopyParty to maybe store logs and makes them available or PeerTube to have videos from my camera, I can look at HA integrations, or CopyParty... issues maybe, or PeerTube npm registry.
My point being that HA is a good example with integrations but it's just one example. If I do take this example seriously though, is there a mechanism beside manual search in the list of integration that would list integrations with my services directly?
TVA likes this.
David J. Atkinson
in reply to utopiah • • •Home Assistant Community
Home Assistant Communityabeorch
in reply to utopiah • •#YunoHost, #StartOS, #Cloudron, #coopcloud, #uncloud and the people that write the recipies for the applications running on them are the answer.
Modelled solutions that check for related applications as they install are the way that encourage more people to self host by making installation, configuration, backup, update and removal easier.
Selfhosted reshared this.
utopiah
in reply to abeorch • • •abeorch
in reply to utopiah • •Selfhosted reshared this.
abeorch
in reply to abeorch • •Selfhosted reshared this.
abeorch
in reply to utopiah • •One of my thoughts is that Yunohost could be usefully integrated with something like #coopcloud to allow it to add additional server instances to scale the solution - it could either via an application or a component of yunohost itself manage #abra deployment and execution.
Similarly it could reach out to other devices like Routers on your home network and help configure their firewalls, port forwarding etc. - That would make configuring a home server easier - for those that aren't experienced with these things.
Selfhosted reshared this.
artyom
in reply to utopiah • • •😡 ntfy.sh 😡
utopiah
in reply to artyom • • •like this
TVA likes this.
non_burglar
in reply to utopiah • • •APIs. Or the ends are achieved by sharing data between apps in common data storage. But I prefer to be a tourist in my infrastructure, I no longer hand-bomb changes to systems.
My design pattern is essentially to integrate more and more of the container creation into config. Right now I'm using ansible and it's nice. More automation means troubleshooting has fewer variables.
I had issues yesterday with a package upgrade across several containers, and it ended up being two config changes. I cycle the apps and done. That's it.
utopiah
in reply to non_burglar • • •moonpiedumplings
in reply to utopiah • • •I use authentik, which emables single sign on (the same account) between services.
Authentik is a bit complex and irritating at times, so I would recommend voidauth or kanidm as alternatives for most self hosters.
utopiah
in reply to moonpiedumplings • • •Thanks, that's indeed exactly the kind of thing I'm looking for "The authentication glue you need." but even more generalized than that, e.g. just "the glue you need." not solely for authentication.
Edit: to clarify and coming back after leaving few other comments, the 1 thing authentik has is that it is a cross-service need, namely nearly all services do need authentication AND, probably consequence of that, there are conventions and standards already in place, e.g. SAML, OAuth2/OIDC, LDAP, Auth0. So that makes everything much easier.
diecknet
in reply to utopiah • • •github.com/n8n-io/n8n
GitHub - n8n-io/n8n: Fair-code workflow automation platform with native AI capabilities. Combine visual building with custom code, self-host or cloud, 400+ integrations.
GitHubutopiah
in reply to diecknet • • •Thanks I'll dig deeper. I guess I do want something like n8n but ideally :
which makes me wonder what they do provide, e.g. is it mostly indexing existing plugins and then some scaffolding for non coders?