Any way to use a Postgres in RDS? We use orchest on EKS. If yes, how do I pass this in yml?

hi @Alexsander Pereira! if I understand correctly, RDS would only account for a PostgreSQL database - what part of Orchest do you want exactly to run there?

Tagging @Yannick for visibility. Currently the application assumes Postgres is running as a container managed by the Orchest controller. That manifests itself as the Postgres service being addressed in multiple places using the internal cluster network: e.g. in https://github.com/orchest/orchest/blob/e76376941d2d7465b3e837dd3cebbc8035cd9214/services/orchest-api/app/config.py#L15 Making Postgres "pluggable" is generally not a configuration option we're looking to introduce. Although we might be able to make minor changes to make it easier for you to maintain a fork where Postgres is running external to the k8s cluster that's running Orchest.

Isn't there a way to make this optional, taking the postgres out of the manifest and passing environment variables with the accesses?

In the long term we will probably make the application database more modular and configurable. Allowing you to host it in a completely separate location. To simplify upgrades & application hosting management we've embedded it in the k8s manifest for the time being.

@Rick Lamers Would it be too complicated to implement this? I believe it's just putting the database as environment variables, correct?

The bigger concern would be future constraints. It we introduce support for it (externally configurable databases) and want to change database systems/storage approach later we wouldn't be able to do that without breaking existing people's deployments and/or maintaining multiple storage approaches. As a general rule we consider any exposed interface (like a configurable database) something that we want to plan long term support for as to not break anything for anyone using/deploying Orchest. Similar to the concept of private APIs, we'll never prohibit use, but we also don't want to guarantee stability around its use. If you want you can contribute a PR that would make the database configurable and we'll mark it as a "private API" (possible to use but no guaranteed future stability).

@Alexsander Pereira I think these should be the only places that you would need to configure (to point to another DB):

# orchest-api config.py
SQLALCHEMY_DATABASE_URI = "<postgresql://postgres@orchest-database/orchest_api>"
_result_backend_server = "postgres@orchest-database/celery_result_backend"
result_backend_sqlalchemy_uri = f"postgresql://{_result_backend_server}"
result_backend = f"db+postgresql+psycopg2://{_result_backend_server}"

# orchest-webserver config.py
SQLALCHEMY_DATABASE_URI = "<postgresql://postgres@orchest-database/orchest_webserver>"

# auth-server config.py
SQLALCHEMY_DATABASE_URI = "<postgresql://postgres@orchest-database/auth_server>"

Thanks @Yannick! Do you know if I can pass environment variables in EKS yml?

Not sure what exactly you mean, but you should be able to set environment variables for the Orchest services (eg the orchest-api) through the OrchestCluster CR object.