https://www.orchest.io/ logo
Title
a

Alexsander Pereira

08/29/2022, 1:26 PM
Any way to use a Postgres in RDS? We use orchest on EKS. If yes, how do I pass this in yml?
j

juanlu

08/29/2022, 1:44 PM
hi @Alexsander Pereira! if I understand correctly, RDS would only account for a PostgreSQL database - what part of Orchest do you want exactly to run there?
r

Rick Lamers

08/29/2022, 1:54 PM
Tagging @Yannick for visibility. Currently the application assumes Postgres is running as a container managed by the Orchest controller. That manifests itself as the Postgres service being addressed in multiple places using the internal cluster network: e.g. in https://github.com/orchest/orchest/blob/e76376941d2d7465b3e837dd3cebbc8035cd9214/services/orchest-api/app/config.py#L15 Making Postgres "pluggable" is generally not a configuration option we're looking to introduce. Although we might be able to make minor changes to make it easier for you to maintain a fork where Postgres is running external to the k8s cluster that's running Orchest.
👀 2
a

Alexsander Pereira

08/29/2022, 4:56 PM
Isn't there a way to make this optional, taking the postgres out of the manifest and passing environment variables with the accesses?
@Rick Lamers I find it very useful, many companies like us, need to ensure that the database is in a safe place, with secure credentials and automatic backup.
👀 1
@juanlu Exactly
r

Rick Lamers

08/30/2022, 8:37 AM
In the long term we will probably make the application database more modular and configurable. Allowing you to host it in a completely separate location. To simplify upgrades & application hosting management we've embedded it in the k8s manifest for the time being.
a

Alexsander Pereira

09/05/2022, 12:00 PM
@Rick Lamers Would it be too complicated to implement this? I believe it's just putting the database as environment variables, correct?
r

Rick Lamers

09/05/2022, 12:24 PM
The bigger concern would be future constraints. It we introduce support for it (externally configurable databases) and want to change database systems/storage approach later we wouldn't be able to do that without breaking existing people's deployments and/or maintaining multiple storage approaches. As a general rule we consider any exposed interface (like a configurable database) something that we want to plan long term support for as to not break anything for anyone using/deploying Orchest. Similar to the concept of private APIs, we'll never prohibit use, but we also don't want to guarantee stability around its use. If you want you can contribute a PR that would make the database configurable and we'll mark it as a "private API" (possible to use but no guaranteed future stability).
y

Yannick

09/05/2022, 12:53 PM
@Alexsander Pereira I think these should be the only places that you would need to configure (to point to another DB):
# orchest-api config.py
SQLALCHEMY_DATABASE_URI = "<postgresql://postgres@orchest-database/orchest_api>"
_result_backend_server = "postgres@orchest-database/celery_result_backend"
result_backend_sqlalchemy_uri = f"postgresql://{_result_backend_server}"
result_backend = f"db+postgresql+psycopg2://{_result_backend_server}"

# orchest-webserver config.py
SQLALCHEMY_DATABASE_URI = "<postgresql://postgres@orchest-database/orchest_webserver>"

# auth-server config.py
SQLALCHEMY_DATABASE_URI = "<postgresql://postgres@orchest-database/auth_server>"
a

Alexsander Pereira

09/05/2022, 6:38 PM
Thanks @Yannick! Do you know if I can pass environment variables in EKS yml?
The idea would be to modify the code to use environment variables.
y

Yannick

09/05/2022, 6:57 PM
Not sure what exactly you mean, but you should be able to set environment variables for the Orchest services (eg the orchest-api) through the OrchestCluster CR object.