Hey! Is there a feature for the pipelines yet where a single node can just be a sql query as opposed to a full fledged notebook?

hi @Eshwaran Venkat! at the moment the only supported technologies are Python/Julia/R/Bash scripts and Jupyter notebooks. having SQL queries would need some way to configure the connection details I assume. you could do it with Python using something like this:

import psycopg2
con = psycopg2.connect(host="your-host", user="postgres", dbname="postgres", password="postgres")

with con.cursor() as cursor:
    cursor.execute("SELECT * FROM pg_stat_activity WHERE pid <> pg_backend_pid( ) AND datname = current_database()")
    while row := cursor.fetchone():
        print(row)

Understood. I am thinking of probably even using a "template" notebook (that can generically read a query from file and run it) where I can pass the SQL query filename as the environment variable. And then use the output in another notebook. It would be something along the lines of SQL runner.

@Eshwaran Venkat that's a very neat idea, perhaps using step parameters might even be nicer since environment variables are shared across all steps of the pipeline, while step parameters aren't

notice that, to run a

.sql

file in Python, it's enough to read all the contents and pass the string to the

execute

method of the psycopg2 cursor, see https://stackoverflow.com/a/50006361/554319

We've had the idea of "template" steps for a while! This is another good reason to do more with that idea. @Eshwaran Venkat thanks for nudging us again.

Thanks for the feedback guys! I'll give some of the other methods a shot.

If you have any other questions, just let us know 🙂!

Sure thing, thanks! And likewise if there's anything I can do to help with your project then by all means :)