https://www.orchest.io/ logo
e

ekeras

11/21/2022, 12:31 PM
Hi! Has anyone have tried running Selenium on Orchest pipeline before? Now, I have tried it with both Firefox and Chrome drivers, but both of them doesn't work. For example, with Chrome drivers, I am getting this error:
Copy code
unknown error: DevToolsActivePort file doesn't exist
Simple Google search would lead to this thread. Top answer says that Selenium web drivers should be initialized as regular user instead of root user. So, I thought maybe I could retake ownership with this command line:
Copy code
sudo chown -R $(whoami):$(whoami) ~/
Unfortunately, it doesn't help. Maybe you have any other ideas how I could initialize Selenium as a regular user? In comments section I am attaching both environment and python code I am using.
Copy code
sudo apt update -y

sudo apt-get install -y libglib2.0 libnss3 libgconf-2-4 libfontconfig1
sudo apt-get install -y chromium-browser

pip install -U selenium
pip install webdriver-manager
Copy code
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

options = Options()
options.add_argument("--headless")
options.add_argument("--disable-extensions") # disabling extensions
options.add_argument("--disable-dev-shm-usage") # overcome limited resource problems
options.add_argument("--no-sandbox") # Bypass OS security model
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
driver.get("<https://www.google.com>")
r

Rick Lamers

11/21/2022, 12:34 PM
Hi @ekeras, I helped a friend set up selenium on Orchest a while ago. I'll share with you a minimal example of how to get it to work 👍 Give me some time to find it & clean it up.
orchest 1
https://github.com/ricklamers/orchest-selenium-example The essential bit is regarding how to install a chromium based browser inside a container context, and the most foolproof method there is using the Chrome debian package. Environment setup script: https://github.com/ricklamers/orchest-selenium-example/blob/master/.orchest/environments/97585f3f-69a1-4c18-b1eb-cdd3c246aa91/setup_script.sh
Copy code
#!/bin/bash

# Install any dependencies you have in this shell script,
# see <https://docs.orchest.io/en/latest/fundamentals/environments.html#install-packages>

# E.g. mamba install -y tensorflow
sudo apt update
cd /tmp
wget <https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb>
sudo apt install ./google-chrome-stable_current_amd64.deb -y

pip install selenium webdriver-manager
You should be able to import that project and it should just work™
e

ekeras

11/21/2022, 3:37 PM
Thanks! That works nicely
👍 1
7 Views