python 3.x - Selenium & Heroku: urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

Question

Setup:

• selenium: 3.141.0
• python: 3.6.7
• heroku-stack: heroku-18
• headless-chrome: v71.0.3578.80 buildpack installed
• chromedriver: v2.44.609551 buildpack installed

I'm getting this error when using selenium in heroku:

urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

I googled but didn't have luck. The error happens at the last line of this code.

---

Code

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

UA = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36' 
     '(KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'
DRIVER_PATH = '/app/.chromedriver/bin/chromedriver'

chrome_options = webdriver.ChromeOptions()
chrome_options.binary_location = '/app/.apt/usr/bin/google-chrome'
chrome_options.add_argument(f'--user-agent={UA}')
chrome_options.add_argument(f'--proxy-server=http://my_private_proxy.com:my_port')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--no-sandbox')

chrome = webdriver.Chrome(executable_path=DRIVER_PATH, options=options)

Answer 1

This error message...

urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

...implies that the ChromeDriver in Headless mode was unable to initiate/spawn a new WebBrowser i.e. Chrome Browser session.

Some information regarding the versions of the binaries you are using would have helped us to analyze the error in a better way. However, this issue of urllib3 can be obseved due to several reasons as mentioned below:

• As you are using non Windows OS, the argument --disable-gpu is not needed as in the discussion Headless: make --disable-gpu flag unnecessary skyos...@chromium.org mentioned:

This flag is no longer necessary on Linux or macOS. It will become unnecessary on Windows as soon as SwiftShader fails an assert on Windows in headless mode gets fixed.

• You can find a detailed discussion in Lost UI shared context : while initializing Chrome browser through ChromeDriver in Headless mode

• Additionally, you can add the argument --disable-dev-shm-usage to overcome limited resource problems:

  chrome_options.addArguments("--disable-dev-shm-usage"); // overcome limited resource problems
  

• You can find a detailed discussion on --disable-dev-shm-usage in the discussion org.openqa.selenium.WebDriverException: unknown error: DevToolsActivePort file doesn't exist while trying to initiate Chrome Browser

• As per urllib3.exceptions.ProtocolError: ('Connection aborted.', error(10054, 'An existing connection was forcibly closed by the remote host')) this issue was observed when there is some incompatibility between the version of the binaries you are using.

Solution

• Upgrade ChromeDriver to current ChromeDriver v2.44 level.
• Keep Chrome version between Chrome v69-71 levels. (as per ChromeDriver v2.44 release notes)
• Clean your Project Workspace through your IDE and Rebuild your project with required dependencies only.
• If your base Web Client version is too old, then uninstall it and install a recent GA and released version of Web Client.
• Take a System Reboot.
• Execute your @Test.