2025-11-08 –, Room 313
PhD students, postdocs and independent researchers often struggle when trying to scale their code and data beyond their local machine, to a HPC cluster or the cloud. This is even more difficult if they don’t happen to have access to IT staff and resources to set up the necessary infrastructure, as is the case in many developing countries. We introduce a new open source, extensible remote development architecture, supported in version 6.1 of the Spyder scientific environment and IDE, that allows users to manage packages, browse files and run code remotely on a completely austere host from the comfort of their local machine.
The last 10 years have witnessed the rise of the PyData ecosystem, becoming the lingua franca of scientific computing. Behind that incredible success has been its open source nature, enabling students and researchers in any part of the world to use, share and build upon one anothers’ work. However, the fact that the ecosystem is a public good doesn't necessarily make it a democratic one. While PyData itself is free, the ecosystem still requires a substantial amount of technical knowledge to set it up and get the most out of it.
This is especially the case for remote development. It's not an easy task to move code developed locally to HPC clusters or the cloud in order to access the vastly larger computing, memory and storage resources available. The most basic workflow involves logging in via SSH to the remote machine, recreating a Python environment there, syncing your code and data using terminal commands and finally running it. That's certainly not straightforward for many scientists and engineers, who often have limited experience on the command line.
Furthermore, what happens when your code doesn't work because the local and remote environments are slightly different? Or when you need to fix things to make it work remotely? Once the local and remote versions of the code start to diverge, keeping everything in sync becomes exponentially more complicated given the unfriendly UX. Setting up Jupyter Server, JupyterHub and JupyterLab is one possible solution, but that can be a daunting endeavor without IT staff to rely on for help—which are often small or nonexistent in the developing countries from which many Spyder developers and users hail.
We introduce a new open source, fully-extensible remote development architecture, built on top of Jupyter Server, to democratize this process by drastically simplifying what is required from users to get started with remote development. It is fully supported by the newly-released version 6.1 of the Spyder scientific environment and IDE, and can be easily leveraged and extended by other applications and plugins as well.
As we’ll showcase in a live demo, all users have to do is enter their SSH credentials and specify what packages to install on the remote machine (or duplicate an existing local environment), and Spyder will establish a connection to the remote machine, install a server to handle all facilities required for remote development, and create the necessary Conda/Pixi environment. Once that initial automatic setup is complete, users will then be able to connect to a remote host to manage packages, run code and manipulate files with a single click.
Finally, we’ll explain how the architecture behind these enhancements was carefully designed to make it easily usable and extensible by third-party projects and plugins. Spyder Remote Services is a Jupyter Server extension, so it can be customized by other extensions and also installed in an existing JupyterHub deployment. On the Spyder side, the Remote Client plugin offers a context manager to interact with the server API as needed. This allows it to offer capabilities beyond those of other IDEs, such as VSCode and PyCharm, which thanks to its modular open architecture can be leveraged by other applications and plugins and augmented even further.
Python and Spyder core developer, specializing in docs, infra, and UI. Python Docs Team and PEP Editor. Star✦Fleet Commander. Former NASA-funded lighting UAH researcher.