Troubleshooting

This topic lists the usual problems users have and how to fix them.

I cannot log in

Access to the cluster is only enabled for students participating in a course that uses the cluster as well as all TAs of such a course. Access is revoked on the last Monday of the semester holidays after a course.

Access to student-jupyter.inf.ethz.ch is furthermore restricted to students participating in a course that runs Jupyter notebooks.

My job is not starting

Run squeue to check the queue. If in the right-most column it lists a node name (studgpu-node??) then you will need to wait up to five minutes until that node is powered up.

Instead of the node name you may also get a status code:

(QOSMaxGRESPerUser)

You have requested too much RAM, too many GPUs or cpu cores.

(Resources) or (QOSGrpGRESMinutes)

You have not properly set the course name and runtime. See here for how to properly start a job. Try to run the examples, they should always work.

My job got canceled

This only affects courses that have long-running jobs. These jobs get canceled when the cluster if full and users of other courses start more short jobs or jupyter notebooks. Interrupted jobs will be automatically restarted if the cluster has less load.

I am out of GPU time for a course

Each user only gets a fixed amount to finish a course. If that is not enough then please contact a TA to find a solution.

My home directory is full

You have 20GB of space and will need to get by with this for all courses.

One place where space is usually wasted is the pip cache. Run these two commands to get its size and purge the cache:

To avoid filling the cache run pip always with the --no-cache-dir option.

If you are working with conda then there is only an option to periodically clean:

To see where your space is used run this command:

If you have data sets or models that were downloaded via git, for instance from Hugging Face, run

In git repositories, every file exists twice and this will reduce it to one copy that is hard linked to two or more locations.

Ultimately you can also copy away data that you do not actively need right now to your personal device.

My JupyterHub server is not starting

Please tick the check mark for write a slurm logfile and try again. If the server does not start then log in to a login node and look at the logfile that was created in your home directory. This should show you what went wrong.

VSCode cannot monitor all files for changes

Because of performance problems, the number of files that can be watched by a user is currently limited to 16384 but may be set to a lower limit in the future. Please edit the file .vscode-server/data/Machine/settings.json and exclude all files in a project that do not need monitoring. Common known directories that do not need watching are already excluded.

Alternatively move everything that is not code outside of the project directory.

The number of active inotify watches can be displayed with the command /cluster/admin/tools/inotify-info.

I get errors about incompatible Cuda version when I install python modules

Read the error message as it usually tells you which version of Cuda you need. Read on here for how to activate a particular version of Cuda.

Some software that I need is not installed

Write to support@inf.ethz.ch and let us know what you are missing and why you need it. We'll have a look at your request and install the necessary packages on all login nodes and GPU nodes if it is not too complicated.