Hidden directories and package management

If you use any of the following packages, you may have encountered something strange: Your home directory runs out of disk space, even though your files are small.

• conda
• pip
• uv
• Visual Studio (vscode)
• keras

The reason is likely hidden directories. By default, in UNIX files and directories whose names begin with a period (.) are not listed when using the ls command. To see these files, use ls -a. For example:

> ls
CreateSubdirectories.C     Untitled.ipynb

> ls -a
.              .condarc                .ipython    .profile            .viminfo
..             .condor                 .jupyter    .root_hist          .vimrc
.astropy       .config                 .local      .python_history     .spamassassin      .Xauthority
.bash_history  CreateSubdirectories.C  .myprofile  .ssh                .zprofile
.bashrc        .npm                    Untitled.ipynb
.cache         .ipynb_checkpoints      .pyzor      .vim                .vscode-server

In UNIX, files and directories created for bookkeeping purposes often are hidden, so you can focus on what you've created for your own work. As you can see from the example above, many programs create additional files that you don't usually see. For example, you can use hidden files like .myprofile to configure your shell environment.

For the most part, these hidden files are small. Among the exceptions are the directories used by the packages listed at the top of this page.

Here are a couple of examples:

The conda packages are typically stored in ~/.conda. I use the du command (for "disk usage") to see how much disk space is used by the directory:

> cd
> du -shx .conda
4.0G	.conda

The pip user packages are normally stored in ~/.cache, and the python packages in ~/.local:

> cd
> du -shx .cache
4.8G	.cache
> du -shx .local
1.3G	.local

As you can see from the above examples, a substantial fraction of your available disk storage can be used by these utilities, in a place where you may not easily see them.

The rest of this page discusses how to store these directories outside of their default locations, presumably on your group's file servers which have much more available free space. You only have to go through the following steps once; any changes you've made to your configuration files will be preserved between login sessions.

If you don't understand what an example path like /nevis/olga/data means, you may want to review the automount page.

Although I suggest storing your package-management directories on a file server, don't forget the warning from the disk guide: files in /data partitions are not backed up. You may want to periodically save the names of the packages you've downloaded so that you can re-create your environments should the worst occur, e.g., with conda list or pip freeze.

conda

conda is a package manager that runs on many platforms. It creates a custom environment for a set of packages specified by the user. Since these environments contain the packages the user requests plus any packages needed to make those packages work, conda environments can consume a lot of disk space.

Maybe it's already there

Before creating a custom environment, consider looking at the packages already installed on the Nevis Linux cluster. It might be that all the packages you need are already there. There's a list in the conda topic.

Conda file management

This is covered in more detail in the conda wiki page. Here's the short version.

# Define a handy variable for these operations. The variable
# $USER is your account name on the UNIX system. 
# Note that /nevis/olga/data/$USER/ does not exist at
# Nevis; this is an example that you'll have to edit to suit
# your group's file servers. 
prefix=/nevis/olga/data/$USER/conda

# Create the new package environment directory.
mkdir -p ${prefix}/envs

# If you have an existing environment, clone it to the new location;
# assume your original environment is named 'myenv'.
conda create --name myenv --clone ${prefix}/envs/myenv

# Move the package directory to the new location
(cd ~/.conda; tar -cf - ./pkgs) | (cd ${prefix}; tar -xf -)

# Delete the old environment
conda remove --name myenv --all

# Delete the old package directory
rm -rf ~/.conda/pkgs

# Add the new definitions to ~/.condarc
conda config --add envs_dirs ${prefix}/envs
conda config --add pkgs_dirs ${prefix}/pkgs

pip and python locations

pip is a tool for managing Python packages.

Note: This section is meant for those who use pip outside of a conda environment. If you're using pip within a conda environment (which I don't recommend if you can avoid it; see the conda topic), then the version of pip within that environment will automatically store files within that environment's directory.

There are two aspects to pip's package management: The location of its cache, and the location in which it installs packages. Both of these directories can grow quite large.

If you're using multiple python versions (this is extremely likely, even if you're not aware of it), you want the path of the package directory to include the python version somehow. In the example below, I'm using fancy UNIX tricks to make that happen.

This command looks cryptic. What it does it detect the version of Python you're running, and set the variable $pvers to 'pythonM.N', where M.N is the major and minor version of Python. For example, if you're using Python 3.9.6, $pvers will be set to 'python3.9`.

pvers=$(echo $(python --version) | sed "s/Python \([0-9]\+\.[0-9]\+\).*/python\1/")

For these examples, assume you want to store pip's cache in /nevis/olga/data/$USER/cache, and have pip store the python libraries in /nevis/olga/data/$USER/${pvers}/site-packages. Note that these names are examples; in particular, there's no machine named olga at Nevis.

To change the default location of pip's directories, you have to modify pip's configuration file. Use variables (to refer to the directories later) and pip config to set new values:

# Define variables for our pip/python directories. The machine name "olga"
# does not exist at Nevis; you'll have to learn which file server your group uses.
cacheDir=/nevis/olga/data/$USER/cache
pkgDir=/nevis/olga/data/$USER/${pvers}/site-packages

# Modify pip's configuration
pip config set global.cache-dir ${cacheDir}
pip config set global.target ${pkgDir}

Create these directories. You can copy your existing files if you wish:

mkdir -p ${cacheDir}
mkdir -p ${pkgDir}

# Copy the old cache files to the new locations. This is
# optional. 
(cd ~/.cache/pip; tar -cf -) | (cd ${cacheDir}; tar -xvf -)
# For the libraries, you may have to reinstall them. The following
# line may work, but I make no promises:
( cd $(python -m site --user-site); tar -cf - ) | (cd ${pkgDir}; tar -xvf -)

# Delete the old pip cache directory
rm -rf ~/.cache/pip
# Erase the contents of your python packages directory (see below).
( cd $(python -m site --user-site); rm -rf * )

You've instructed pip to install the python packages in a new location. Now you have tell python to search this location. You can do this by modifying the $PYTHONPATH shell variable, but you have to do this for every login session. What I did was to create a .pth file. To find out where to put the file:

python -m site --user-site

To create the file, I got fancy with shell scripting (the command in a $() expression is executed and its output is put into the command line):

pthFile=$(python -m site --user-site)/my-packages.pth

Note that you can use any name of the form name.pth for the path-name file. In my-packages.pth, you would put in the location of your relocated python library; here that's what's in the variable ${pkgDir}. I'm using another UNIX shell trick to add the name of the new package directory to the end of my .pth file:

echo ${pkgDir} | cat >> ${pthFile}

Again, I'm aware that all of the above is cryptic to anyone unfamiliar with UNIX shells. Assuming that you cut-and-paste the above commands, and edit the name olga to something suitable for your servers, you will hopefully end up with a consistent configuration for current future pip install commands. You can check with:

pip config list

uv

uv is yet another tool for managing python environments and packages.

uv requires two work directories: a cache directory, by default in ~/.cache/uv; a data directory, by default in ~/.local/share/uv.

uv follows the XDG conventions for program work files. You can relocate those directories with statements like the following:

# Pick a location for these files. 

# Note that /nevis/olga/data/$USER/ does not exist at
# Nevis; this is an example that you'll have to edit to suit
# your group's file servers. 

# These command have to be used before your first uv command in a login session.
# You might want to put them in your shell's initialization script.
prefix=/nevis/olga/data/$USER/xdg
export XDG_CACHE_HOME=${prefix}/cache
export XDG_DATA_HOME=${prefix}/data

# The following command only has to be executed once, to create
# these directories.
mkdir -p ${XDG_CACHE_HOME} ${XDG_DATA_HOME}

For tips on shell initialization scripts, see the shells page.

Note that other packages than uv make use of the XDG work areas, such as some X11 desktop programs.

vscode

By default, Visual Studio will create either a ~/.vscode-server or ~/.vscode-remote directory. Depending on your project, the size of this folder can grow quite large.

This page contains tips for picking a different location. As a quick guide:

# Pick a location for vscode's work directory and create it. This only has to be done once.

# The machine name "olga" does not exist at Nevis. You'll have to pick a location that suits
# your group's file servers.
prefix=/nevis/olga/data/$USER/vscode
mkdir -p ${prefix}

In your local VSCode program, you'll have to set the serverInstallPath variable. If you know how to edit your computer's settings.json file, you could add lines like the following at the end; these lines assume your that your server's name is login.nevis.columbia.edu and the location of your vscode directory is /nevis/olga/data/ACCOUNT/vscode, where ACCOUNT is the same as the value of $USER on your server; that is, it's your Nevis account name:

"remote.SSH.serverInstallPath": {
    "login.nevis.columbia.edu": "/nevis/olga/data/ACCOUNT/vscode"
}

If you prefer to use VSCode user interface to make this change:

1. File > Preferences > Settings
2. Filter: @ext:ms-vscode-remote.remote-ssh install
3. Under "Server Install Path" > Add Item
   1. Item = login.nevis.columbia.edu
   2. Value = /nevis/olga/data/ACCOUNT/vscode

keras

Keras is a tool for machine learning.

The ~/.keras, which is used to download pre-trained models, can grow unexpectedly large. Here's an example:

> cd
> du -shx .keras
2.6G    .keras 

At the time of this writing (Jul-2023), there is only one way to re-direct keras downloads to a different directory. That is to set the shell variable $KERAS_HOME to a different value before you start a program that invokes keras. For example:

export KERAS_HOME="/nevis/olga/data/$USER/keras"
# Then start jupyter or run python or whatever.

You can put the above statement (suitably edited, since the machine olga does not exist at Nevis) into your ~/.myprofile file so you can be assured that the variable will be set at every login session.

Is that all?

No. This page focuses on unexpected large hidden directories that have occurred on the Nevis particle-physic Linux cluster. Depending on your work and programs you use, there may be other "hidden" directories that store files that you did not expect, such as in the Library folder on MacOS. If you use python virtualenv in a project, you may want to look at the directories venv or .venv in the project's directory, or ~/.virtualenvs in your home directory.

Treat this page as a guide to finding other such directories: ls -a to find hidden files/directories, du to see how much space they use.