Install python package to a custom location

Sang -

The goal: get the aws command from the awscli package to a custom directory, namely /home/transang/my-python-packages.

Initially, everything was simple. But latterly it got complicated as many bugs appeared.

I will introduce the bugs one by one before coming up with a stable solution for this use case.

If you arrive here to know the post title's solution, I highly recommend reading from the end of the post.

Environments

There are 3 PC, called PC1, PC2, PC3

In PC1:

$ python3 --version               
Python 3.6.9
$ pip3 --version                  
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)
l%                                                                                                                                             $ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.3 LTS
Release:	18.04
Codename:	bionic

In PC2:

ubuntu@ip-10-0-102-219:~$ python3 --version
Python 3.6.9
ubuntu@ip-10-0-102-219:~$ pip3 --version
lsb_pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)
ubuntu@ip-10-0-102-219:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.1 LTS
Release:	18.04
Codename:	bionic

In PC3:

piproot@vultr:~# python3 --version
Python 3.7.3
root@vultr:~# pip3 --version
lsb_repip 18.1 from /usr/lib/python3/dist-packages/pip (python 3.7)
root@vultr:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 19.04
Release:	19.04
Codename:	disco

PC1 and PC2 introduce bugs while the first solution works perfectly in PC3.

Initial solution

Very simple with the --target option

pip3 install --target=/home/transang/my-python-packages awscli

The command successfully finished in the PC3. Yet the PC1 and PC2. There was the following error

Exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 360, in run
    prefix=options.prefix_path,
  File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 1064, in move_wheel_files
    isolated=self.isolated,
  File "/usr/lib/python3/dist-packages/pip/wheel.py", line 247, in move_wheel_files
    prefix=prefix,
  File "/usr/lib/python3/dist-packages/pip/locations.py", line 153, in distutils_scheme
    i.finalize_options()
  File "/usr/lib/python3.6/distutils/command/install.py", line 274, in finalize_options
    raise DistutilsOptionError("can't combine user with prefix, "
distutils.errors.DistutilsOptionError: can't combine user with prefix, exec_prefix/home, or install_(plat)base

I found the bug report here. I needed to specify --system option to prevent the default --user option.

Second step solution

After adding the --system option

pip3 install --system --target=/home/transang/my-python-packages awscli

The command successfully ended in all PCs.

However, another bug was introduced. In the PC3, there is binary installed in /home/transang/my-python-packages. Yet the PC1, PC2.

I again discovered the solution from here. With the new solution, I came up with the next solution

The currently stable solution

Adding --install-option solves the problem introduced previously.

pip3 install --system --target=/home/transang/my-python-packages --install-option=--install-scripts=/home/transang/my-python-packages/bin awscli

Now, the binary appears in all PCs in /home/transang/my-python-packages/bin directory.

The story does not end yet

I wonder about the stability of python's package management. Compared to npm, yarn of nodejs, there are too many errors for even a straightforward operation.

When I ran the above solution in Mac, python 3.8.1, pip3 20.0.2, it complains no such option: --system.

Removing the --system flag makes the command work. However, it throws another warning

DEPRECATION: Location-changing options found in --install-option: ['--install-scripts'] from command line. This configuration may cause unexpected behavior and is unsupported. pip 20.2 will remove support for this functionality. A possible replacement is using pip-level options like --user, --prefix, --root, and --target. You can find discussion regarding this at https://github.com/pypa/pip/issues/7309.

The deprecation warning links to this github issue. It looks like the pip maintainers are going to remove the --install-option flag. This issue is relatively new and is on the TODO list at the time of writing this blog.

After ignoring the warning and finished the installation, I tried to run the installed package. Another error was thrown

Traceback (most recent call last):
  File "/Users/transang/python/packages-bin/aws", line 19, in <module>
    import awscli.clidriver
ModuleNotFoundError: No module named 'awscli'

Actually, installing the package by this way in my Mac is just optional. So I temporarily installed the package directly to the root directory of python. I will come back and continue this part later.

Now, the most stable version

  • Upgrade pip. The following command also specifies a custom location for the pip package
PYTHONUSERBASE=/home/transang/my-site-packages python3 -m pip install --user --upgrade pip
  • Check pip version
PYTHONUSERBASE=/home/transang/my-site-packages python3 -m pip --version

It should show the installed latest version of pip (current is 20.0.2 with the customized package location)

  • Install awscli
PYTHONUSERBASE=/home/transang/my-site-packages python3 -m pip install --upgrade --target /home/transang/my-python-packages awscli
  • Invoke the installed package
PYTHONPATH=/home/transang/my-python-packages /home/transang/my-python-packages/bin/aws --version

Happy coding!