The goal: get the
aws command from the
awscli package to a custom directory, namely
Initially, everything was simple. But latterly it got complicated as many bugs appeared.
I will introduce the bugs one by one before coming up with a stable solution for this use case.
If you arrive here to know the post title's solution, I highly recommend reading from the end of the post.
There are 3 PC, called PC1, PC2, PC3
$ python3 --version Python 3.6.9 $ pip3 --version pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6) l% $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.3 LTS Release: 18.04 Codename: bionic
ubuntu@ip-10-0-102-219:~$ python3 --version Python 3.6.9 ubuntu@ip-10-0-102-219:~$ pip3 --version lsb_pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6) ubuntu@ip-10-0-102-219:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.1 LTS Release: 18.04 Codename: bionic
piproot@vultr:~# python3 --version Python 3.7.3 root@vultr:~# pip3 --version lsb_repip 18.1 from /usr/lib/python3/dist-packages/pip (python 3.7) root@vultr:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 19.04 Release: 19.04 Codename: disco
PC1 and PC2 introduce bugs while the first solution works perfectly in PC3.
Very simple with the
pip3 install --target=/home/transang/my-python-packages awscli
The command successfully finished in the PC3. Yet the PC1 and PC2. There was the following error
Exception: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 215, in main status = self.run(options, args) File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 360, in run prefix=options.prefix_path, File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 784, in install **kwargs File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 851, in install self.move_wheel_files(self.source_dir, root=root, prefix=prefix) File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 1064, in move_wheel_files isolated=self.isolated, File "/usr/lib/python3/dist-packages/pip/wheel.py", line 247, in move_wheel_files prefix=prefix, File "/usr/lib/python3/dist-packages/pip/locations.py", line 153, in distutils_scheme i.finalize_options() File "/usr/lib/python3.6/distutils/command/install.py", line 274, in finalize_options raise DistutilsOptionError("can't combine user with prefix, " distutils.errors.DistutilsOptionError: can't combine user with prefix, exec_prefix/home, or install_(plat)base
I found the bug report here. I needed to specify
--system option to prevent the default
Second step solution
After adding the
pip3 install --system --target=/home/transang/my-python-packages awscli
The command successfully ended in all PCs.
However, another bug was introduced. In the PC3, there is binary installed in
/home/transang/my-python-packages. Yet the PC1, PC2.
I again discovered the solution from here. With the new solution, I came up with the next solution
The currently stable solution
--install-option solves the problem introduced previously.
pip3 install --system --target=/home/transang/my-python-packages --install-option=--install-scripts=/home/transang/my-python-packages/bin awscli
Now, the binary appears in all PCs in
The story does not end yet
I wonder about the stability of python's package management. Compared to
yarn of nodejs, there are too many errors for even a straightforward operation.
When I ran the above solution in Mac, python 3.8.1, pip3 20.0.2, it complains
no such option: --system.
--system flag makes the command work. However, it throws another warning
DEPRECATION: Location-changing options found in --install-option: ['--install-scripts'] from command line. This configuration may cause unexpected behavior and is unsupported. pip 20.2 will remove support for this functionality. A possible replacement is using pip-level options like --user, --prefix, --root, and --target. You can find discussion regarding this at https://github.com/pypa/pip/issues/7309.
The deprecation warning links to this github issue. It looks like the pip maintainers are going to remove the
--install-option flag. This issue is relatively new and is on the TODO list at the time of writing this blog.
After ignoring the warning and finished the installation, I tried to run the installed package. Another error was thrown
Traceback (most recent call last): File "/Users/transang/python/packages-bin/aws", line 19, in <module> import awscli.clidriver ModuleNotFoundError: No module named 'awscli'
Actually, installing the package by this way in my Mac is just optional. So I temporarily installed the package directly to the root directory of python. I will come back and continue this part later.
Now, the most stable version
- Upgrade pip. The following command also specifies a custom location for the pip package
PYTHONUSERBASE=/home/transang/my-site-packages python3 -m pip install --user --upgrade pip
- Check pip version
PYTHONUSERBASE=/home/transang/my-site-packages python3 -m pip --version
It should show the installed latest version of pip (current is
20.0.2 with the customized package location)
- Install awscli
PYTHONUSERBASE=/home/transang/my-site-packages python3 -m pip install --upgrade --target /home/transang/my-python-packages awscli
- Invoke the installed package
PYTHONPATH=/home/transang/my-python-packages /home/transang/my-python-packages/bin/aws --version