Package Managers

Course Code: U10814 

Course Name: Networking and Operating Systems

Credits: 20

Module Leaders: Dr Alexios Louridas & Mr Danny Werb

Material created by Seb Blair, modified by Danny Werb

center

Package Management Overview

  • Unix systems are superior to Windows: Package Management.
  • Can install anything with ease from your terminal.
  • Update to the latest version with one command.
    • No more downloading the latest installer nonsense!
  • Various tools can be installed by installing a package.
    • A package contains the files and other instructions to set up a piece of software.
    • Many packages depend on each other.
    • High-level package managers download packages, figure out the dependencies for you, and deal with groups of packages.
    • Low-level managers unpack individual packages, run scripts,
      and get the software installed correctly.

Many different philosophies

  • Monolithic binary packages: one big "app" with everything
    bundled together
    • Docker containers, most Windows programs
  • Small binary packages: separate common code into independently-installed "libraries"
    • MSI files, Ubuntu, most of Linux
  • Source-based packages: no installers at all! Compile all your programs
    • Language-based package managers, (e.g., brew, portage)
  • Benefits to all approaches
    • Monolithic binary: fast-ish install, very independent programs
    • Small binary: very fast install, less wasted space
    • Source-based: fastest code, smallest install, easy to use open-source

Package Managers in the Wild

  • GNU/Linux:
    • Low-level: two general families of binary packages exist: deb, and rpm.
    • High-level package managers you are likely to encounter:
      • Debian/Ubuntu: apt-get, apt, aptitude.
      • SUSE/OpenSUSE: zypper.
      • Fedora: dnf (Fedora 22+) / yum.
      • RHEL/CentOS: yum (until they adopt dnf).
      • Arch: pacman
      • Gentoo: portage, emerge
  • macOS:
    • Others exist, but the only one you should ever use is brew.
    • Don’t use others (e.g. port), they are outdated/EOSL.

Using Package Managers

  • Though the syntax for each package manager is different, the concepts are all the same:
    • This lecture will focus on apt, dnf, emerge, and brew.
    • The dnf commands are almost entirely interchangeable with yum, by design.
    • Note that brew is a "special snowflake", more on this later.
  • What does your package manager give you? The ability to
    • install new packages you do not have.
    • remove packages you have installed.
    • update installed packages.
    • update the lists to search for files / updates from.
    • view dependencies of a given package.
    • do a whole lot more!!!

A Note on update

  • These "subcommands" are by category, not name: update is not always called update
  • The update command has different meanings in different package managers.
  • Most do not default to system (read Linux kernel) updates.
    • Fedora does; most others do not.
  • It depends on your operating system, and package manager.
    • Know your operating system, and look up what the default behaviour is.
  • If your program needs a specific version of the Linux kernel, you need to be very careful!
    • Very, very few programs care about your kernel version.

A Note on Names and their Meanings

  • Package names sometimes specify architecture:
    • i386 or i686 or x86: these are the 32-bit packages.
    • x86_64 or amd64: these are the 64-bit packages.
    • noarch: these are independent of the architecture.
  • Ubuntu/Fedora often splits packages into smaller pieces:
    • The header files are usually called something like:
      • deb: usually <package>-dev
      • rpm: usually <package>-devel
    • The library you will need to link against:
      • If applicable, lib<package> or something similar.
    • The binaries (executables), often provided by just <package>.
    • Most relevant for C and C++, but also Python and others.
    • Use the search functionality of your package manager.

Example Development Tool Installation

  • To compile and link against Xrandr (X.Org X11 libXrandr runtime library) on Ubuntu, you would have to install:
    • libxrandr2: the library.
    • libxrandr-dev: the header files.
    • Usually don’t explicitly include the architecture (e.g. x86_64), it’s inferred
    • If you’re getting link errors, try installing explicit 32/64-bit version.
      • Just Google your error
  • Splitting "-devel" files more common for binary package managers, less for source-based ones.

System Specific Package Managers

RHEL/Fedora Package Managers (yum and dnf)

  • Installing and uninstalling:
    • Install a package:
      • dnf install <pkg1> <pkg2> ... <pkgN>
    • Remove a package:
      • dnf remove <pkg1> <pkg2> ... <pkgN>
    • Only one package required, but can specify many.
    • "Group" packages are available, but different command:
      • dnf groupinstall 'Package Group Name'
  • Updating components:
    • Update EVERYTHING: dnf upgrade.
    • update exists, but is essentially upgrade.
      • Specify a package name to only upgrade that package.
    • Updating repository lists: dnf check-update
  • Searching for packages:
    • Same command: dnf search <pkg>

Gentoo package manager (portage with emerge)

  • Source-based package manager: compiles your packages
    • Just runs a special bash script to compile
    • Very, very fine-grained control over dependencies and features
    • Use the latest software tailored to your hardware!
  • USE flags control special "optional" features
    • would be separate packages on Ubuntu
    • Want java or emacs integration? USE="java emacs…"
  • Installing, uninstalling, and updating
    • emerge <pkg> to install
    • emerge -v --depclean to remove
      • Explicitly checks to ensure other packages don’t depend on it first
    • emerge -uND @world to upgrade everything
      • flags are "update", "newuse" (if you turned on a feature), "deep"(also checks dependencies for this stuff)

Cautionary Tales

  • WARNING: if you install package Y, which installs X as a dependency, and later remove Y
    • Sometimes X will be removed immediately!
    • Sometimes X will be removed during a clean-up operation later
  • Solution?
    • Basically, pay attention to your package manager
    • Install packages that you need explicitly
    • Check lists of packages when removing things
  • Why does this happen at all?
    • Distros split things into dependencies: avoids lots of extra copies
    • Side effect: dependencies are visible to you; you can use directly
    • In Windows: dependencies are hidden

Package Management is a core Philosophy

  • Most of what makes a Linux distribution is its package manager
  • Reflects the distribution’s philosophy
    • Ubuntu: "just work" and don’t think too hard+keep it stable
    • Fedora: "latest everything" but keep it stable+not too hard
    • Arch: I want to understand how my distro works+"latest everything"
    • Gentoo: I do understand how my distro works+"latest everything"

If you’re thinking of installing Linux, by the way…

  • Ubuntu

    • Benefits: easy install, out-of-the-box setup, common things "just work"
    • Drawbacks: too much magic; system "just work" scripts break if you need to do too many uncommon things and aren’t really careful

  • Fedora

    • Benefits: still pretty easy to install, lots of good "get started
      quick" stuff
    • Drawbacks: marginally less stable; can change deep system things
      but also not hard to break your system that way.

If you’re thinking of installing Linux, by the way 2…

  • Archlinux
    • Benefits: wealth of knowledge, really helps you understand why your system works and what makes it work, good community, similar to Gentoo, great way to learn Linux, but without the time-consuming compile times
    • Drawbacks: limited "automagic". Takes real time to set things up, or change important things
  • Gentoo
    • Benefits: similar to Arch, plus the source-based Portage package manager is pure gold. Great if you’re doing serious programming/systems work, or if you really need a thing from Github that was released last week, or you have a limited environment. Great way to really learn Linux.
    • Drawbacks: absolutely no automagic. Takes real time to set things up, compiling is time-consuming, all the docs think you know what you’re doing.

macOS Package Management: Install "brew" on your own

  • Sitting in class right now with a Mac?
  • DON'T DO THIS IN CLASS. You will want to make sure you do not have to interrupt the process.
  • Make sure you have the "Command Line Tools" installed.
    • Visit https://brew.sh/
    • Copy-paste the given instructions in the terminal as a regular user (not root!).
  • VERY IMPORTANT: READ WHAT THE OUTPUT IS!!!! It will tell you to do things, and you have to do them. Specifically, you should run 'brew doctor' BEFORE you install anything!

PS: Brew can also work on Linux, but I've never seen a good reason to do so, as all Linux distribution package managers are much more advanced than brew.

macOS Package Management (Homebrew, aka. "brew")

  • Installing and uninstalling:
    • Install a formula (i.e., package):
      brew install <fmla1> <fmla2> ... <fmla2>
      Remove a formula:
      brew uninstall <fmla1> <fmla2> ... <fmlaN>
    • Only one fmla required, but can specify many.
    • "Group" packages have no meaning in brew.
  • Updating components:
    • Update brew, all taps, and installed formulae listings. This does not update the actual software you have installed with brew, just the definitions: brew update.
    • Upgrade all installed formulae: brew upgrade.
      • Specify a formula name to only upgrade that formula.
  • Searching for packages:
    • Same command: brew search <formula>

macOS: One of These Kids is Not Like the Others (Part I)

  • Safe: confines itself (by default) in /usr/local/Cellar:
    • Common feature of "non-system" package managers
    • No sudo, plays nicely with macOS (e.g., Applications, python3).
    • Non-linking by default. If a conflict is detected, it will tell you.
    • Really important to read what brew tells you!!!
  • brew is modular. Additional repositories ("taps") available:
    • This concept exists for all package managers
  • Common taps people use:
    • brew tap homebrew/science
      • Various "scientific computing" tools, e.g., OpenCV.
    • brew tap caskroom/cask
      • Install .app applications! Safe: installs in the "Cellar"
      • For example, brew cask install vlc
      • Symlinks to ~/Applications, but now these update with brew all on their own when you brew upgrade!

macOS: One of These Kids is Not Like the Others (Part II)

  • brew installs formulae
    • A ruby script that provides rules for where to download something from and how to compile it. Similar concept to Gentoo's portage
  • Sometimes the packager creates a "bottle":
    • If a bottle for your version of macOS exists, you don’t have to compile locally.
    • The bottle just gets downloaded and then "poured".
  • Otherwise, brew downloads the source and compiles locally.
  • Though more time-consuming, can be quite convenient!
    • brew options opencv
    • brew install --with-cuda --c++11 opencv
    • It really really really is magical. Just like USE flags in Gentoo!
    • brew reinstall --with-missed-option formula

macOS: One of These Kids is Not Like the Others (Part III)

  • Reiteration: pay attention to brew and what it says. Seriously.
  • Example: after installing opencv, it will return:
==> Caveats
Python modules have been installed and Homebrews site-packages
is not in your Python sys.path, so you will not be able to
import the modules this formula installed. If you plan to
develop with these modules, please run:
mkdir -p /Users/sven/.local/lib/python2.7/site-packages
echo 'import site; site.addsitedir( \
"/usr/local/lib/python2.7/site-packages")' >> \
/Users/sven/.local/lib/python2.7/site-packages/homebrew.pth
  • brew gives copy-paste format, above is just so you can read.
  • I want to use opencv in Python, so I do what brew tells me.

Language-specific package management

  • Modern programming language environments have their own package managers
    • Haskell: cabal
    • Ocaml: opam
    • Python: conda/pip/pip3
    • Ruby: bundler / gem
    • Rust: cargo
  • Work basically exactly like brew and Linux distro package managers, but concerned only with packages relevant to the programming language
    • Separate, user-specific, install directory
    • Preferred to system packages but does not replace them
  • Be careful when using these!
    • System packages are not preferred, but sometimes get used anyway
    • When languages rely on external packages, things can get really hairy

Other Managers

  • There are so many package managers out there for different things, too many to list them all!:
    • Ruby: gem
    • Anaconda Python: conda
    • Python: pip
    • LaTeX: tlmgr (uses the CTAN database)
      • Must install TeX from source to get tlmgr
    • Perl: cpan
    • Sublime Text: Package Control
    • Many, many others… see here

Some notes and warnings about Python package management

  • Notes:
    • If you want X in Python 2 and 3:
      • pip install X and pip3 install X
      • macOS Specifically: advise only using brew or conda. The system-installed python can get really damaged if you modify it, you are better off leaving it alone
      • So even if you want to use python2 on Mac (though, you really shouldn't want to), you are strongly encouraged to install it with brew
  • Warnings:
    • Don’t mix easy_install (Python 2) and pip. Choose one, stick with it.
      • But the internet told me if I want pip on Mac, I should easy_install pip
      • NO! Because this pip will modify your system python, USE BREW.
  • Don’t mix pip with conda. If you have Anaconda Python, just stick to using conda

Concepts in language-specific (per-user) package management

  • Packages do not require root to install
  • Packages installed to per-user directory
    • Normally a "dotfile" directory in your home
    • Better-behaved things in ~/.local/share
  • Need to change your environment variables to use correctly
    • Usually at least $PATH and $LD_LIBRARY_PATH
    • Sometimes also $JAVA_HOME, $PYTHON_PATH, etc.
  • Can control selection of package managers with edits to $PATH

Useful Resources

Well that is my (Seb's) opinion

MSI Micrssoft Software Installer

RHEL - Red Hat Enterpise Linux EOSL End of Service Life

xrandr is an official configuration utility to the RandR (Resize and Rotate) X Window System extension