There are many command line tools and scripts you can download and install that are very useful for Mac Admins. McxToProfile make-profile-pkg munkipkg quickpkg pycreateuserpkg pkgdiff newosxbook.c. Learn about the key features available in Xcode, the integrated development environment for building apps for Mac, iPhone, iPad, Apple Watch, and Apple TV.
- Installing tools used on this site
This page was created before conda became the wonderful powerhouse it is today. Conda is a package and environment manager that is by far the easiest way to handle installing most of the tools we would want to use.
There is an introduction to conda page here that I highly recommend looking over and trying first for any installation goals.
This page will stay here for archival purposes and to hopefully help if we run into something that conda can’t help with 🙂
ATTENTION!The installation examples here all download and install things into a directory called
~/happy_bin. If you are new to this stuff and want to be able to follow the examples here more closely, be sure to familiarize yourself with what the PATH is here and to follow these instructions in order to create that directory and add it to your PATH.
As we’ve seen in the Unix crash course and real-life examples sections, you can do some amazing things with standard Unix commands as far as manipulating plain-text files goes. But that’s just the start of things. If you’re working towards more bioinformatic-leaning applications, you’re certainly also going to need to download and/or install lots of tools that don’t come standard. What’s required on our end to get something working properly varies by the tool, and most that are highly used by people have excellent documentation on how to properly download and install them. This page will be an ongoing list of the installation process for any of the tools used on tutorials from this site, and they will hopefully serve as examples to help guide you through installing other things. Many programs have differences between which version we’d need and how we’d install them on our personal computer vs installing on a server. I plan to add a section soon covering those differences, but for now these are examples for installing on a Mac personal computer and would need to be adjusted for other systems.
You may have heard or read an expression before referring to whether there is a binary or executable available for download for a particular program. Programs stored in these formats can just be downloaded and then utilized usually with no other work required – they are already “built” when you download them. This is in contrast to programs that you need to download and then “compile from source” – you need to “build” the program first in order for it to work. As usual there is a give and take here, binaries are easier to grab and get running, but you have more freedom and often access to the newest developments when installing something from its source code. There is a good explanation on stack exchange about some of these differences. The examples here so far are all binaries (as I’ve been selecting programs like that on purpose for ease of use in tutorials), but I will try to get an example up soon of installing from source.
Often a big part of getting things to work properly is having them in a location on the computer that you can access no matter where you are. As we covered here, a list of directories that are scanned for programs automatically by your computer is stored in the special variable called “PATH”. For the sake of simplicity, many tools we use in tutorials on this site are going to be put in a directory called
~/happy_bin. You can check to see if this is in your PATH already like this:
If this returns something like this:
You’re good to go. If nothing was returned, then you can create this directory and add it to your PATH like such:
And if you’re unsure of what’s going on here, be sure to visit the modifying your PATH page.
As explained above, most things we install here will be put in a directory in our home location called
~/happy_bin, so pay attention to modify any code here accordingly if you want to put things somewhere else. If you’re unfamiliar with what’s going on in any of these code blocks below, and want to be more familiar, run through the Unix crash course and modifying your PATH pages sometime 🙂
My BioInformatics Tools (bit)
These are a collection of one-liners and short scripts I use frequently enough that it’s been worth it for me to have them instantly available anywhere. This includes things like:
- downloading NCBI assemblies in different formats by just providing accession numbers (
- pulling out sequences by their coordinates (
- splitting a fasta file based on headers (
- renaming sequences in a fasta (
- pulling amino acid or nucleotide sequences out of a GenBank file (
And other just convenient things to have handy like removing those annoying soft line wraps that some fasta files have (
bit-remove-wraps) and printing out the column names of a TSV with numbers (
bit-colnames) to quickly see which columns need to be provided to things like
awk. Some require biopython and pybedtools, but all is taken care of if you use the the conda installation 🙂
Each command has a help menu accessible by either entering the command alone or by providing
-h as the only argument. Once installed, you can see all available by entering
bit- and pressing tab twice.
Some of these require the python packages BioPython and pybedtools (as marked at the github repo). Fortunately we can use pip (a python package manager).
In the unlikely case you are using a python2 version < 2.7.9 or a python3 version < 3.4, then you may not have pip. But there are helpful instructions for getting it at the pip installation page.
If you’re dancing in the bioinformatics world, at some point you will no doubt find yourself wanting to download a massive amount of gene sequences or reference genomes or other information from the glorious NCBI. One of the ways you can download things in bulk from the command line is using their EDirect command-line tools.
It can be tricky to use these, I’ve struggled quite a bit with it, but it’s also very powerful and sometimes the only way to get what I’ve needed. I have some examples up on this page.
This link has the following installation instructions, which I’ve been able to execute successfully on both some Mac and on a Linux systems, but have run into issues on others. Conda has been much more consistently successful for me (that’s pretty much true for every program 🙂).
Copying and pasting these commands into your terminal should do the trick:
This downloads and installs EDirect. The last step may take a minute or two, and when it’s done it might tell you a command you need to run in order to add the appropriate directory to your PATH. Copy and paste that command into the terminal, open a new terminal window or run
source ~/.bashrc (or
source ~/.bash_profile if it had you add it to that file). Then you’re ready to test out that all is well by running
esearch -help, and you should hopefully see something like this:
And you can find some example usage here.
There are instructions to get vsearch up and running on its github, but these commands should work for you if you’re on a Mac (if you’re not, you’ll have to download a different version you can find following the above link).
Here we changed into our
~/happy_bin directory, downloaded the vsearch tool with
curl, unpacked it and unzipped things with
tar, copied the main executable file into our
~/happy_bin directory so that it is in our PATH and can be called from anywhere, then finally we deleted the compressed downloaded file. Lastly, one way we can quickly test that the program seems to be working as it should be is by checking the version:
Which should return something like this:
To get the free version of usearch, you first need to go to https://www.drive5.com/usearch/download.html, and fill out a (very) short form in order to have a download link sent to you. This usually happens virtually instantly. After getting the link and downloading the file, assuming you’re working on a Mac and you downloaded the same version as noted above (v10.0.240) into your default download directory, the following commands will move usearch into our
~/happy_bin directory and change its properties so we can execute it (if you download a different version, adjust the commands accordingly):
Here we changed into our
~/happy_bin directory, moved the downloaded file from our downloads directory to here and renamed it to simply “usearch” with the
mv command, and then ran the
chmod command to make the program executable so we can actually run it. Sometimes this
chmod step is needed and sometimes it isn’t depending on how the developer packaged the program. If we had tried to run
usearch before running the
chmod command, we would have gotten an error saying permission denied. We can check the permissions and properties ascribed to a file or directory by using the
ls command with the
-l optional argument:
The details of what all of the letters and their positions mean here are a bit further into the weeds than we need to worry about right now, but quickly, an “r” is for readable, a “w” is for writable, and an “x” is for executable. Note that the vsearch executable file has an “x” in the position I highlighted for the usearch file. After running
chmod +x usearch, the output of
ls -l looks like this:
And now the usearch program can be executed (run) without throwing any errors, which we can again check quickly by asking for the version:
FastQC is a handy and user-friendly tool that will scan your fastq files to generate a broad overview summarizing some useful information, and possibly identify any commonly occurring problems with your data. But as the developers note, its modules are expecting random sequence data, and any warning or failure notices should be interpreted within the context of your experiment.
A link to download the current version of fastqc for your system can be found here. And there are pretty good instructions for different platforms provided here, including possibly needing to update your Java installation. For OSX there is a disk image, or application bundle, available to download. So if you’re on a Mac, you can download the “fastqc_v*.dmg” file from the above downloads link, open it to install the program, and then move the resulting “FastQC.app” file (which is actually a directory) into your
~/Applications directory or into your
~/happy_bin if you’d like. I haven’t gone through this on anyone else’s computer, but on mine the installation itself seems adds the executable file to my
/usr/local/bin so that we can call it from anywhere. We can check this again by asking for which version:
And an example usage at the command line to run two fastq files through would look like this:
You can list multiple files like this delimited by a space, and the program is capable of dealing with gzipped files. This will produce a .html file for each fastq file you run it on that you can then open and analyze.
Trimmomatic is a pretty flexible tool that lets you trim and/or filter your sequences based on several quality thresholds and some other metrics (e.g. minimum length filtering, or removing adapters). It runs as a java program, so the same binary seems to work across systems. The binary can be downloaded from this link, and here is one way to do this at the command line to grab the current version at the time I’m putting this together:
At this point, Trimmomatic, should be ready to rock, and we can again check quickly by asking for the version:
I never figured out how to get a .jar to be callable from anywhere without providing the full path like I’ve done here, so this is just how I run it (always providing the full path). If you’re reading this and you do know how to do that, please shoot me a message 🙂
sabre is an awesomely simple and quick tool for demultiplexing your samples and trimming off the barcodes. The installation seems to run smoothly wherever I’ve tried it, and the usage examples on their github are very straightforward. Here’s how I installed it on my mac:
Brian Bushnell has made a very handy set of tools called bbtools (also called bbmap). Installation instructions from the developer are available here, but here are steps I used to install it on my MacOS in our
Since bbtools comes with many programs, stored within the directory we just untarred, it may be preferable to add that directory to our PATH, rather than copying all of the programs to our location here. So here we are adding another directory to our PATH:
And then checking the installation using one of the programs:
If this finds the program and tells you the version (even if there are some memory messages in there), then you’re good to go!
The illumina-utils library provides a suite of tools for working with Illumina paired-end data put out by merenlab.org. I most commonly use some of the quality filtering programs they make available. A few paths to installation can be found here, which are pretty straightforward other than you do need to be working in a python 3 environment. Since the world is amid the switch from python 2 to python 3, this may complicate things for you in some cases. The easiest way around it I’ve found is working with virtual environments, which I hope to add in soon. For now, since I have a python 3 setup on my computer that I call with
python3, and a pip for it that I call with
pip3, I can install this way:
You can then see a list of all the programs by typing
iu- and hitting tab twice:
And to be sure things are functioning properly, and the correct version of python is being utilized, we can check for a version:
I hope to get to virtual environments soon, I apologize if I’m leaving you hanging on this for the moment!
QUAST is a really nice tool for comparing multiple assemblies, and for metagenome assemblies there is a comparable MetaQUAST. Some discussion and example usage can be found here. To install on my personal computer, I followed the instructions laid out here, and, because of the way QUAST compiles things as needed if used, I added its location to my PATH:
And testing it’s actually callable:
SPAdes is an assembly program. You can read some of my thoughts on assemblies here. SPAdes is packaged as a binary, and the developers provide excellent installation instructions here. This is how I installed the latest release at the time on my local computer:
Since SPAdes comes with many programs, stored in a subdirectory of the SPAdes directory we just untarred, it may be preferable to add that directory to our PATH, rather than copying all of the programs to our location here. So here we are adding another directory to our PATH:
And then checking the installation:
MEGAHIT is another assembly program that is great on memory requirements and speed. This can be installed from source, or through
git as noted on the above linked page, but there are also binaries available here. As there is an extra layer of complexity due to OSX
g++, noted here, I grabbed the latest available binary for Mac OSX, which only seems to be missing one bug fix I haven’t happened to run into:
And quick testing, all seems to be well:
NCBI’s Magic-BLAST is a tool based on general BLAST principles but built to deal with high-throughput data, like Illumina reads, considers paired-reads, and can deal with fastq files. It has binaries available for mac, linux, and windows which can be downloaded here. Here’s how I grabbed it:
Install Command Line Tools Mac Xcode
Mac users with macOS Mojave and macOS Catalina, and new operating systems in place can now install Command Line Tools from the Xcode IDE without needing to install the entire Xcode package, or opening an Apple developers account.
Did you know: Using Xcode app leaves behind a lot of junk files a.k.a. 'development junk.' These files are usually hidden from view but you can delete them with a Mac maintenance app CleanMyMac X.
In this article, we cover how you can install this optional and highly useful Command Line Tools package.
X code 11. Credit: developerinsider
What is the Xcode Command Line Tools package?
For Mac power users — we wouldn't recommend downloading this unless you are comfortable with Terminal — the Xcode Command Line Tools package gives you a complete Unix toolkit accessible through Terminal. No developer account needed and you don't need to download the entire — and quite large Xcode package of executables.
Within the Xcode Command Line toolkit, Mac users gain access to numerous useful tools, utilities, and compilers, including make, GCC, clang, perl, svn, git, size, strip, strings, libtool, cpp, and many others. All of these commands are a default part of Linux systems and programs.
Install Command Line Developer Tools Mac
We recommend following these steps for those with the following operating systems running on a Mac: macOS 10.13 High Sierra, macOS 10.14 Mojave, and macOS 10.15 Catalia onward. It isn't always possible to download these Xcode Command Line Tools, following these steps, on Mac’s running older operating systems. Other ways to install command tools and gcc (without needing Xcode) is available through the Apple Developer website.
Here is how you install Xcode Command Line Tools.
Install Command Line Tools Mac
How to install Xcode Command Line Tools?
- Go to Terminal in /Applications/Utilities/.
- Input the following command string in Terminal:
- In the same way when you are downloading new software and apps, a popup update window will appear asking you: “The xcode-select command requires the command line developer tools. Would you like to install the tools now?”
- Select confirm by clicking Install.
- Wait for the Xcode Command Line Tools package to install. It is around 130 MB and usually installs fairly quickly; although it depends on your connection.
- Once everything is installed, the installer goes away and you should be able to any of the new commands that you’ve now got access to. Enjoy using your new Unix command line tools!
Alternative way to install command line tools
If the Terminal combination didn't work, open the Finder > Go to Folder...menu.
Paste in the following path:
In that location, find the app called Install Command Line Developer Tools.app — it will have a blue globe icon.
Click this app and then try the Terminal method explained above.
With this new download, you should have access to 61 Unix command line tools. For example, one of the advantages of having these tools is you can install new apps and software directly from the source code instead of needing to go through the package manager and usual download route.
To access or view everything you've now got, go to the following directory:
Please note, this is the root /Library of your macOS/OS X, not the ~/Library directory.
All of these tools can also be found in:
These tools like small executable apps (which they are):
What happens if I encounter problems downloading these?
If you get an error message that says “Can’t install the software because it is not currently available from the Software Update server”, it means you've already got the Xcode package on your Mac. Mac OS X 10.9 onward, Xcode was already installed, which is why you aren't able to download these tools. However, what you can do is uninstall the entire Xcode app if you'd prefer to only access these tools and not a whole load of software that isn’t going to be of much use.
Watch out for Xcode junk
The Xcode junk is one of those types of clutter that is keeps accumulating in remote places on your Mac. It could take up a few gigs of your space. The only app that seems to address this problem is CleanMyMac X by MacPaw. It’s loved by many Mac developers because it collects those specific types of development junk, like Xcode or outdated libraries.
Once you launch the app, click on System Junk > Scan. Then, click “Review Details”
CleanMyMac X is a powerful Mac performance improvement app. It makes your Mac as good as new. Let it scan your system and clear out any unwanted Xcode, development and system junk that is taking up too much space and cluttering up your Mac. In a few clicks, your Mac could be running smoother, quicker and more efficiently.