MLlib Linear Algebra Acceleration Guide
This guide provides necessary information to enable accelerated linear algebra processing for Spark MLlib.
Spark MLlib defines Vector and Matrix as basic data types for machine learning algorithms. On top of them, BLAS and LAPACK operations are implemented and supported by netlib-java (the algorithms may call Breeze and it will in turn call
netlib-java can use optimized native linear algebra libraries (refered to as “native libraries” or “BLAS libraries” hereafter) for faster numerical processing. Intel MKL and OpenBLAS are two popular ones.
However due to license differences, the official released Spark binaries by default don’t contain native libraries support for
The following sections describe how to enable
netlib-java with native libraries support for Spark MLlib and how to install native libraries and configure them properly.
netlib-java with native library proxies
netlib-java depends on
libgfortran. It requires GFORTRAN 1.4 or above. This can be obtained by installing
libgfortran package. After installation, the following command can be used to verify if it is installed properly.
strings /path/to/libgfortran.so.3.0.0 | grep GFORTRAN_1.4
To build Spark with
netlib-java native library proxies, you need to add
-Pnetlib-lgpl to Maven build command line. For example:
$SPARK_SOURCE_HOME/build/mvn -Pnetlib-lgpl -DskipTests -Pyarn -Phadoop-2.7 clean package
If you only want to enable it in your project, include
com.github.fommil.netlib:all:1.1.2 as a dependency of your project.
Install native linear algebra libraries
Intel MKL and OpenBLAS are two popular native linear algebra libraries. You can choose one of them based on your preference. We provide basic instructions as below. You can refer to netlib-java documentation for more advanced installation instructions.
- Download and install Intel MKL. The installation should be done on all nodes of the cluster. We assume the installation location is $MKLROOT (e.g. /opt/intel/mkl).
- Create soft links to
libmkl_rt.sowith specific names in system library search paths. For instance, make sure
/usr/local/libis in system library search paths and run the following commands:
$ ln -sf $MKLROOT/lib/intel64/libmkl_rt.so /usr/local/lib/libblas.so.3 $ ln -sf $MKLROOT/lib/intel64/libmkl_rt.so /usr/local/lib/liblapack.so.3
The installation should be done on all nodes of the cluster. Generic version of OpenBLAS are available with most distributions. You can install it with a distribution package manager like
For Debian / Ubuntu:
sudo apt-get install libopenblas-base sudo update-alternatives --config libblas.so.3
For CentOS / RHEL:
sudo yum install openblas
Check if native libraries are enabled for MLlib
To verify native libraries are properly loaded, start
spark-shell and run the following code:
scala> import com.github.fommil.netlib.BLAS; scala> System.out.println(BLAS.getInstance().getClass().getName());
If they are correctly loaded, it should print
com.github.fommil.netlib.NativeSystemBLAS. Otherwise the warnings should be printed:
WARN BLAS: Failed to load implementation from:com.github.fommil.netlib.NativeSystemBLAS WARN BLAS: Failed to load implementation from:com.github.fommil.netlib.NativeRefBLAS
If native libraries are not properly configured in the system, the Java implementation (f2jBLAS) will be used as fallback option.
The default behavior of multi-threading in either Intel MKL or OpenBLAS may not be optimal with Spark’s execution model 1.
Therefore configuring these native libraries to use a single thread for operations may actually improve performance (see SPARK-21305). It is usually optimal to match this to the number of
spark.task.cpus, which is
1 by default and typically left at
You can use the options in
config/spark-env.sh to set thread number for Intel MKL or OpenBLAS:
- For Intel MKL:
- For OpenBLAS: