2017-01-31

How to set DYLD_LIBRARY_PATH in Xcode

Open Product > Scheme > Edit Scheme > Run > Arguments
Add DYLD_LIBRARY_PATH under Environment Variables


dyld: Library not loaded: libboost_system.dylib

Got an error when compile boost library and run a sample program.
$ ./PROGRAM
dyld: Library not loaded: libboost_system.dylib
  Referenced from: /PATH/TO/PROGRAM
  Reason: image not found
Abort trap: 6
Check libraries then tried install_name_tool, but failed.
$ otool -L PROGRAM
PROGRAM:
libboost_system.dylib (compatibility version 0.0.0, current version 0.0.0)
libboost_thread.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.4.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.0.0)

$ install_name_tool -change libboost_system.dylib /PATH/TO/libboost_system.dylib PROGRAM
$ install_name_tool -change libboost_thread.dylib /PATH/TO/libboost_thread.dylib PROGRAM
Solution, add the following to .profile or .bash_profile
export DYLD_LIBRARY_PATH=/PATH/TO/BOOST/stage/lib

2017-01-11

Install Spark on Ubuntu

# Install Java

$ sudo apt install python-software-properties
$ sudo add-apt-repository ppa:webupd&team/java
$ sudo apt update
$ sudo apt install oracle-java8-installer

# Install Scala
$ sudo apt install scala
# Install Spark
Download prebuilt Spark from http://spark.apache.org/downloads.html
$ wget http://d3kbcqa49mib13.cloudfront.net/spark-2.1.0-bin-hadoop2.7.tgz
$ sudo mv spark-2.1.0-bin-hadoop2.7.tgz /opt
$ sudo tar xfz spark-2.1.9-bin-hadoop2.7.tgz
$ sudo ln -s spark-2.1.9-bin-hadooop2.7 spark
# Add environment variables and add path
$ vi ~/.profile
JAVA_HOME=/usr/lib/jvm/java-8-oracle
SCALA_HOME=/usr/share/scala
SPARK_HOME=/opt/spark
PYTHONPATH=$SPARK_HOME/python/lib/pyspark.zip:$SPARK_HOME/python/lib/py4j-0.10-4-src.zip
export JAVA_HOME SPARK_HOME PYTHONPATH
PATH="$HOME/bin:$HOME/.local/bin:$PATH:$SPARK_HOME/bin"
# Edit configuration(Optioinal) to reduce logs displayed
$ cd $SPARK_HOME/conf
$ sudo cp log4j.properties.template log4j.properties
$ sudo vi log4j.properties
FIND : log4j.rootCategory=INFO, console
REPLACE : log4j.rootCategory=WARN, console
# Test if Spark works
$ run-example SparkPi 10
Pi is roughly 3.140963140963141
# Launch Spark Shell(run scala language) or PySpark(run python language)
$ spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0
      /_/
        
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_111)
Type in expressions to have them evaluated.
Type :help for more information.
scala >
$ pyspark
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.1.0
      /_/
Using Python version 2.7.12 (default, Nov 19 2016 06:48:10)
SparkSession available as 'spark'.
 > > > 
$ Spark UI
http://localhost:4040

Install Spark on Windows 10

# Download
JAVA from http://www.oracle.com/technetwork/java/javase/downloads/index.html
SCALA from http://www.scala-lang.org/download/
Prebuilt Spark from http://spark.apache.org/downloads.html
WINUTILS from https://github.com/steveloughran/winutils
# Installation
Install Java and Scala
Unzip prebuilt Spark into C:\Usr
Unzip Winutils(hadoop 2.7.1/bin) into C:\Usr\spark-2.10-bin-hadoop2.7\bin
# Set environment variables
JAVA_HOME=C:\Program Files\Java\jdk1.8.0_111
_JAVA_OPTIONS=-Xmx512M -Xms512M
SCALA_HOME=C:\Program Files\scala
SPARK_HOME=C:\Usr\spark-2.1.0-bin-hadoop2.7
HADOOP_HOME=C:\Usr\spark-2.1.0-bin-hadoop2.7
# Add path
%SCALA_HOME%\bin
%SPARK_HOME%\bin
# Grant permissions
winutils.exe chmod 777 C:\tmp\hive
# Run command
> scala -version
Picked up _JAVA_OPTIONS: -Xmx512M -Xms512M
Scala code runner version 2.12.1 -- Copyright 2002-2016, LAMP/EPFL and Lightbend, Inc.
> spark-shell 
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0
      /_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_111)
Type in expressions to have them evaluated.
Type :help for more information.
scala>