Spark Shell
Shutdown info log in spark-shell:
cd $SPARK_HOME/conf // here it's ~/apps/spark-1.0.2-bin-hadoop2/conf
cp log4j.properties.template log4j.properties
sed -i '2s/INFO/WARN/g' log4j.properties
The sed command here replace "INFO" with "WARN" in the second line of log4j.properties.
on cloud61
Modify /opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/spark/conf/log4j.properties, add:
log4j.appender.filelog=org.apache.log4j.RollingFileAppender
log4j.appender.filelog.layout=org.apache.log4j.PatternLayout
log4j.appender.filelog.File=sparkShell.log
log4j.appender.filelog.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
Customize shell parameters
master and memory
spark-shell --master spark://cloud60:7077 --driver-memory 2g --executor-memory 2g -i wfp-spark.script