:: Data Analysis

Showing posts with label Data Analysis. Show all posts

2020-05-27

Tensorflow Error : Failed to get convolution algorithm

Tensorflow Error

tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.

Solution

shell
```
export CUDA_VISIBLE_DEVICES=0
```

for tf 2

import os

os.environ['TF_FORCE_GPU_ALLOW_GROWTH'] = 'true'

for tensorflow-gpu v1.8

from tensorflow.compat.v1 import ConfigProto
from tensorflow.compat.v1 import InteractiveSession

config = ConfigProto()
config.gpu_options.allow_growth = True
session = InteractiveSession(config=config)

2019-12-14

face recognizer

2019-11-04

Install Mysql, Mariadb, Postgresql and Mongodb on Docker on Mac

How variables are represented in memory

2018-07-07

Warning message during R startup on Mac

Got the following warning messages, when run R Studio on Mac.

During startup - Warning messages:
1: Setting LC_CTYPE failed, using "C"
2: Setting LC_COLLATE failed, using "C"
3: Setting LC_TIME failed, using "C"
4: Setting LC_MESSAGES failed, using "C"
5: Setting LC_MONETARY failed, using "C"
>

SOLUTION

Open Terminal
Run the following command

$ defaults write org.R-project.R force.LANG en_US.UTF-8

KOREAN

$ defaults write org.R-project.R force.LANG ko_KR.UTF-8

export LC_ALL=en_US.UTF-8

KOREAN

export LC_ALL=ko_KR.UTF-8

Close Terminal
Start R Studio again

2017-01-11

Install Spark on Ubuntu

# Install Java

$ sudo apt install python-software-properties
$ sudo add-apt-repository ppa:webupd&team/java
$ sudo apt update
$ sudo apt install oracle-java8-installer

# Install Scala

$ sudo apt install scala

# Install Spark
Download prebuilt Spark from http://spark.apache.org/downloads.html

$ wget http://d3kbcqa49mib13.cloudfront.net/spark-2.1.0-bin-hadoop2.7.tgz
$ sudo mv spark-2.1.0-bin-hadoop2.7.tgz /opt
$ sudo tar xfz spark-2.1.9-bin-hadoop2.7.tgz
$ sudo ln -s spark-2.1.9-bin-hadooop2.7 spark

# Add environment variables and add path

$ vi ~/.profile

JAVA_HOME=/usr/lib/jvm/java-8-oracle
SCALA_HOME=/usr/share/scala
SPARK_HOME=/opt/spark
PYTHONPATH=$SPARK_HOME/python/lib/pyspark.zip:$SPARK_HOME/python/lib/py4j-0.10-4-src.zip
export JAVA_HOME SPARK_HOME PYTHONPATH
PATH="$HOME/bin:$HOME/.local/bin:$PATH:$SPARK_HOME/bin"

# Edit configuration(Optioinal) to reduce logs displayed

$ cd $SPARK_HOME/conf
$ sudo cp log4j.properties.template log4j.properties
$ sudo vi log4j.properties

FIND : log4j.rootCategory=INFO, console
REPLACE : log4j.rootCategory=WARN, console

# Test if Spark works

$ run-example SparkPi 10

Pi is roughly 3.140963140963141

# Launch Spark Shell(run scala language) or PySpark(run python language)

$ spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0
      /_/
        
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_111)
Type in expressions to have them evaluated.
Type :help for more information.
scala >

$ pyspark
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.1.0
      /_/
Using Python version 2.7.12 (default, Nov 19 2016 06:48:10)
SparkSession available as 'spark'.
 > > >

$ Spark UI

http://localhost:4040

Install Spark on Windows 10

# Download

JAVA from http://www.oracle.com/technetwork/java/javase/downloads/index.html
SCALA from http://www.scala-lang.org/download/
Prebuilt Spark from http://spark.apache.org/downloads.html
WINUTILS from https://github.com/steveloughran/winutils

# Installation

Install Java and Scala
Unzip prebuilt Spark into C:\Usr
Unzip Winutils(hadoop 2.7.1/bin) into C:\Usr\spark-2.10-bin-hadoop2.7\bin

# Set environment variables

JAVA_HOME=C:\Program Files\Java\jdk1.8.0_111
_JAVA_OPTIONS=-Xmx512M -Xms512M
SCALA_HOME=C:\Program Files\scala
SPARK_HOME=C:\Usr\spark-2.1.0-bin-hadoop2.7
HADOOP_HOME=C:\Usr\spark-2.1.0-bin-hadoop2.7

# Add path

%SCALA_HOME%\bin
%SPARK_HOME%\bin

# Grant permissions

winutils.exe chmod 777 C:\tmp\hive

# Run command

> scala -version

Picked up _JAVA_OPTIONS: -Xmx512M -Xms512M
Scala code runner version 2.12.1 -- Copyright 2002-2016, LAMP/EPFL and Lightbend, Inc.

> spark-shell

Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.1.0
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_111)
Type in expressions to have them evaluated.
Type :help for more information.
scala>

2013-08-23

Count number of visitors(IP addresses) with IIS log files

* Instruction

Mount IIS log directory into Linux box

$ sudo mount -t cifs -o ro,username=USER,password=PASSWD //HOST_IP/SHARE /MOUNT_POINT

Run

#!/bin/bash

site_num=(1 2 7 8)
site_addr=("AAA.com" "BBB.com" "CCC.com" "DDD.com")
echo "SITE_NAME DATE WEEK TRAFFIC DB_CONN VISITOR"
for ix in ${!site_num[*]}
do
  d=${site_num[$ix]}
  for f in $(ls W3SVC$d)
  do
    if [[ ${f:4:6} -ge $(date -d 'last monday' +"%y%m%d") && ${f:4:6} -lt $(date +"%y%m%d") ]]
    then
      visitors=$(cat W3SVC$d/$f | grep ^[0-9] | awk '{if(!match($9, /^192.168./) && !match($9, /^10.10./)) print $9}' | sort | uniq | wc -l)
      sendbytes=$(cat W3SVC$d/$f | grep ^[0-9] | awk '{print $16}' | awk '{total=total+$1}END{printf "%.f", total}')
      dbconns=$(cat W3SVC$d/$f | awk '{if(match($5,/^\/$/) || match($5,/asp/)) print $9}' | wc -l)
      weeknum=$(date -dyesterday +%V)
      [ $(date -d yesterday +%u) -eq 1 ] && {mon=$(date -d yesterday +%m.%d); true} || mon=$(date -d last-monday +%m.%d)
      [ $(date -d yesterday +%u) -eq 7 ] && {sun=$(date -d yesterday +%m.%d); true} || sun=$(date -d sunday +%m.%d)
      printf "%s 20%s-%s-%s %s(%s-%s) %s %s %s\n" ${site_addr[$ix]} ${f:4:2} ${f:6:2} ${f:8:2} $weeknum %mon %sun $sendbytes $dbconns $visitors
    fi
  done
done

* Sample result

SITE_NAME DATE WEEK TRAFFIC DB_CONN VISITOR
AAA.com 2013-08-25 34(08.19-08.25) 12329500782 54908 7335
BBB.com 2013-08-25 34(08.19-08.25) 13074318914 66700 6442
CCC.com 2013-08-25 34(08.19-08.25) 23734108053 140609 12877
DDD.com 2013-08-25 34(08.19-08.25) 8913154535 58417 5172