MongoDB OSX Setup

1.) Install brew. Go to https://brew.sh/
     Then copy the installation script then paste it to terminal
     /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

2.) Install node.
     brew install node

3.) Install mongodb.
     brew install mongo

4.) Test it. Go to terminal and write
      mongod

You should be able to see 'waiting for connections on port 27017' at end of the logs
   



Two ways of starting the mongodb server

1.) By calling the mongo itself in the terminal
      mongod


2.) By starting the services itself. This will automatically start mongodb at any time.
      brew services start mongo

Apache Spark Code Collection

Read 'csv' File

lines = sc.textFile("..../u.data")

Get the first line of  an RDD file type

lines = sc.textFile("..../u.data")

firstRow=lines.first()

Count the number by Appearance and Show the Results

lines = sc.textFile("/Users/edmondlegaspi/Desktop/Datasets/u.data")

ratings = lines.map(lambda x: x.split()[2])

results = ratings.countByValue()

sortedResults = collections.OrderedDict(sorted(results.items()))

for key, value in sortedResults.items():
    print(key, value)

Install Apache Spark 2.0 on Amazon EC2 - Quick Setup

1.) Install java
     sudo yum install java-1.8.0-openjdk

2.) Download the apache spark

     wget http://mirror.rise.ph/apache/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz

3.) Unzip it

    tar -xvf spark-2.4.0-bin-hadoop2.7.tgz

4.) Create a symbolic Link

     ln -s spark-2.4.0-bin-hadoop2.7 spark

5.) Edit the bashprofile

     sudo nano .bashrc

    

    Add the following codes:

    export SPARK_HOME=/home/ec2-user/spark 

    export PATH=$PATH:$SPARK_HOME

6.) Save changes with .bashrc

      . .bashrc

7.) Test spark

      spark-shell

Installing Apache Kafka

You can set up Apache Kafka by using brew or downloading the binary.

Installing via Binary Download

1.) Go to apache kafka site

2.) Under Binary downloads, download
    "Scala 2.12  - kafka_2.12-2.0.0.tgz (asc, sha512)"

3.) Open a terminal, move the downloaded file to the root directory then extract
     mv Downloads/kafka_2.12-2.0.0.tgz .
     tar -xvf kafka_2.12-2.0.0.tgz

4.) Verify if it works
      cd kafka_2.12-2.0.0
      bin/kafka-topics.sh

    4.1) (If it doesn't work)
          brew tap caskroom/versions
          brew cask install java8

Installing Kafka via Brew

1.)  Open a terminal,

      brew install kafka

      (That's it)

Note: The one notable difference between installing via brew vs binary is that if you install by brew,

          there's no need to add the .sh extension to run kafka commands

Advanced Indexing from Multi-Index/Pivoted Dataframe

In this tutorial we will learn how to select row x column  in python multi-level dataframe using loc() function .  Lets see with an example

Consider the following data:

d = {
    'app' : ['J', 'J', 'J', 'J', 'J', 'J', 'J', 'J', 'J', 'B'],
    'geo':
        ['US', 'US', 'US', 'Asia', 'Asia', 'Asia', 'Europe', 'Europe', 'Europe', 'US'],
    'device':
        ['iPhone', 'iPad', 'Android', 'iPhone', 'iPad', 'Android', 'iPhone', 'iPad', 'Android', 'iPhone'],
    'cost':
        [4500, 4000, 4500, 2000, 2000, 2500, 500, 500, 500, 250]}

df = pd.DataFrame(d)

and a pivot out of it

df_pivoted =  pd.pivot_table(df, index = 'app', columns=['geo', 'device'], values=['cost'], aggfunc=np.sum)

and you want to select the cost under row 'B' with geo 'Asia'

df.loc[['J'], ('cost', 'Asia')]

Lorenz Equations with Python

import matplotlib.pyplot as plt
import numpy as np

vSigma = 10
vBeta = 8/3
vRho = 28


def f1(t,x,y,z):
    f1 = vSigma*y - vSigma*x
    return f1

def f2(t,x,y,z):
    f2 = vRho*x - y - x*z
    return f2

def f3(t,x,y,z):
    f3 = -vBeta*z + x*y
    return f3


print("Sample input: lorenzEquation(0,20,200,5,5,5)")

def lorenzEquation(first,second,N,alpha1,alpha2, alpha3):
    h = (second-first)/N
    t = first

    w1 = alpha1
    w2 = alpha2
    w3 = alpha3

    A = [w1]
    B = [w2]
    C = [w3]
    tempTime = alpha1
    time = [alpha1]
    
    A1 = [w1]
    B1 = [w2]
    C1 = [w3]


    for num in range(1,N):
        k1x = h*f1(t,w1,w2,w3)
        k1y = h*f2(t,w1,w2,w3)
        k1z = h*f3(t,w1,w2,w3)
        
        k2x = h*f1(t + (h/2), w1 + (k1x/2), w2 + (k1y/2), w3 + (k1z/2))
        k2y = h*f2(t + (h/2), w1 + (k1x/2), w2 + (k1y/2), w3 + (k1z/2))
        k2z = h*f3(t + (h/2), w1 + (k1x/2), w2 + (k1y/2), w3 + (k1z/2))
        
        k3x = h*f1(t + (h/2), w1 + (k2x/2), w2 + (k2y/2), w3 + (k2z/2))
        k3y = h*f2(t + (h/2), w1 + (k2x/2), w2 + (k2y/2), w3 + (k2z/2))
        k3z = h*f3(t + (h/2), w1 + (k2x/2), w2 + (k2y/2), w3 + (k2z/2))
        
        
        k4x = h*f1(t + h, w1 + k3x, w2 + k3y, w3 + k3z)
        k4y = h*f2(t + h, w1 + k3x, w2 + k3y, w3 + k3z)
        k4z = h*f3(t + h, w1 + k3x, w2 + k3y, w3 + k3z)
        
        w1 = w1 + (1/6)*(k1x + 2*k2x + 2*k3x + k4x)
        w2 = w2 + (1/6)*(k1y + 2*k2y + 2*k3y + k4y)
        w3 = w3 + (1/6)*(k1z + 2*k2z + 2*k3z + k4z)

        A.append(w1)
        B.append(w2)
        C.append(w3)
        tempTime = alpha1 + num*h
        time.append(tempTime)

    w1 = alpha1 + 0.001
    w2 = alpha2
    w3 = alpha3


    for num in range(1,N):
        k1x = h*f1(t,w1,w2,w3)
        k1y = h*f2(t,w1,w2,w3)
        k1z = h*f3(t,w1,w2,w3)
        
        k2x = h*f1(t + (h/2), w1 + (k1x/2), w2 + (k1y/2), w3 + (k1z/2))
        k2y = h*f2(t + (h/2), w1 + (k1x/2), w2 + (k1y/2), w3 + (k1z/2))
        k2z = h*f3(t + (h/2), w1 + (k1x/2), w2 + (k1y/2), w3 + (k1z/2))
        
        k3x = h*f1(t + (h/2), w1 + (k2x/2), w2 + (k2y/2), w3 + (k2z/2))
        k3y = h*f2(t + (h/2), w1 + (k2x/2), w2 + (k2y/2), w3 + (k2z/2))
        k3z = h*f3(t + (h/2), w1 + (k2x/2), w2 + (k2y/2), w3 + (k2z/2))
        
        
        k4x = h*f1(t + h, w1 + k3x, w2 + k3y, w3 + k3z)
        k4y = h*f2(t + h, w1 + k3x, w2 + k3y, w3 + k3z)
        k4z = h*f3(t + h, w1 + k3x, w2 + k3y, w3 + k3z)
        
        w1 = w1 + (1/6)*(k1x + 2*k2x + 2*k3x + k4x)
        w2 = w2 + (1/6)*(k1y + 2*k2y + 2*k3y + k4y)
        w3 = w3 + (1/6)*(k1z + 2*k2z + 2*k3z + k4z)

        A1.append(w1)
        B1.append(w2)
        C1.append(w3)
    #
    plt.plot(time, A)
    plt.plot(time, A1)
    plt.legend(['Initial condition [5,5,5]', 'Initial condition [5.001,5,5]'], loc='upper left')
    plt.show()

    plt.plot(A, B)
    plt.show()
    
    plt.plot(A, C)
    plt.show()

    return



Lotka Voltera Equations with Python

import matplotlib.pyplot as plt
import numpy as np

a = 1.2
b = 0.6
c = 0.8
d = 0.3

def f1(t,x,y):
    f1 = a*x - b*x*y
    return f1

def f2(t,x,y):
    f2 = -c*y + d*x*y
    return f2


print("Sample input:  predatorPrey(0, 30, 300, 2, 1)")


def predatorPrey(first,second,N,alpha1,alpha2):
    h = (second-first)/N
    t = first
    
    w1 = alpha1
    w2 = alpha2
    
    A = [w1]
    B = [w2]
    tempTime = alpha1
    time = [alpha1]
    
    for num in range(1,N):
        k1x = h*f1(t,w1,w2)
        k1y = h*f2(t,w1,w2)
        
        k2x = h*f1(t + (h/2), w1 + (k1x/2), w2 + (k1y/2))
        k2y = h*f2(t + (h/2), w1 + (k1x/2), w2 + (k1y/2))
        
        k3x = h*f1(t + (h/2), w1 + (k2x/2), w2 + (k2y/2))
        k3y = h*f2(t + (h/2), w1 + (k2x/2), w2 + (k2y/2))
        
        
        k4x = h*f1(t + h, w1 + k3x, w2 + k3y)
        k4y = h*f2(t + h, w1 + k3x, w2 + k3y)
        
        w1 = w1 + (1/6)*(k1x + 2*k2x + 2*k3x + k4x)
        w2 = w2 + (1/6)*(k1y + 2*k2y + 2*k3y + k4y)
        
        A.append(w1)
        B.append(w2)
        tempTime = alpha1 + num*h
        time.append(tempTime)
        
    
    plt.plot(time, A)
    plt.plot(time, B)
    plt.legend(['x, prey', 'y, predator'], loc='upper left')
    ax.grid()
    ax.set_xlabel("Time (h)")
    plt.show()
    
    plt.plot(A, B)
    plt.show()
    
    return

Regula Falsi Or Method of False Position with Python

Regula Falsi or Method of False Position

     The regula falsi method iteratively determines a sequence of root enclosing intervals, $(a_n, b_n)$, and a sequence of approximations, which shall be denoted by $p_n$. Similar to the bisection method, the root should be in ther interval being considered. During each iteration, a single point is selected from $(a_n, b_n)$ to approximate the location of the root and serve as $p_n$. If $p_n$ is an accurate enough approximation, the iterative process is terminated. Otherwise, the Intermediate Value Theorem is used to determine whether the root lies on the subinterval $(a_n, p_n)$ or the subinterval $(p_n, b_n)$. The entire process is then repeated on that subinterval. It was developed because the Bisection method converges at a fairly slow rate.

Let f be a continuous function on the interval $[a,b]$ s.t. $f(a) \cdot f(b) < 0$, locate the point $(p1,0)$ where the line joining the points $(a, f(a))$ and $(b,f(b))$ crosses the x-axis. Hence,
       $$p_1 = b -  \frac{f(b)(b-a)}{f(b)-f(a)} = \frac{af(b) - bf(a)}{f(b) - f(a)}$$

Algorithm

To find a solution to $f(x) = 0$ given the continuous function $f$ on the interval $[a, b]$, where $f(a)$ and $f(b)$ have opposite signs:

INPUT endpoints a, b; tolerance TOL; maximum number of iterations $N_0$.

STEP 1 Set $i = 1$
                     $FA = f(a)$.

STEP 2 While $i \le N_0$ do Steps 3-6.

        STEP 3 Set $p = \frac{af(b) - bf(a)}{f(b) - f(a)}$
                               $FP = f(p)$

        STEP 4 If $FP = 0$ or |f(p)| < TOL  then
                         STOP
                     else OUTPUT(P)
                 
        STEP 5 Set $i = i + 1$

       STEP 6 If $FA \times FP > 0$ then set $a = p$;
                          $FA = FP$
                     else set $b = p$.

STEP 7 OUTPUT("Method failed after $N_0$")

Sample Problem:

Use Regula Falsi method to approximate the solution of $f(x) = x^3 + 2x^2 - 3x - 1 = 0$ within $[1, 2]$ that is accurate to at least within $10^-4$.


For the approximation, see the outpout below:

    n                    $a_n$                                $b_n$                        $p_n$                                  $f(p_n)$
         
   1                     1                                   2                       1.1                                  -0.549            

   2                    1.1                                 2                       1.1517436                      -0.27440072      

   3                    1.1517436                     2                       1.1768409                      -0.13074253      

   4                    1.1768409                     2                       1.1886277                      -0.060875863      

  5                    1.1886277                      2                       1.1940789                      -0.028040938      

  6                    1.1940789                      2                       1.1965821                      -0.01285224      

  7                    1.1965821                      2                       1.1977278                       -0.0058772415    

  8                    1.1977278                      2                       1.1982513                       -0.0026848163    

  9                    1.1982513                      2                       1.1984904                       -0.001225881      

 10                   1.1984904                     2                        1.1985996                       -0.0005596125    

 11                   1.1985996                     2                        1.1986494                        -0.00025543669    

 12                   1.1986494                     2                        1.1986721                        -0.0001165895    

Python Code:

import math
import numpy as np



def f(x):
    f = math.pow(x,3) + 2*math.pow(x,2) - 3*x - 1
    return f
 
 
print("Sample input: regulaFalsi(1,2,10**-4, 100)")
 
def regulaFalsi(a,b,TOL,N):
    i = 1
    FA = f(a)
    
    print("%-20s %-20s %-20s %-20s %-20s" % ("n","a_n","b_n","p_n","f(p_n)"))
     
    while(i <= N):
        p = (a*f(b)-b*f(a))/(f(b) - f(a))
        FP = f(p)
         
        if(FP == 0 or np.abs(f(p)) < TOL):
            break
        else:
             print("%-20.8g %-20.8g %-20.8g %-20.8g %-20.8g\n" % (i, a, b, p, f(p)))
        
         
        i = i + 1
         
        if(FA*FP > 0):
            a = p
        else:
            b = p
     
    return