spark setup notes

####verify java
java -version

####verify scala
scala -version

####download scala if not available then run version command again
wget http://www.scala-lang.org/files/archive/scala-2.12.0-M5.rpm
rpm -ihv scala-2.12.0-M5.rpm

####download spark binaries
cd /opt
wget http://www-eu.apache.org/dist/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz

####untar/install spark
sudo tar -xzf spark-2.4.0-bin-hadoop2.7.tgz

####create link
ln -s /opt/spark-2.4.0-bin-hadoop2.7 /opt/spark

#### add to .bashrc
export PATH=$PATH:/opt/spark-2.4.0-bin-hadoop2.7/bin

#### validate spark install
spark-shell

Advertisements

Column substring match in unix

Use awk command to match find if a substring match a column then return the entire line

awk ‘index($<colnum>,”<string>”)’ <filename>

Or

awk ‘(index($<colnum>,”<string>”) != 0) {print $0}’ <filename>

Big Data

Big data is no question an innovative tech
Works perfectly for data centric company
What they don’t tell you is the adaption requires you to invest more not just on infrastructure otherwise you will end up with redundant and fragmented data defeating the purpose of big data.

Video call

“Let’s call mommy”
I’m getting this request from the kids quite frequently.
Possible because we’ve been doing video call recently.
So I was like, maybe it’s time to teach them how to use video calling
for you know, some emergency situations

I do have a few options:
FB messenger – it’s fun and what we’ve been using for video call but it require a FB account so..skip
WhatsApp – requires phone number..skip
Skype – supports group chat and only requires email.

Skype looks perfect so I created email for both kids
Installs on my phone, setup my account. Works perfect!
Install on Julian’s book( that’s what they call their iPad) setup account. No problem!
Install on Basti’s book, failed. Skype doesn’t support older version of iOS(9.x.x).
What a bummer! I mean @Microsoft, what are you thinking?!

Google Hangouts comes to the rescue(yes they haven’t retired the app)
Installed and setup on all 3 devices, easy breezy!
Added everyone’s contact(email) to each account
Created a group, perfect! I’m done.

Number of call answered: 4..four!
Number of missed call: 3 so far

Haiz, technology is a love and hate limbo

P.s the wife setup her own because she’s techy like that

Smart home project

Home Automation test phase 1:

Finally give in and got myself a smart speaker. 

Pardon my voice 😛

  • Simple commands
  • TV operation:
  1. switching on
  2. playing YouTube video
  3. switching off
  • Smart light:
  1. switching on
  2. change color
  3. switching off
  • Some Q&A
  • Playing music

ps. Google also responded to my recorded voice even if my voice id is enabled.need to check my settings

#googlehomemini

Awesome Google translate

It’s not a big secret that Google can handle translation quite well. 

But do you know that it can copy the vernacular of the country as well? Totally awesome!

I asked Google to say CUPCAKE in Japanese, Korean, Chinese and Thai.

It can’t speak Filipino though which is a bummer. 

If you are an android user you can test with Google assistant(ok Google). For Mac/PC user, you can go to Google translate website and click the speaker icon
#googlehomemini

Excel mini game, AI ver1

AI engine in action for my excel mini game. Will need to define more strategy. 

My plan is to use VBA but after experimenting with excel formula, was able to compute moves and position by aggregating my data using the position of each piece.

The excel tab containing piece details and aggregated data for the positions and possible moves

Excel mini game beta

Decided to start a side project as refresher for Machine Learning and Artificial Intelligence (AI).

They always say that if you want to learn something, building a game is the best way to go. Below is a video demo of the mini game that I created using excel. 

-It’s based on the game called Salpakan(game of the generals).It is basically a cross between chess and battleship.

https://en.m.wikipedia.org/wiki/Game_of_the_Generals

-the rules is fully incorporated. 

-AI is in place, you can see how P1 piece move from cell to cell.

-Still a crappy AI, need to revise the algorithm. Currently, I’m just ranking the piece based on the position in the board and incorporate randomness on the direction of the movement. 

-will try to improve the AI by building a move datasets then evaluate the move instead of the position of the piece.

-player Vs player is something that I will look into as well. 

-since this is a side project, will post some more update once I make more progress