Mapping should not (n)or does not affect actual performance.

This is so fundamental that there is no reason to test it again but anyway i did it..

This gives me 55 seconds..

This gives me 54 seconds.. not much difference as expected.
....

So just map away and think about efficient reducing later..
@But this map-reduce thingy is not for my ultimate goal.. I wonder what Spark can give me when it comes to that..

Input and output of .reduce must have same type

Apperently, .reduce(lambda a,b : a+b) works like this:

(((a+b)+c)+d)+e)....

so, if a rdd has

this will not work

---
for reduce, input and output should be same otherwise it cannot be "chained"..
in above example, first two rows go in as tuples, but output is a scala value (float).
the result of reducing the first two rows, scala, goes into as an input of the the second reducing process. The second reducing process tries to extract element from the scala value (which is the result of the first reducing process) and fails.


therefore below works pretty well.

---
since it returns a tuple. But what i am trying to get is sum of squares. Since it is "chained", it cannot be calculated by something like:
---
(it gives me value error because chain reaction? gets too big..) Guess i have to go back to remapping..

Studying RDD

wordcount example:



.flatMap is used.. according to here , it "flattens" the mapping results.
For example, in this case usual .map would return something like [['i','am','fat'],['you','too']..

Right.. then i should use some more complicated function than (word,1).

and reduceByKey only except tuples with two element: k,v pairs.

So, map->reduce, map->reduce and life goes on..

@ i need to find a way to paste codes from terminal, with syntax ...

PySpark

I want to check local[*] works with Vbox virtual 4 cores..




[update]

It "seems" working, but i will never be sure until i torture them while monitoring cpu usage..

Start!

Test passed..
Not much but it's a start.