Apperently, .reduce(lambda a,b : a+b) works like this:
(((a+b)+c)+d)+e)....
so, if a rdd has
this will not work
---
for reduce, input and output should be same otherwise it cannot be "chained"..
in above example, first two rows go in as tuples, but output is a scala value (float).
the result of reducing the first two rows, scala, goes into as an input of the the second reducing process. The second reducing process tries to extract element from the scala value (which is the result of the first reducing process) and fails.
therefore below works pretty well.
---
since it returns a tuple.
But what i am trying to get is sum of squares. Since it is "chained", it cannot be calculated by something like:
---
(it gives me value error because chain reaction? gets too big..)
Guess i have to go back to remapping..
No comments:
Post a Comment