Behavioral Modelling Example

This article aims to explain the (theoretical) importance of behavioral modelling.

Warning: the data and results shown in this article is artificially generated for the theoretical demonstration. They can never be related to anything realistic.

Suppose a lunch delivery agency. It has a set of data that describe 1) who ordered 2) how many of 3) what kinds of food. For example,

user   menu      quantity
0001   Italian   3
0002   Chinese   4
0001   Korean    2
0002   Italian   5

Someone (from upstairs) wants to know what is the individual preference for each user. Interpreted as, predict the menu at quantity 1 for each user.

Without any kinds of modelling, a behavioral one or not, the data still can be fitted and predicted. However, to add some social scientific reality, let's assume the following:

Network conformity characteristic
- Indicates how much a person wants to follow the group's norm
- A person with low network conformity will insisit that she must eat something she wants
- A person with high conformity will say something like "I will eat whatever you guys want to eat"
- Network conformity is uniformly distributed among users

How the menu is chosen
- Randomly generate a dining group (power law distribution)
- In the group, the person has the lowest network conformity insist that she must choose the menu
- The dice rolls (0.0~1.0, which represents fierce discussions) and if it is higher than her network conformity, she chooses the menu
- If fails, the person with the next lowest network conformity insist that she must choose the menu
- repeat until the menu is decided
- someone orders the lunch

What I want to show is that without the estimation of network conformity, the data contains too much noises (other person's decisions). If naively fitted, it should yield baised predictions. To correct this, first the individual network conformity should be estimated by variance / diversity of menu, next drop the observations with large group size and high network conformity.

Therefore we have three models:
1. Naive fitting, using all the data
2. Estimate ind. network conformity from menu diversity, ignore some data accordingly, then fit
3. Already knows ind. network conformity, ignore some data accordingly (god mode)

Some very specific results:

Naive fitting : 69% accuracy
Estimation    : 76% accuracy
God mode      : 88% accuracy

Again, this experiment is massively tweaked to show the largest performance difference so do not take this "network conformity" as a general power tool in behavioral modelling.

Source codes:
Github Link

MPI Grid Search

Grid Searching Hyperparameters, scikit-learn ,
Naively parallelized with mpi4py



Performance test..




Also see here: Nested Cross-Validation for scikit-learn using MPI

(Manual) Linear Regression with mpi4py

Taking detour from Spark.
Map-reduce linear regression (1-var) with mpi4py.

Unsurprisingly, blazing fast. so fast i wonder what was i doing for the last few months..


Somehow I cannot find well-documented scikit-learn mpi extensions.. No way they have not yet implemented mpi-like parallel functionalities..

Solve Differential Equations with Spark, Part 2

Alright, Here it is. 1D-Lapace Spark Parallel Solver! Somehow this works!!


Result is somewhat different from reference no-parallel solver, and I don't think this should happen, but let's look into that later..

At this point, with 9 meshes, it is a lot slower than no-parallel version. Next time i will test performance.

Questions:

• I do not think I have guaranteed continuous split of meshes. This is very important to me, because to parallelize the computation, each partition should have as much as continuous meshes, but Spark does not explicitly care about this. 
• Broadcasting all the partition edge values is a lot of memory waste. Each partition only needs neighboring (updated) edge values but because of the way broadcasting works, each partition holds all edge values, allocate memory to non-usable information. This is not much for 1-d, but for 3-d or 4-d problem with lots of meshes, this will be critical.
• Maybe there is a way to access local file system to each cluster, so master sends information to shared (memory) folder and each partition can access cluster-customized information. However, it will be too much of micromanagement.
• .mapPartitionWithIndex is a good method, but it gives iteration "generator" (at least for Python). Therefore I needed to be "lazy" (and cache currently generated values) to access 1 step ahead of the generator. This results another memory inefficiency, for 3D or 4D the cache will consume a lot of memory..
• What I do not understand yet is, each partition should already have rdd elements in memory or whatever storage, so making it a "list" instead of "generator" should not make much difference. I have to ask this to Spark forum..
• .filter to collect update edge values. This is simply ridiculously inefficient. However I count not find any other way...

Spark Jobs Screenshot:

##UPDATE: 2015-06-29
I cant manage memory properly with mapPartitionsWithIndex. Increasing number of meshes ends up various memory related issues..

##UPDATE: 2015-06-30
For now, setting higher recursion limit by sys.setrecursionlimit() stabilizes the system. I hope this is just Python specific problem.

Solve Differential Equations with Spark, Part 1

##UPDATE: bugfix

Alright. It's a bit early but I couldnt wait.

1D Laplace, Dirchlet BC, FDM, No weight accelerator, No Parallelism
(See here for the actual problem description)

This is good and all, and I KNOW there are many OpenMPI-EC2 Backend ready for launch at my fingertips, but I am just curious how Spark can do this.

After hours of research, it seems that Spark can do this in 2+ way.

1. Application of Pagerank method:

• https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/SparkPageRank.scala
• If I strictly follow this, I have to always carry s[i-1] and s[i+1] with every s[i] in the same row, 
• Since Spark does not allow .lookup/.filter inside .map, i.e. nested transformation/action  .map(lambda (k,v) : (k,v + someRDD.lookup(k))) 

2. Use or Apply matrix multiplier:

• https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
• If I follow this, I have to broadcast (-_-;;) entire solution vector (S), since it can only be local matrix. This is because actually the same problem with 1.
• Considering that S is the only actual vector in this situation (and it will be very large) this is not very practical
• I think the reason is that usual ML problems need to estimate "estimators, A in AS=B"
• While here we have simple, sparce, known but very large A and equally large unknown S to be solved

3. For each partition, convert (parted) RDD to array and reference away:

• Oh well, then why I am struggling with Spark at the first place? Ironically, currently this is the best solution from memory usage perspective. 
• For 1: x3 for tridiagonal, x5 for pentadiagonal, x7 for..
• For 2: xC, C is number of clusters..
• For 3: x2 Maximum. If referenced, even less. (but not likely)
• But the conversion should take (significant) computing time.

There may be other ways for do this but matrix multiplier code gave me good reason to stop looking.

Other issues such as keep partitioning, repartitioning after every iterations. I will keep looking.

Anyway, after implemented these I will ask for help/comments to Spark User forum.

MLlib, Linear Regression Test

Source:

Result:
I don't know why i am getting those EOF errors. Maybe some bugs in new Spark 1.4.0 package..?

And this MLlib doesnt care about R square of P-values .. maybe because of speed issue? (So far this is very slow indeed...) Calculating them including MSE will require remapping and re-reducing. I am not sure this is the way it should be, but even if it is, it's another business opportunity I guess. (That is, if i survive from poverty until then)

I will try to find out about getting P-values in MLlib trainer in the next session..

Also, Spark does provide distributed matrix data types:
https://spark.apache.org/docs/latest/mllib-data-types.html#distributed-matrix

I will test them with some simple numerical anlaysis examples.. soon..

Manual Regression Test Done

Full Code


But something's wrong.. b1 should not be significant...
I will test this with real mllib regression.

And streaming regression (more like auto-update) looks fun. But PySpark still do not have it. I really don't want to use anything even remotely looks like Java..