python - networkx stochastic_block_model takes too much memory to build

Question

guys! I'm trying to create an SBM from mobility data. I'm calculating sizes_list and probability matrices from the mobility data and trying to use that as input to build the SBM.

My current sizes_list is [3583, 2809, 8, 3250, 1404, 7382, 18, 4450, 4, 1247, 22, 6], with only 12 'blocks'.

And the probability matrix is:

array([[5.26374546e-01, 1.98437064e-01, 2.79095730e-04, 3.60033491e-02, 2.40022328e-02, 1.26988557e-01, 2.79095730e-04, 6.00055819e-02, 0.00000000e+00, 2.17694669e-02, 5.02372314e-03, 8.37287190e-04], [4.39658241e-01, 2.80882876e-01, 7.11997152e-04, 5.01957992e-02, 1.49519402e-02, 1.15343539e-01, 0.00000000e+00, 5.51797793e-02, 0.00000000e+00, 3.20398718e-02, 1.06799573e-02, 3.55998576e-04], [1.25000000e-01, 5.00000000e-01, 0.00000000e+00, 3.75000000e-01, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00], [7.60000000e-02, 5.29230769e-02, 6.15384615e-04, 5.81538462e-01, 4.61538462e-03, 5.96923077e-02, 3.07692308e-04, 5.07692308e-02, 0.00000000e+00, 1.61538462e-01, 1.20000000e-02, 0.00000000e+00], [1.06125356e-01, 2.92022792e-02, 0.00000000e+00, 2.56410256e-02, 6.80199430e-01, 6.55270655e-02, 2.84900285e-03, 5.19943020e-02, 0.00000000e+00, 2.84900285e-03, 3.34757835e-02, 2.13675214e-03], [8.23625034e-02, 3.88783527e-02, 0.00000000e+00, 2.72283934e-02, 2.39772419e-02, 4.46085072e-01, 3.25115145e-03, 3.00325115e-01, 0.00000000e+00, 9.88891899e-03, 6.74613926e-02, 5.41858575e-04], [5.55555556e-02, 5.55555556e-02, 0.00000000e+00, 0.00000000e+00, 5.55555556e-02, 4.44444444e-01, 0.00000000e+00, 3.88888889e-01, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00], [9.61797753e-02, 4.42696629e-02, 0.00000000e+00, 3.03370787e-02, 2.13483146e-02, 3.78876404e-01, 1.79775281e-03, 3.08314607e-01, 0.00000000e+00, 1.61797753e-02, 1.02471910e-01, 2.24719101e-04], [0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 7.50000000e-01, 0.00000000e+00, 2.50000000e-01, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00], [4.57097033e-02, 3.04731355e-02, 0.00000000e+00, 4.29029671e-01, 4.00962310e-03, 5.21251002e-02, 1.60384924e-03, 6.01443464e-02, 0.00000000e+00, 3.63271852e-01, 1.36327185e-02, 0.00000000e+00], [4.54545455e-02, 4.54545455e-02, 0.00000000e+00, 9.09090909e-02, 0.00000000e+00, 3.63636364e-01, 0.00000000e+00, 1.36363636e-01, 0.00000000e+00, 4.54545455e-02, 2.72727273e-01, 0.00000000e+00], [1.66666667e-01, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 1.66666667e-01, 0.00000000e+00, 1.66666667e-01, 0.00000000e+00, 0.00000000e+00, 5.00000000e-01, 0.00000000e+00]])

I try running

g = nx.stochastic_block_model(sizes_list, p_matrix, selfloops=True)

but it quickly runs out of memory in a GCP server (64Gb).

It takes a lot of RAM to build the network, even though the final graph is not so great in size. I have looked at the documentation and the scripts and I cannot figure out how to change the building function to not use so much RAM.

Is there a proper way to build such a graph?

Thanks

question from:https://stackoverflow.com/questions/65617698/networkx-stochastic-block-model-takes-too-much-memory-to-build

Answer 1

Waitting for answers