So I have the data stored in Mongodb in the following structure.
[{
"id": 1,
"hour": 1,
"users": [{
"userid": 1,
"data": [{
"channelID": 1,
"count": 15,
"metrics": [{
"metricID": 1,
"count": 10
},
{
"metricID": 2,
"count": 5
}
]
},
{
"channelID": 2,
"count": 130,
"metrics": [{
"metricID": 1,
"count": 30
},
{
"metricID": 2,
"count": 100
}
]
}
]
},
{
"userid": 2,
"data": [{
"channelID": 1,
"count": 15,
"metrics": [{
"metricID": 1,
"count": 10
},
{
"metricID": 2,
"count": 5
}
]
},
{
"channelID": 2,
"count": 140,
"metrics": [{
"metricID": 1,
"count": 30
},
{
"metricID": 2,
"count": 110
}
]
}
]
}
]
},
{
"id": 2,
"hour": 2,
"users": [{
"userid": 1,
"data": [{
"channelID": 1,
"count": 11,
"metrics": [{
"metricID": 1,
"count": 1
},
{
"metricID": 2,
"count": 10
}
]
},
{
"channelID": 3,
"count": 107,
"metrics": [{
"metricID": 1,
"count": 99
},
{
"metricID": 2,
"count": 8
}
]
}
]
},
{
"userid": 2,
"data": [{
"channelID": 1,
"count": 40,
"metrics": [{
"metricID": 2,
"count": 30
},
{
"metricID": 4,
"count": 10
}
]
},
{
"channelID": 2,
"count": 130,
"metrics": [{
"metricID": 1,
"count": 30
},
{
"metricID": 2,
"count": 100
}
]
}
]
}
]
}
]
I need the following output from mongodb.
// output of sum of both hours data. Merging the data.
{
"id": 1,
"hour": [1, 2], //not important
"users": [{
"userid": 1,
"data": [{
"channelID": 1,
"count": 26,
"metrics": [{
"metricID": 1, // suming up same metrics count if same id against same channel
"count": 11
},
{
"metricID": 2,
"count": 15
}
]
},
{
"channelID": 2, // adding channel from hour 1 since it was not in hour 2
"count": 130,
"metrics": [{
"metricID": 1,
"count": 30
},
{
"metricID": 2,
"count": 100
}
]
},
{
"channelID": 3, // adding channel from hour 2 since it was not in hour 1
"count": 107,
"metrics": [{
"metricID": 1,
"count": 99
},
{
"metricID": 2,
"count": 8
}
]
}
]
},
{
"userid": 2,
"data": [{
"channelID": 1,
"count": 55,
"metrics": [{
"metricID": 1,
"count": 10
},
{
"metricID": 2,
"count": 35
},
{
"metricID": 4,
"count": 10
}
]
},
{
"channelID": 2,
"count": 300,
"metrics": [{
"metricID": 1,
"count": 90
},
{
"metricID": 2,
"count": 210
}
]
}
]
}
]
}
Idk if it can be done using MapReduce feature of mongodb. I'm trying to handle this output on query side rather writing some script in JS to do MapR job after retrieving data(Making response time slow). This is a Big data problem. My records of users are like in thousands and my metrics and channels are in hundreds.
I need ideas on how I should handle this kind of big data structure that would give me response in miliseconds. Any fast service idea would be really be helpful. How should I handle Big data structure with milisecond response.
I am fairly new at big data and mongodb
