• 设为首页
  • 点击收藏
  • 手机版
    手机扫一扫访问
    迪恩网络手机版
  • 关注官方公众号
    微信扫一扫关注
    迪恩网络公众号

thomasnield/kotlin-statistics: Idiomatic statistical operators for Kotlin

原作者: [db:作者] 来自: 网络 收藏 邀请

开源软件名称(OpenSource Name):

thomasnield/kotlin-statistics

开源软件地址(OpenSource Url):

https://github.com/thomasnield/kotlin-statistics

开源编程语言(OpenSource Language):

Kotlin 100.0%

开源软件介绍(OpenSource Introduction):

Kotlin Statistics

NOTE: UNSUPPORTED. PLEASE FORK AND SUPPORT.

Idiomatic math and statistical extensions for Kotlin

This library contains helpful extension functions to perform exploratory and production statistics in a Kotlin-idiomatic way.

Read the introductory blog post here

Community

Join the #datscience community on Kotlin Slack for community discussion on this library as well as Kotlin for data science.

Build Instructions

You can use Gradle or Maven to pull the latest release from Maven.

Gradle

dependencies {
    compile 'org.nield:kotlin-statistics:1.2.1'
}

Maven

<dependency>
    <groupId>org.nield</groupId>
    <artifactId>kotlin-statistics</artifactId>
    <version>1.2.1</version>
</dependency>

You can also use Maven or Gradle with JitPack to directly build a snapshot as a dependency.

Gradle

repositories {		
    maven { url 'https://jitpack.io' }
}
dependencies {
    compile 'com.github.thomasnield:kotlin-statistics:-SNAPSHOT'
}

Maven

<repositories>
    <repository>
        <id>jitpack.io</id>
        <url>https://jitpack.io</url>
    </repository>
</repositories>
<dependency>
    <groupId>com.github.thomasnield</groupId>
    <artifactId>kotlin-statistics</artifactId>
    <version>-SNAPSHOT</version>
</dependency>

Basic Operators

There are a number of extension function operators that support Int, Long, Double, Float, BigDecimal and Short numeric types for Sequences, Arrays, and Iterables:

  • descriptiveStatistics
  • sum()
  • average()
  • min()
  • max()
  • mode()
  • median()
  • range()
  • percentile()
  • variance()
  • standardDeviation()
  • geometricMean()
  • sumOfSquares()
  • normalize()
  • simpleRegression()
  • kurtosis
  • skewness

Here is an example of using the median() extension function against a Sequence of Doubles:

val median = sequenceOf(1.0, 3.0, 5.0).median()
println(median) // prints "3.0"

Slicing Operators

There are also simple but powerful xxxBy() operators that allow you slice many of these statistical operators on a given key:

  • countBy()
  • sumBy()
  • averageBy()
  • geometricMeanBy()
  • minBy()
  • maxBy()
  • rangeBy()
  • varianceBy()
  • standardDeviationBy()
  • descriptiveStatisticsBy()
  • simpleRegressionBy()

Below, we slice a sequence of Item objects by their lengths and get the averages and standard deviations by each length.

class Item(val name: String, val value: Double)

val sequence = sequenceOf(
        Item("Alpha", 4.0),
        Item("Beta", 6.0),
        Item("Gamma", 7.2),
        Item("Delta", 9.2),
        Item("Epsilon", 6.8),
        Item("Zeta", 2.4),
        Item("Iota", 8.8)
)

// find sums by name length, using pairs or functional arguments
val sumsByLengths = sequence
       .map { it.name.length to it.value }
       .sumBy()

val sumsByLengths = sequence
       .sumBy(keySelector = { it.name.length }, doubleSelector = {it.value} )

println("Sums by lengths: $sumsByLengths")

// find averages by name length, using pairs or functional arguments

val averagesByLength = sequence
        .map { it.name.length to it.value }
        .averageBy()

val averagesByLength = sequence
        .averageBy(keySelector = { it.name.length }, doubleSelector = {it.value})


//find standard deviations by name length, using pairs or functional arguments

val standardDeviationsByLength = sequence
        .map { it.name.length to it.value }
        .standardDeviationBy()

val standardDeviationsByLength = sequence
        .standardDeviationBy(keySelector = { it.name.length }, valueSelector = {it.value})

println("Std Devs by lengths: $standardDeviationsByLength")

OUTPUT:

Sums by lengths: {5=20.4, 4=17.200000000000003, 7=6.8}
Averages by lengths: {5=6.8, 4=5.733333333333334, 7=6.8}
Std Devs by lengths: {5=2.1416504538945342, 4=2.619584360585134, 7=0.0}

These slicing operators are backed by a common groupApply() function, which can be used to implement other slicing operators easily.

Slicing Using Data Classes

You can slice on multiple fields using data classes with the xxxBy() operators as well. This is similar to using a GROUP BY on multiple fields in SQL. Below we slice a count and average defect of products by their category and section.

//declare Product class
class Product(val id: Int,
              val name: String,
              val category: String,
              val section: Int,
              val defectRate: Double)

// Create list of Products
val products = listOf(Product(1, "Rayzeon", "ABR", 3, 1.1),
        Product(2, "ZenFire", "ABZ", 4, 0.7),
        Product(3, "HydroFlux", "ABR", 3, 1.9),
        Product(4, "IceFlyer", "ZBN", 1, 2.4),
        Product(5, "FireCoyote", "ABZ", 4, 3.2),
        Product(6, "LightFiber", "ABZ",2,  5.1),
        Product(7, "PyroKit", "ABR", 3, 1.4),
        Product(8, "BladeKit", "ZBN", 1, 0.5),
        Product(9, "NightHawk", "ZBN", 1, 3.5),
        Product(10, "NoctoSquirrel", "ABR", 2, 1.1),
        Product(11, "WolverinePack", "ABR", 3, 1.2)
        )

// Data Class for Grouping
data class Key(val category: String, val section: Int)

// Get Count by Category and Section
val countByCategoryAndSection =
        products.countBy { Key(it.category, it.section) }

println("Counts by Category and Section")
countByCategoryAndSection.entries.forEach { println(it) }

// Get Average Defect Rate by Category and Section
val averageDefectByCategoryAndSection =
        products.averageBy(keySelector = { Key(it.category, it.section) }, doubleSelector = { it.defectRate })

println("\nAverage Defect Rate by Category and Section")
averageDefectByCategoryAndSection.entries.forEach { println(it) }

OUTPUT:

Counts by Category and Section
Key(category=ABR, section=3)=4
Key(category=ABZ, section=4)=2
Key(category=ZBN, section=1)=3
Key(category=ABZ, section=2)=1
Key(category=ABR, section=2)=1

Average Defect Rate by Category and Section
Key(category=ABR, section=3)=1.4000000000000001
Key(category=ABZ, section=4)=1.9500000000000002
Key(category=ZBN, section=1)=2.1333333333333333
Key(category=ABZ, section=2)=5.1
Key(category=ABR, section=2)=1.1

Slicing by Ranges/Bins

You can also group by ranges (or known in statistics as "bins" or a "histogram").

Please note that in Kotlin-Statistics 1.2, the gapSize parameter was removed from all binBy() extension functions. A Range interface was implemented to accomodate a ClosedOpenRange needed for binning operations. Hopefully Kotlin will officially support different open range implementations and Kotlin-Statistics can migrate to them.

Slicing By Numbers

There are specialized bin operators that deal with numeric ranges for Int, Long, Double, Float, and BigDecimal. Below, we bin the sales items by increments of 20.0 for the value.

import java.time.LocalDate

fun main(args: Array<String>) {

    data class Sale(val accountId: Int, val date: LocalDate, val value: Double)

    val sales = listOf(
            Sale(1, LocalDate.of(2016,12,3), 180.0),
            Sale(2, LocalDate.of(2016, 7, 4), 140.2),
            Sale(3, LocalDate.of(2016, 6, 3), 111.4),
            Sale(4, LocalDate.of(2016, 1, 5), 192.7),
            Sale(5, LocalDate.of(2016, 5, 4), 137.9),
            Sale(6, LocalDate.of(2016, 3, 6), 125.6),
            Sale(7, LocalDate.of(2016, 12,4), 164.3),
            Sale(8, LocalDate.of(2016, 7,11), 144.2)
            )

    //bin by double ranges
    val binned = sales.binByDouble(
            valueSelector = { it.value },
            binSize = 20.0,
            rangeStart = 100.0
    )

    binned.forEach(::println)
}

OUTPUT:

Bin(range=[100.0..120.0), value=[Sale(accountId=3, date=2016-06-03, value=111.4)])
Bin(range=[120.0..140.0), value=[Sale(accountId=5, date=2016-05-04, value=137.9), Sale(accountId=6, date=2016-03-06, value=125.6)])
Bin(range=[140.0..160.0), value=[Sale(accountId=2, date=2016-07-04, value=140.2), Sale(accountId=8, date=2016-07-11, value=144.2)])
Bin(range=[160.0..180.0), value=[Sale(accountId=7, date=2016-12-04, value=164.3)])
Bin(range=[180.0..200.0), value=[Sale(accountId=1, date=2016-12-03, value=180.0), Sale(accountId=4, date=2016-01-05, value=192.7)])

Slicing by Comparables

You can group any T items into bins composed of Comparable ranges. Below, we group up items by yearly quarters by mapping each item to a Month, and then setting the binSize to 3. We also have to provide an incrementer so the model knows how to build the bins incrementally.

import java.time.LocalDate

fun main(args: Array<String>) {

    data class Sale(val accountId: Int, val date: LocalDate, val value: Double)

    val sales = listOf(
            Sale(1, LocalDate.of(2016,12,3), 180.0),
            Sale(2, LocalDate.of(2016, 7, 4), 140.2),
            Sale(3, LocalDate.of(2016, 6, 3), 111.4),
            Sale(4, LocalDate.of(2016, 1, 5), 192.7),
            Sale(5, LocalDate.of(2016, 5, 4), 137.9),
            Sale(6, LocalDate.of(2016, 3, 6), 125.6),
            Sale(7, LocalDate.of(2016, 12,4), 164.3),
            Sale(8, LocalDate.of(2016, 7,11), 144.2)
            )

    //bin by quarter
    val byQuarter = sales.binByComparable(
            valueSelector = { it.date.month },
            binIncrements = 3,
            incrementer = { it.plus(1L) }
    )

    byQuarter.forEach(::println)
}

OUTPUT:

Bin(range=JANUARY..MARCH, value=[Sale(accountId=4, date=2016-01-05, value=192.7), Sale(accountId=6, date=2016-03-06, value=125.6)])
Bin(range=APRIL..JUNE, value=[Sale(accountId=3, date=2016-06-03, value=111.4), Sale(accountId=5, date=2016-05-04, value=137.9)])
Bin(range=JULY..SEPTEMBER, value=[Sale(accountId=2, date=2016-07-04, value=140.2), Sale(accountId=8, date=2016-07-11, value=144.2)])
Bin(range=OCTOBER..DECEMBER, value=[Sale(accountId=1, date=2016-12-03, value=180.0), Sale(accountId=7, date=2016-12-04, value=164.3)])

Custom Binning Operations

If you want to perform a mathematical aggregation on a certain property for each item (rather than group up the items into a List for a given bin), provide a groupOp argument specifying how to calculate a value on each grouping. Below, we find the sum of values by quarter.

import java.time.LocalDate

fun main(args: Array<String>) {

    data class Sale(val accountId: Int, val date: LocalDate, val value: Double)

    val sales = listOf(
            Sale(1, LocalDate.of(2016,12,3), 180.0),
            Sale(2, LocalDate.of(2016, 7, 4), 140.2),
            Sale(3, LocalDate.of(2016, 6, 3), 111.4),
            Sale(4, LocalDate.of(2016, 1, 5), 192.7),
            Sale(5, LocalDate.of(2016, 5, 4), 137.9),
            Sale(6, LocalDate.of(2016, 3, 6), 125.6),
            Sale(7, LocalDate.of(2016, 12,4), 164.3),
            Sale(8, LocalDate.of(2016, 7,11), 144.2)
    )

    //bin sums by quarter
    val totalValueByQuarter = sales.binByComparable(
            valueSelector = { it.date.month },
            binIncrements = 3,
            incrementer = { it.plus(1L) },
            groupOp = { it.map(Sale::value).sum() }
    )

    totalValueByQuarter.forEach(::println)
}

OUTPUT:

Bin(range=JANUARY..MARCH, value=318.29999999999995)
Bin(range=APRIL..JUNE, value=249.3)
Bin(range=JULY..SEPTEMBER, value=284.4)
Bin(range=OCTOBER..DECEMBER, value=344.3)

Random Selection

Kotlin-Statistics has a few helpful extensions to randomly sample elements from an Iterable<T> or Sequence<T>.

  • randomFirst() - Selects one random element but throws an error if no elements are found.
  • randomFirstOrNull() - Selects one random element but returns null if no elements are found.
  • random(n: Int) - Selects n random elements.
  • randomDistinct(n: Int) - Select n distinct random elements.

Weighted Coin/Dice - Discrete PDF Sampling

Rather than do a pure random sampling, there may be times you want different values of type T to be assigned different probabilities, and then you want to sample a T randomly given those probabilities. This can be helpful for creating simulations or stochastic algorithms in general.

The WeightedCoin and WeightedDice assist in these purposes.

A WeightedCoin accepts a trueProbability from 0.0 to 1.0. If we provide a probability of .80, the coin will flip approximately 80% of the time to be true.

val riggedCoin = WeightedCoin(trueProbability = .80)

// flip coin 100000 times and print outcome counts
(1..100000).asSequence().map { riggedCoin.flip() }
        .countBy()
        .also {
            println(it)
        }

OUTPUT:

{false=20033, true=79967}

You can use the WeightedDice to manage outcomes mapped to any type T. For instance, if we have a dice with sides "A", "B", and "C" with probability outcomes .11, .66, and .22, we can create a WeightedDice effectively like this:

    val threeSidedDice = WeightedDice(
            "A" to .11,
            "B" to .66,
            "C" to .22
    )

    // roll dice 1000 times and print outcome counts
    (1..1000).asSequence().map { threeSidedDice.roll() }
            .countBy()
            .also {
                println(it)
            }

OUTPUT:

{B=682, C=202, A=116}

Typically with WeightedDice, you may likely use an enumerable to assign outcome probabilites to discrete items:

enum class Move {
    ATTACK,
    DEFEND,
    HEAL,
    RETREAT
}

fun main(args: Array<String>) {

    val gameDice = WeightedDice(
            Move.ATTACK to .60,
            Move.DEFEND to .20,
            Move.HEAL to .10,
            Move.RETREAT to .10
    )

    val nextMove = gameDice.roll()

    println(nextMove)
}

Naive Bayes Classifier

The NaiveBayesClassifier does a simple but powerful form of machine learning. For a given set of T items, you can extract one or more F features and associate a category C.

You can then test a new set of features F and predict a category C.

For instance, say you want to identify email as spam/not spam based on the words in the messages. In this case true (spam) or false (not spam) will be the possible categories, and each word will be a feature.

In idiomatic Kotlin fashion we can take a simple List<Email> and call toNaiveBayesClassifier(), provide the higher-order functions to extract the features and category, and then generate a model.

class Email(val message: String, val isSpam: Boolean)

val emails = listOf(
        Email("Hey there! I thought you might find this interesting. Click here.", isSpam = true),
        Email("Get viagra for a discount as much as 90%", isSpam = true),
        Email("Viagra prescription for less", isSpam = true),
        Email("Even better than Viagra, try this new prescription drug", isSpam = true),

        Email("Hey, I left my phone at home. Email me if you need anything. I'll be in a meeting for the afternoon.", isSpam = false),
        Email("Please see attachment for notes on today's meeting. Interesting findings on your market research.", isSpam = false),
        Email("An item on your Amazon wish list received a discount", isSpam = false),
        Email("Your prescription drug order is ready", isSpam = false),
        Email("Your Amazon account password has been reset", isSpam = false),
        Email("Your Amazon order", isSpam = false)
)

val nbc = emails.toNaiveBayesClassifier(
        featuresSelector = { it.message.splitWords().toSet() },
        categorySelector = {it.isSpam }
)



fun String.splitWords() =  split(Regex("\\s")).asSequence()
         .map { it.replace(Regex("[^A-Za-z]"),"").toLowerCase() }
         .filter { it.isNotEmpty() }

We can then use this NaiveBayesClassifier model to predict the spamminess of new emails.

 // TEST 1
val input = "discount viagra wholesale, hurry while this offer lasts".splitWords().toSet()
val predictedCategory = nbc.predict(input)
Assert.assertTrue(predictedCategory == true)

// TEST 2
val input2 = "interesting meeting on amazon cloud services discount program".splitWords().toSet()
val predictedCategory2 = nbc.predict(input2)
Assert.assertTrue(predictedCategory2 == false)

Here is another example that categorizes bank transactions.


鲜花

握手

雷人

路过

鸡蛋
该文章已有0人参与评论

请发表评论

全部评论

专题导读
上一篇:
akshayravikumar/TeXnique: A LaTeX Typesetting Game发布时间:2022-07-09
下一篇:
latex3/hyperref: Hypertext support for LaTeX发布时间:2022-07-09
热门推荐
阅读排行榜

扫描微信二维码

查看手机版网站

随时了解更新最新资讯

139-2527-9053

在线客服(服务时间 9:00~18:00)

在线QQ客服
地址:深圳市南山区西丽大学城创智工业园
电邮:jeky_zhao#qq.com
移动电话:139-2527-9053

Powered by 互联科技 X3.4© 2001-2213 极客世界.|Sitemap