开源软件名称：

开源软件地址：

开源编程语言：

开源软件介绍：

RepoXplorer - Stats explorer for Git repositories

• Demo instance: demo.
• Last release: 1.6.1.

The documentation here is for the master version of repoXplorer. If you look for information about the last stable version, then checkout at version 1.6.1.

RepoXplorer provides a web UI and a REST API to browse git stats such as:

• projects (composed of one or multiple repositories)
• contributors
• groups of contributors

Stats for a project are such as:

• commits and authors count
• date histogram of commits
• date histogram of authors
• top authors by commits
• top authors by lines changed
• top new authors by commits

Stats for a contributor or a group are such as:

• commits, lines changed and projects count
• date histogram of commits
• date histogram of authors (only for group)
• top projects by commits
• top projects by lines changed

Filters can be used to refine stats by:

• dates boundaries
• releases or tags dates
• repositories
• metadata (grabbed from commit message eg. fix-bug: 12)

RepoXplorer is composed of:

• YAML configuration file(s)
• a Git indexer service
• a WSGI application
• an ElasticSearch backend

RepoXplorer is the right tool to continuously watch and index your repositories like for instance your Github organization.

Quickstart - Use the RepoXplorer Docker container to index a Github organization

The docker container bundles ElasticSearch + the last repoXplorer version.

This paragraph describes the usage of the container of repoXplorer 1.6.1. Some commands/options may have changes on the master version. A container for the master version of repoXplorer is published as well and can be deployed using docker-compose -f docker-compose-master.yaml up -d, please refer to docker/aoi-master/README.

Simply run docker-compose to deploy.

git clone --depth 1 https://github.com/morucci/repoxplorer
cd repoxplorer
git checkout 1.6.1
mkdir -p docker-data/conf
docker-compose up -d
firefox http://localhost:51000/index.html

Then, locally, use the github helper to create the configuration file. Here we index the Github organization of the git project.

./bin/repoxplorer-github-organization --org git --skip-fork --output-path docker-data/conf/git

Alternatively, one can execute this command within the repoxplorer docker image with

docker-compose exec repoxplorer bash
repoxplorer-github-organization --org git --skip-fork --output-path /etc/repoxplorer/defs/git

Indexation will take between 2 and 5 minutes. Indexation in progress can be seen into the indexer log file.

docker-compose exec repoxplorer bash
tail -f /root/.local/repoxplorer/repoxplorer-indexer.log

Repoxplorer will check repositories every 60 seconds for news commits and index them if any.

Stop the container. Data are stored in docker volumes so you can safely stop and restart the container. docker-compose down -v can be used to remove data.

docker-compose down

Standard installation

The installation process described here is for the master version of repoXplorer. If you look for the installation process of the last stable version, then checkout at version 1.6.1.

This process has been validated with Fedora 29 but as long as you have Python 3 and a working ElasicSearch 6.X or 7.X deployment repoXplorer should work as expected. Feel free to adapt to your Linux distribution.

ElasticSearch

RepoXplorer relies on ElasticSearch 6.X or 7.X. Below are the installation steps for ElasticSearch 6.x for a RPM based system:

sudo rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
cat << EOF | sudo tee /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
sudo yum install -y elasticsearch java-1.8.0-openjdk
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch

Using a Python virtualenv

This is method to follow especially if you intend to try the master version.

sudo yum install -y python-virtualenv libffi-devel openssl-devel python-devel git gcc
mkdir git && cd git
git clone https://github.com/morucci/repoxplorer.git
cd repoxplorer
virtualenv ~/repoxplorer
. ~/repoxplorer/bin/activate
pip install -U pip
pip install -r requirements.txt
python setup.py install
mkdir -p ~/.local/repoxplorer
cp config.py ~/.local/repoxplorer/
repoxplorer-fetch-web-assets --config ~/.local/repoxplorer/config.py
# Install a demo project definition
cp etc/projects.yaml ~/.local/repoxplorer/

Start the web UI

./bin/start-ui.sh ~/.local/repoxplorer/config.py

Then open a Web browser to access http://localhost:51000/index.html

Start the indexer

repoxplorer-indexer --config ~/.local/repoxplorer/config.py

In order to run the indexer continuously you can use the command's argument "--forever". When indexing continuously, it will sleep for 60 seconds between runs.

Quickstart helpers

Index a Github organization

RepoXplorer comes with an helper to create a yaml file for from indexing a Github organization. The yaml file can then be moved to the configuration directory of repoXplorer.

repoxplorer-github-organization --org <orgname>
mv <orgname>.yaml ~/.local/repoxplorer/

Using the --repo argument in addition to the --org argument will create the yaml file for indexing a single repository.

For private organizations or repositories, you will need to issue a Token and use the --token argument. Tokens can be issued on your Github account under "Settings > Developer Settings > Personal Access Tokens". You will need to give the repo access.

Private repositories

The git credential helper won't recognize private repositories by default. For that, you need to add, your credentials into repoxplorer-git-credentials-helper (if running in container, inside the /bin/ directory of course).

Configuration

RepoXplorer will look for the configuration directory db_path that is defined in config.py. The directory is supposed to contain the content definition.

The content definition (projects, groups, contributors) must be defined in db_path as YAML files. Any YAML files in the configuration directory will be loaded. RepoXplorer will load YAML files and do a basic data update when definition's keys appears in multiple YAML files. The loading is performed by alphabetical order.

Define projects to index

Below is an example of a yaml file, note that Barbican and Swift projects are composed of two Git repositories each, a server and a client.

---
project-templates:
  default:
    uri: https://github.com/openstack/%(name)s
    branches:
    - master
    - stable/mitaka
    - stable/newton
    - stable/ocata
    gitweb: https://github.com/openstack/%(name)s/commit/%%(sha)s

projects:
  Barbican:
    description: The Barbican project
    repos:
      barbican:
        template: default
      python-barbicanclient:
        template: default
  Swift:
    description: The Swift project
    repos:
      swift:
        template: default
      python-swiftclient:
        template: default

After a change in this file you can start the Git indexer manually or let the indexer daemon reads the file (every minute) and handles changes.

Advanced configuration

The branches key of a template definition permits to define which branches to index. This key expects a list of branches name.

A list of tags can be given to each Git repositories. This tag concept should not be understood as Git tags but only as a way to mark Git repositories. For example tags like 'documentation', 'librairies', packaging, ...) could be considered. Tags defined at repositories level will be appended to those defined at the template level.

project-templates:
  default:
    uri: https://github.com/openstack/%(name)s
    branches:
    - master
    tags:
    - openstack

projects:
  Barbican:
    repos:
      barbican:
        templates: default
        tags:
        - language:python

If the list of the repository branches differs to the one defined in the template then you can overwrite it like below.

project-templates:
  default:
    uri: https://github.com/openstack/%(name)s
    branches:
    - master

projects:
  Barbican:
    repos:
      barbican:
        templates: default
        branches:
        - devel
        - stable/1.0.x
      python-barbicanclient:
        templates: default

A list of releases can be defined. It is useful when you want to define release dates across all repositories defined in a project or for a specific repository. Release dates with %Y-%m-%d format can be defined and will be merged with detected Git tags dates.

Furthermore the index-tags flag could be set to False to tell the indexer to skip repositories tags indexation. Also the indexer will make sure to wipe already indexed tags for the project's repositories.

project-templates:
  default:
    uri: https://github.com/openstack/%(name)s
    branches:
      - master
    releases:
      - name: 2.0
        date: 2016-12-20
    index-tags: false

projects:
  Barbican:
    releases:
      - name: Ocata
        date: 2017-02-22
    repos:
      barbican:
        template: default

A list of paths can be given under the paths key. When defined for project repository then only commits including a file changed under one of the list of paths will match during statistics computation. If you want to define a special project Barbican-Tests that is limited to tests directory then:

project-templates:
  default:
    uri: https://github.com/openstack/%(name)s
    branches:
      - master

projects:
  Barbican-Tests:
    repos:
      barbican:
        template: default
        paths:
        - barbican/tests/
        - barbican/functional-tests/
      python-barbicanclient:
        templates: default
        paths:
        - barbicanclient/tests/

A group can be given to the bots-group project's attribute. When set and target an existing group then commits from this group members are excluded from the project's stats computation.

groups:
  ci-bots:
    emails:
      [email protected]: {}
      [email protected]: {}

projects:
  Barbican:
    bots-group: ci-bots
    repos:
      barbican:
        template: default

It is also possible to define metadata parsers. Please refer to the Metadata automatic indexation section.

Sanitize author identities

An unique author can use multiple emails (identities) when contributing to a project. The identities configuration permits to define emails that belong to a contributor.

In the example below, contributions from both author emails 'john.doe@server' and 'jdoe@server' will be stacked for John Doe.

---
identities:
  0000-0000:
    name: John Doe
    default-email: [email protected]
    emails:
      [email protected]:
        groups:
          barbican-ptl:
            begin-date: 2016-12-31
            end-date: 2017-12-31
      [email protected]:
        groups: {}

Group's membership can be defined via the groups key. A group must have been defined (Define groups of authors) before use. Membership bounces can be defined via begin-date and end-date to declare a group's membership between given dates (%Y-%m-%d).

When an identity declares a group's membership then that's not needed to define it again at groups level.

Define groups of authors

You may want to define groups of authors and be able to compute stats for those groups.

---
groups:
  barbican-ptl:
    description: Project team leaders of Barbican project
    emails:
      [email protected]:
      [email protected]:
        begin-date: 2015-12-31
        end-date: 2016-12-31
  barbican-core:
    description: Project team leaders of Barbican project
    emails: {}
  acme:
    description: ACME corp group
    emails: {}
    domains:
      - acme.com
      - acme.org

Group's membership is defined via an author email. Bounces can be defined via begin-date and end-date to declare a group's membership between given dates (%Y-%m-%d).

If an identity has been defined with emails part of a defined group then date bounces will overwrite those defined at the groups level.

To define a group that implicitly include commits of authors from specific domains use the domains key to list domains.

Metadata automatic indexation

In addition to the standard Git object fields, the indexer detects metadata such as:

• closes-bug: #123
• Co-Authored-By: John Doe [email protected]

The full list of matching regexps is defined in indexer/git/metadata_re.py. Do not hesitate to propose additional regexp if relevant.

Furthermore it is possible to specify custom capturing regexs to extract metadata that does not follow to the default regex.

All regexs specified in the parsers key will be executed on each commit message line. You need to have two captured elements and the first one will be used as the key, the second as the value.

project-templates:
  default:
    uri: https://github.com/openstack/%(name)s
    branches:
    - master
    gitweb: https://github.com/openstack/%(name)s/commit/%%(sha)s
    parsers:
    - .*(blueprint) ([^ .]+).*

Custom capturing regexs must be defined prior to the indexation of the Git repository it apply.

Validate the configuration

The command repoxplorer-config-validate can be used to check that yaml definition files follow the right format.

repoxplorer-config-validate --config ~/.local/repoxplorer/config.py

REST API endpoints

Endpoints

All endpoints can be called with or without the suffix .json. If called without the .json suffix then the request's header Accept: application/json must be set.

Some endpoints can return CSV data by adding the suffix .csv or without the suffix but by adding the request's header Accept: text/csv (curl -H "accept: text/csv").

See below for available parameters. Keep in mind that some parameters are mandatory, while some others are optional or only available for certain calls.

/api/v1/status/status

This endpoint permits to retrieve the platform status.

curl "http://localhost:51000/api/v1/status/status"

{
    "customtext": "",
    "projects": 2,
    "repos": 4,
    "users_endpoint": False,
    "version": "1.6.1"
}

/api/v1/projects/projects

This endpoint permits to retrieve defined projects.

curl "http://localhost:51000/api/v1/projects/projects"

{
    "projects": {
        "Barbican": {
            "description": "The barbican project",
            "repos": [
                {
                    "branch": "master",
                    "gitweb": "https://github.com/openstack/barbican/commit/%(sha)s",
                    "name": "barbican",
                    "parsers": [],
                    "releases": [],
                    "tags": [],
                    "uri": "https://github.com/openstack/barbican"
                },
                {
                    "branch": "stable/ocata",
                    "gitweb": "https://github.com/openstack/barbican/commit/%(sha)s",
                    "name": "barbican",
                    "parsers": [],
                    "releases": [],
                    "tags": [],
                    "uri": "https://github.com/openstack/barbican"
                },
                {
                    "branch": "master",
                    "gitweb": "https://github.com/openstack/python-barbicanclient/commit/%(sha)s",
                    "name": "python-barbicanclient",
                    "parsers": [],
                    "releases": [],
                    "tags": [],
                    "uri": "https://github.com/openstack/python-barbicanclient"
                },
                {
                    "branch": "stable/ocata",
                    "gitweb": "https://github.com/openstack/python-barbicanclient/commit/%(sha)s",
                    "name": "python-barbicanclient",
                    "parsers": [],
                    "releases": [],
                    "tags": [],
                    "uri": "https://github.com/openstack/python-barbicanclient"
                }
            ]
        }
    },
    "tags": [],
}

The pid parameter can be passed to request only a project.

curl "http://localhost:51000/api/v1/projects/projects?pid=Barbican"

{"Barbican": {
    "description": "The barbican project",
    "repos": [
        {
            "branch": "master",
            "gitweb": "https://github.com/openstack/barbican/commit/%(sha)s",
            "name": "barbican",
            "parsers": [],
            "releases": [],
            "tags": [],
            "uri": "https://github.com/openstack/barbican"
        },
        ...
    ]
}}

/api/v1/projects/repos

This endpoint permits to retrieve repositories for a project or tag.

curl "http://localhost:51000/api/v1/projects/repos?pid=Barbican"

[
    {
        "branch": "master",
        "gitweb": "https://github.com/openstack/barbican/commit/%(sha)s",
        "name": "barbican",
        "parsers": [],
        "releases": [],
        "tags": [],
        "uri": "https://github.com/openstack/barbican"
    },
    ...
]

/api/v1/infos/infos

This endpoint is used to fetch project, contributor, group or tag general information.

curl "http://localhost:51000/api/v1/infos/infos?cid=DwAQCBtCFg0WDg4FLAYFBg0SBQ0XAUsFDhg-"

{
    "authors_amount": 1,
    "commits_amount": 13,
    "duration": 33695365,
    "first": 1401312787,
    "last": 1435008152,
    "line_modifieds_amount": 4180,
    "ttl_average": 184525
}

This endpoint can also output to CSV.

curl "http://localhost:51000/api/v1/infos/infos.csv?cid=DwAQCBtCFg0WDg4FLAYFBg0SBQ0XAUsFDhg-"
# or
curl -H "accept: text/csv" "http://localhost:51000/api/v1/infos/infos?cid=DwAQCBtCFg0WDg4FLAYFBg0SBQ0XAUsFDhg-"

last,authors_amount,commits_amount,ttl_average,duration,line_modifieds_amount,first
1435008152,1,13,184525,33695365,4180,1401312787

/api/v1/infos/contributor

This endpoint is used to fetch contributor-specific information. Note the cid parameter is mandatory.

curl "http://localhost:51000/api/v1/infos/contributor?cid=DwAQCBtCFg0WDg4FLAYFBg0SBQ0XAUsFDhg-"

{ "name": "John Doe",
  "mails_amount": 1,
  "gravatar": "b726b19f8e7c2e23e403e4b5d3ab4508",
}

This endpoint can also output to CSV.

curl "http://localhost:51000/api/v1/infos/contributor.csv?cid=DwAQCBtCFg0WDg4FLAYFBg0SBQ0XAUsFDhg-"
# or
curl -H "accept: text/csv" "http://localhost:51000/api/v1/infos/contributor?cid=DwAQCBtCFg0WDg4FLAYFBg0SBQ0XAUsFDhg-"

gravatar,mails_amount,name
a08de2c8fe4fb685409091d3b6dbd0bf,1,John Doe

/api/v1/commits/commits

This endpoint is used to fetch commits.

Examples:

curl "http://localhost:51000/api/v1/commits/commits.json?pid=Barbican&limit=1"

[
    1,
    2214,
    [
        {
            "Change-Id": [
                "I03080db776eb4c9c2991eca8f5df43f74eb6bf24"
            ],
            "author_date": 1507229976,
            "author_email_domain": "lists.openstack.org",
            "author_gravatar": "5718d97082d0499d42ea0a291c46ec40",
            "author_name": "OpenStack Proposal Bot",
            "ccid": "CxUDDwYYFQcOSwgbCgYFJQoIBhgHSgoWBBsfAAUGDU8aHhM-",
            "cid": "CxUDDwYYFQcOSwgbCgYFJQoIBhgHSgoWBBsfAAUGDU8aHhM-",
            "commit_msg": "Updated from global requirements",
            "committer_date": 1507229976,
            "committer_gravatar": "5718d97082d0499d42ea0a291c46ec40",
            "committer_name": "OpenStack Proposal Bot",
            "files_list": [
                "requirements.txt"
            ],
            "gitwebs": [
                "https://github.com/openstack/python-barbicanclient/commit/77eedac597fb99745751a049a11d0719c4b67a85"