在线时间:8:00-16:00
迪恩网络APP
随时随地掌握行业动态
扫描二维码
关注迪恩网络微信公众号
开源软件名称:morucci/repoxplorer开源软件地址:https://github.com/morucci/repoxplorer开源编程语言:Python 72.4%开源软件介绍:RepoXplorer - Stats explorer for Git repositoriesThe documentation here is for the master version of repoXplorer. If you look for information about the last stable version, then checkout at version 1.6.1. RepoXplorer provides a web UI and a REST API to browse git stats such as:
Stats for a project are such as:
Stats for a contributor or a group are such as:
Filters can be used to refine stats by:
RepoXplorer is composed of:
RepoXplorer is the right tool to continuously watch and index your repositories like for instance your Github organization. Quickstart - Use the RepoXplorer Docker container to index a Github organizationThe docker container bundles ElasticSearch + the last repoXplorer version. This paragraph describes the usage of the container of repoXplorer 1.6.1. Some
commands/options may have changes on the master version. A container for the
master version of repoXplorer is published as well and can be deployed using
Simply run docker-compose to deploy. git clone --depth 1 https://github.com/morucci/repoxplorer
cd repoxplorer
git checkout 1.6.1
mkdir -p docker-data/conf
docker-compose up -d
firefox http://localhost:51000/index.html Then, locally, use the github helper to create the configuration file. Here we index the Github organization of the git project. ./bin/repoxplorer-github-organization --org git --skip-fork --output-path docker-data/conf/git Alternatively, one can execute this command within the repoxplorer docker image with docker-compose exec repoxplorer bash
repoxplorer-github-organization --org git --skip-fork --output-path /etc/repoxplorer/defs/git Indexation will take between 2 and 5 minutes. Indexation in progress can be seen into the indexer log file. docker-compose exec repoxplorer bash
tail -f /root/.local/repoxplorer/repoxplorer-indexer.log Repoxplorer will check repositories every 60 seconds for news commits and index them if any. Stop the container. Data are stored in docker volumes so you can safely stop and restart the container. docker-compose down -v can be used to remove data. docker-compose down Standard installationThe installation process described here is for the master version of repoXplorer. If you look for the installation process of the last stable version, then checkout at version 1.6.1. This process has been validated with Fedora 29 but as long as you have Python 3 and a working ElasicSearch 6.X or 7.X deployment repoXplorer should work as expected. Feel free to adapt to your Linux distribution. ElasticSearchRepoXplorer relies on ElasticSearch 6.X or 7.X. Below are the installation steps for ElasticSearch 6.x for a RPM based system: sudo rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
cat << EOF | sudo tee /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
sudo yum install -y elasticsearch java-1.8.0-openjdk
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch Using a Python virtualenvThis is method to follow especially if you intend to try the master version. sudo yum install -y python-virtualenv libffi-devel openssl-devel python-devel git gcc
mkdir git && cd git
git clone https://github.com/morucci/repoxplorer.git
cd repoxplorer
virtualenv ~/repoxplorer
. ~/repoxplorer/bin/activate
pip install -U pip
pip install -r requirements.txt
python setup.py install
mkdir -p ~/.local/repoxplorer
cp config.py ~/.local/repoxplorer/
repoxplorer-fetch-web-assets --config ~/.local/repoxplorer/config.py
# Install a demo project definition
cp etc/projects.yaml ~/.local/repoxplorer/ Start the web UI./bin/start-ui.sh ~/.local/repoxplorer/config.py Then open a Web browser to access http://localhost:51000/index.html Start the indexerrepoxplorer-indexer --config ~/.local/repoxplorer/config.py In order to run the indexer continuously you can use the command's argument "--forever". When indexing continuously, it will sleep for 60 seconds between runs. Quickstart helpersIndex a Github organizationRepoXplorer comes with an helper to create a yaml file for from indexing a Github organization. The yaml file can then be moved to the configuration directory of repoXplorer.
Using the --repo argument in addition to the --org argument will create the yaml file for indexing a single repository. For private organizations or repositories, you will need to issue a Token and use the --token argument. Tokens can be issued on your Github account under "Settings > Developer Settings > Personal Access Tokens". You will need to give the repo access. Private repositoriesThe git credential helper won't recognize private repositories by default.
For that, you need to add, your credentials into ConfigurationRepoXplorer will look for the configuration directory db_path that is defined in config.py. The directory is supposed to contain the content definition. The content definition (projects, groups, contributors) must be defined in db_path as YAML files. Any YAML files in the configuration directory will be loaded. RepoXplorer will load YAML files and do a basic data update when definition's keys appears in multiple YAML files. The loading is performed by alphabetical order. Define projects to indexBelow is an example of a yaml file, note that Barbican and Swift projects are composed of two Git repositories each, a server and a client. ---
project-templates:
default:
uri: https://github.com/openstack/%(name)s
branches:
- master
- stable/mitaka
- stable/newton
- stable/ocata
gitweb: https://github.com/openstack/%(name)s/commit/%%(sha)s
projects:
Barbican:
description: The Barbican project
repos:
barbican:
template: default
python-barbicanclient:
template: default
Swift:
description: The Swift project
repos:
swift:
template: default
python-swiftclient:
template: default After a change in this file you can start the Git indexer manually or let the indexer daemon reads the file (every minute) and handles changes. Advanced configurationThe branches key of a template definition permits to define which branches to index. This key expects a list of branches name. A list of tags can be given to each Git repositories. This tag concept should not be understood as Git tags but only as a way to mark Git repositories. For example tags like 'documentation', 'librairies', packaging, ...) could be considered. Tags defined at repositories level will be appended to those defined at the template level. project-templates:
default:
uri: https://github.com/openstack/%(name)s
branches:
- master
tags:
- openstack
projects:
Barbican:
repos:
barbican:
templates: default
tags:
- language:python If the list of the repository branches differs to the one defined in the template then you can overwrite it like below. project-templates:
default:
uri: https://github.com/openstack/%(name)s
branches:
- master
projects:
Barbican:
repos:
barbican:
templates: default
branches:
- devel
- stable/1.0.x
python-barbicanclient:
templates: default A list of releases can be defined. It is useful when you want to define release dates across all repositories defined in a project or for a specific repository. Release dates with %Y-%m-%d format can be defined and will be merged with detected Git tags dates. Furthermore the index-tags flag could be set to False to tell the indexer to skip repositories tags indexation. Also the indexer will make sure to wipe already indexed tags for the project's repositories. project-templates:
default:
uri: https://github.com/openstack/%(name)s
branches:
- master
releases:
- name: 2.0
date: 2016-12-20
index-tags: false
projects:
Barbican:
releases:
- name: Ocata
date: 2017-02-22
repos:
barbican:
template: default A list of paths can be given under the paths key. When defined for project repository then only commits including a file changed under one of the list of paths will match during statistics computation. If you want to define a special project Barbican-Tests that is limited to tests directory then: project-templates:
default:
uri: https://github.com/openstack/%(name)s
branches:
- master
projects:
Barbican-Tests:
repos:
barbican:
template: default
paths:
- barbican/tests/
- barbican/functional-tests/
python-barbicanclient:
templates: default
paths:
- barbicanclient/tests/ A group can be given to the bots-group project's attribute. When set and target an existing group then commits from this group members are excluded from the project's stats computation. groups:
ci-bots:
emails:
[email protected]: {}
[email protected]: {}
projects:
Barbican:
bots-group: ci-bots
repos:
barbican:
template: default It is also possible to define metadata parsers. Please refer to the Metadata automatic indexation section. Sanitize author identitiesAn unique author can use multiple emails (identities) when contributing to a project. The identities configuration permits to define emails that belong to a contributor. In the example below, contributions from both author emails 'john.doe@server' and 'jdoe@server' will be stacked for John Doe. ---
identities:
0000-0000:
name: John Doe
default-email: [email protected]
emails:
[email protected]:
groups:
barbican-ptl:
begin-date: 2016-12-31
end-date: 2017-12-31
[email protected]:
groups: {} Group's membership can be defined via the groups key. A group must have been defined (Define groups of authors) before use. Membership bounces can be defined via begin-date and end-date to declare a group's membership between given dates (%Y-%m-%d). When an identity declares a group's membership then that's not needed to define it again at groups level. Define groups of authorsYou may want to define groups of authors and be able to compute stats for those groups. ---
groups:
barbican-ptl:
description: Project team leaders of Barbican project
emails:
[email protected]:
[email protected]:
begin-date: 2015-12-31
end-date: 2016-12-31
barbican-core:
description: Project team leaders of Barbican project
emails: {}
acme:
description: ACME corp group
emails: {}
domains:
- acme.com
- acme.org Group's membership is defined via an author email. Bounces can be defined via begin-date and end-date to declare a group's membership between given dates (%Y-%m-%d). If an identity has been defined with emails part of a defined group then date bounces will overwrite those defined at the groups level. To define a group that implicitly include commits of authors from specific domains use the domains key to list domains. Metadata automatic indexationIn addition to the standard Git object fields, the indexer detects metadata such as:
The full list of matching regexps is defined in indexer/git/metadata_re.py. Do not hesitate to propose additional regexp if relevant. Furthermore it is possible to specify custom capturing regexs to extract metadata that does not follow to the default regex. All regexs specified in the parsers key will be executed on each commit message line. You need to have two captured elements and the first one will be used as the key, the second as the value. project-templates:
default:
uri: https://github.com/openstack/%(name)s
branches:
- master
gitweb: https://github.com/openstack/%(name)s/commit/%%(sha)s
parsers:
- .*(blueprint) ([^ .]+).* Custom capturing regexs must be defined prior to the indexation of the Git repository it apply. Validate the configurationThe command repoxplorer-config-validate can be used to check that yaml definition files follow the right format. repoxplorer-config-validate --config ~/.local/repoxplorer/config.py REST API endpointsEndpointsAll endpoints can be called with or without the suffix .json. If called without the .json suffix then the request's header Accept: application/json must be set. Some endpoints can return CSV data by adding the suffix .csv or without the suffix but by adding the request's header Accept: text/csv (curl -H "accept: text/csv"). See below for available parameters. Keep in mind that some parameters are mandatory, while some others are optional or only available for certain calls. /api/v1/status/statusThis endpoint permits to retrieve the platform status. curl "http://localhost:51000/api/v1/status/status" {
"customtext": "",
"projects": 2,
"repos": 4,
"users_endpoint": False,
"version": "1.6.1"
} /api/v1/projects/projectsThis endpoint permits to retrieve defined projects. curl "http://localhost:51000/api/v1/projects/projects" {
"projects": {
"Barbican": {
"description": "The barbican project",
"repos": [
{
"branch": "master",
"gitweb": "https://github.com/openstack/barbican/commit/%(sha)s",
"name": "barbican",
"parsers": [],
"releases": [],
"tags": [],
"uri": "https://github.com/openstack/barbican"
},
{
"branch": "stable/ocata",
"gitweb": "https://github.com/openstack/barbican/commit/%(sha)s",
"name": "barbican",
"parsers": [],
"releases": [],
"tags": [],
"uri": "https://github.com/openstack/barbican"
},
{
"branch": "master",
"gitweb": "https://github.com/openstack/python-barbicanclient/commit/%(sha)s",
"name": "python-barbicanclient",
"parsers": [],
"releases": [],
"tags": [],
"uri": "https://github.com/openstack/python-barbicanclient"
},
{
"branch": "stable/ocata",
"gitweb": "https://github.com/openstack/python-barbicanclient/commit/%(sha)s",
"name": "python-barbicanclient",
"parsers": [],
"releases": [],
"tags": [],
"uri": "https://github.com/openstack/python-barbicanclient"
}
]
}
},
"tags": [],
} The pid parameter can be passed to request only a project. curl "http://localhost:51000/api/v1/projects/projects?pid=Barbican"
/api/v1/projects/reposThis endpoint permits to retrieve repositories for a project or tag. curl "http://localhost:51000/api/v1/projects/repos?pid=Barbican" [
{
"branch": "master",
"gitweb": "https://github.com/openstack/barbican/commit/%(sha)s",
"name": "barbican",
"parsers": [],
"releases": [],
"tags": [],
"uri": "https://github.com/openstack/barbican"
},
...
] /api/v1/infos/infosThis endpoint is used to fetch project, contributor, group or tag general information. curl "http://localhost:51000/api/v1/infos/infos?cid=DwAQCBtCFg0WDg4FLAYFBg0SBQ0XAUsFDhg-" {
"authors_amount": 1,
"commits_amount": 13,
"duration": 33695365,
"first": 1401312787,
"last": 1435008152,
"line_modifieds_amount": 4180,
"ttl_average": 184525
} This endpoint can also output to CSV. curl "http://localhost:51000/api/v1/infos/infos.csv?cid=DwAQCBtCFg0WDg4FLAYFBg0SBQ0XAUsFDhg-"
# or
curl -H "accept: text/csv" "http://localhost:51000/api/v1/infos/infos?cid=DwAQCBtCFg0WDg4FLAYFBg0SBQ0XAUsFDhg-"
/api/v1/infos/contributorThis endpoint is used to fetch contributor-specific information. Note the cid parameter is mandatory. curl "http://localhost:51000/api/v1/infos/contributor?cid=DwAQCBtCFg0WDg4FLAYFBg0SBQ0XAUsFDhg-" { "name": "John Doe",
"mails_amount": 1,
"gravatar": "b726b19f8e7c2e23e403e4b5d3ab4508",
} This endpoint can also output to CSV. curl "http://localhost:51000/api/v1/infos/contributor.csv?cid=DwAQCBtCFg0WDg4FLAYFBg0SBQ0XAUsFDhg-"
# or
curl -H "accept: text/csv" "http://localhost:51000/api/v1/infos/contributor?cid=DwAQCBtCFg0WDg4FLAYFBg0SBQ0XAUsFDhg-"
/api/v1/commits/commitsThis endpoint is used to fetch commits. Examples: curl "http://localhost:51000/api/v1/commits/commits.json?pid=Barbican&limit=1" [
1,
2214,
[
{
"Change-Id": [
"I03080db776eb4c9c2991eca8f5df43f74eb6bf24"
],
"author_date": 1507229976,
"author_email_domain": "lists.openstack.org",
"author_gravatar": "5718d97082d0499d42ea0a291c46ec40",
"author_name": "OpenStack Proposal Bot",
"ccid": "CxUDDwYYFQcOSwgbCgYFJQoIBhgHSgoWBBsfAAUGDU8aHhM-",
"cid": "CxUDDwYYFQcOSwgbCgYFJQoIBhgHSgoWBBsfAAUGDU8aHhM-",
"commit_msg": "Updated from global requirements",
"committer_date": 1507229976,
"committer_gravatar": "5718d97082d0499d42ea0a291c46ec40",
"committer_name": "OpenStack Proposal Bot",
"files_list": [
"requirements.txt"
],
"gitwebs": [
"https://github.com/openstack/python-barbicanclient/commit/77eedac597fb99745751a049a11d0719c4b67a85"
|
2023-10-27
2022-08-15
2022-08-17
2022-09-23
2022-08-13
请发表评论