• 设为首页
  • 点击收藏
  • 手机版
    手机扫一扫访问
    迪恩网络手机版
  • 关注官方公众号
    微信扫一扫关注
    迪恩网络公众号

hashquine/hicetnunc-dataset: Getting hicetnunc.xyz data from blockchain, IPFS an ...

原作者: [db:作者] 来自: 网络 收藏 邀请

开源软件名称:

hashquine/hicetnunc-dataset

开源软件地址:

https://github.com/hashquine/hicetnunc-dataset

开源编程语言:

Python 60.5%

开源软件介绍:

hicetnunc.xyz open dataset and parsers

The project development is suspended, parsers will fail to launch on current data

Hic et nunc is a new eco-friendly NFT marketplace, built on top of Tezos blockchain.

It is especially popular in generative graphics and data viz community, so I've decided to share data and all scripts that I've made for https://hashquine.github.io/hicetnunc rating.

It is published under CC BY license, so that it is even possible to sell NFTs that use that data (or modified scripts) as long as there is the following phrase somewhere in the token description: based on @hashquine dataset.

Since hicetnunc servers are already under an extreme load due to quick growth, I've reorganized code, so that all data is taken from Tezos blockchain and IPFS without any calls to the hicetnunc.xyz website or API.

Data sources

What data is available

  • Money data: list of all purchases, prices and commissions.
  • All NFTs raw files, their previews and thumbnails, although 3d files and interactive SVG/HTML files are not yet processed properly.
  • Authors metadata verified via tzkt.io like Twitter account address.
  • Token transfers: list of changes of tokens owners including burns and direct transfers.
  • All metadata available for tokens.
  • Swaps and mints.

Data not available:

  • Everything connected with hDAO tokens and hDAO feed. Although all related transactions are already being collected, they are not analysed yet.
  • Twitter statistics like the number of followers.
  • Direct money transfers between users, when NFT tokens are not transferred in the same transaction.

Dataset schema

The goal was to simplify data analysis and visualization with a wide range of existing tools, so there are lots of redundant fields, which contain precalculated aggregations and different representations of the same data.

All files have two equivalent versions: JSON and CSV.

  • JSON files are dictionary of dictionaries with rows of CSV files are indexed by the *_id field.
  • CSV files have commas as delimiters.
  • Fields values are ether numbers or strings, empty values represented by -1 or "".
  • All identifiers are strings.

Any field, which references some event in the blockchain (for example, mint time) have 4 representations:

  • mint_iso_date — string with UTC date and time: "2021-03-01T15:00:00Z",
  • mint_stamp — integer Unix timestamp in seconds: 1614610800,
  • mint_hash — string with transaction hash, where event occurred: "oom5Ju6X9nYpBCi...",
  • mint_row_id — integer with global unique operation id (internal to TzStats) with that event: 42181049

Any field, which references a set of values (like the set of prices of sold works), have following aggregations:

  • sold_count — values count,
  • sold_nonzero_count — number of positive values,
  • sold_zero_count — number of zeros,
  • sold_price_min — minimum value (excl. zeros),
  • sold_price_max — maximum value,
  • sold_price_sum — sum of values,
  • sold_price_avg — average value (sum divided by count excl. zeros).

tokens.json and tokens.csv — of all NFTs tokens

There is a confusing fact, that in hicetnunc each NFT can have multiple identical instances, which are fungible. In this document term "token" refers to the set of all that instances.

There are following invariants:

mint_count = author_owns_count + available_count + available_zero_count + other_own_count + burn_count
author_sent_count <= other_own_count + available_count + available_zero_count
field type example description
token_id string "152" Token identifier: numeric strings with OBJKT ids from hicetnunc.xyz/objkt url.
issuer string "tz1UBZUk..." Address of the person, who intiated mint transaction and receives royalties. It is also referred to as token author in this dataset.
mint_count integer 100 How many token instances were minted. This number does not change: it is impossible to mint additional token instances in hicetnunc for now.
mint_iso_date
mint_stamp
mint_hash
mint_row_id
string
integer
string
integer
"2021-03-01T15:00:03Z"
1614610803
"oom5Ju6X9nYpBCi..."
42181049
Mint event.
artifact_ipfs string "Qma11k..." IPFS address of NFT content.
artifact_mime string "image/gif" Mime type of artifact_ipfs.
artifact_file_size integer 2043418 File size of artifact_ipfs in bytes.
artifact_preview_width integer 1024 Preview image width of artifact_ipfs in pixels. Not more, than 1024.
artifact_preview_height integer 781 Preview image height of artifact_ipfs in pixels. Not more, than 1024.
info_title string "Dali tower" Title of the NFT specified by creator.
info_description string "generated by ..." Description of the NFT specified by creator.
info_tags string "tag1 tag2" List of the NFT tags specified by creator delimited with space character.
ban_status string "banned" If empty - token is ok, banned - token is banned, author_banned - token author is banned. Ban status is taken from hicetnunc repository
Statistics fields
author_sold_count
author_sold_nonzero_count
author_sold_zero_count
author_sold_prices_min
author_sold_prices_max
author_sold_prices_sum
author_sold_prices_avg
integer
integer
float
float
float
float
100
10
1.25
2.5
33.7
1.325
Prices of sold works (in swaps) by author in XTZ.

Note, that theoretically single NFT instance can be sold by author multiple times in case the buyer returns it back.
secondary_sold_count
secondary_sold_nonzero_count
secondary_sold_zero_count
secondary_sold_prices_min
secondary_sold_prices_max
secondary_sold_prices_sum
secondary_sold_prices_avg
integer
integer
float
float
float
float
100
10
1.25
2.5
33.7
1.325
Prices of sold works (in swaps) by other users in XTZ.
available_count
available_nonzero_count
available_zero_count
available_prices_min
available_prices_max
available_prices_sum
available_prices_avg
integer
integer
float
float
float
float
100
10
1.25
2.5
33.7
1.325
Prices of available works (in swaps) both from author and on secondary market in XTZ.
burn_count integer 15 How many token instances were burned by creator or other users.
author_owns_count integer 5 How many token instances are owned by author (not including available_count).
other_own_count integer 95 How many token instances are owned by other users (not by burn, swap or author).
author_sent_count integer 5 How much instances were directly transferred by author to users without official swaps.
Auxiliary fields
info_ipfs string "Qma11k..." IPFS identifier of the NFT JSON metadata
display_uri_ipfs string "Qma11k..." Token preview image identifier on IPFS or empty string. Note, that it is rare and only used for specific file types like HTML
royalties float 10.0 "Royalties" parameter passed during token mint in percent.
info_creator string "tz1UBZUk..." Address of the token creator as specified in token meta JSON file. In 99.99% the same as issuer field, but sometimes empty probably due to some bug.
mint_tokens_receiver string "tz1UBZUk..." Address of the user, who received tokens after mint transaction. In 99% the same as issuer field.
mint_ah_row_id integer None Row id of mint operation by "Art house" contract (not by NFT contract).

addrs.json and addrs.csv — of all hicetnunc users

All users, who ever created or owned NFT token.

field type example description
address string "tz1UBZUk..." Address of the user in Tezos blockchain. Can be contract address as well.
first_action_iso_date
first_action_stamp
first_action_hash
first_action_row_id
string
integer
string
integer
"2021-03-01T15:00:03Z"
1614610803
"oom5Ju6X9nYpBCi..."
42181049
When first transaction (related to hicetnunc) with this address occurred.
tzkt_info_name string "dacosta works" User alias field according to tzkt.io metadata.
tzkt_info_twitter string "dacosta_works" User Twitter account according to tzkt.io metadata.
tzkt_info_email string "[email protected]" User e-mail according to tzkt.io metadata.
tzkt_info_instagram string "dacosta_works" User Instagram account according to tzkt.io metadata.
tzkt_info_site string "https://jsh.sh" User website according to tzkt.io metadata.
tzkt_info_description string "NFT artist ..." User description according to tzkt.io metadata.
tzkt_info_logo string "huson.png" User logo according to tzkt.io metadata. Add prefix services.tzkt.io/v1/avatars2/ to get the full url
tzkt_info_github string "josim" User GitHub account according to tzkt.io metadata.
tzkt_info_telegram string "thebaskia" User Telegram account according to tzkt.io metadata.
tzkt_info_facebook string "barbeaucodeart" User Facebook account according to tzkt.io metadata.
tzkt_info_reddit string "user/mathmakesart" User Reddit account according to tzkt.io metadata.
ban_status string "banned" If empty - author is ok, banned - user is banned, some_tokens_banned - user is not banned, but it minted banned tokens. Ban status is taken from hicetnunc repository
Statistics fields
mint_count integer 125 Number of minted works.
bought_count
bought_nonzero_count
bought_zero_count
bought_prices_min
bought_prices_max
bought_prices_sum
bought_prices_avg
integer
integer
float
float
float
float
100
10
1.25
2.5
33.7
1.325
Prices of bought works (in swaps) by this address in XTZ.
author_sold_count
author_sold_nonzero_count
author_sold_zero_count
author_sold_prices_min
author_sold_prices_max
author_sold_prices_sum
author_sold_prices_avg
integer
integer
float
float
float
float
100
10
1.25
2.5
33.7
1.325
Prices of sold works (in swaps) by author in XTZ.
secondary_sold_count
secondary_sold_nonzero_count
secondary_sold_zero_count
secondary_sold_prices_min
secondary_sold_prices_max
secondary_sold_prices_sum
secondary_sold_prices_avg
integer
integer
float
float
float
float
100
10
1.25
2.5
33.7
1.325
Prices of sold works on secondary market (i.e. swaps not by author) in XTZ.
available_count
available_nonzero_count
available_zero_count
available_prices_min
available_prices_max
available_prices_sum
available_prices_avg
integer
integer
float
float
float
float
100
10
1.25
2.5
33.7
1.325
Prices of available works (in swaps) both by author and other users in XTZ.
in_op_count integer 25 Number of incoming transactions
out_op_count integer 33 Number of outcoming transactions
money_received float 2.78999 Total volume of XTZ received by address in transactions related to hicetnunc
money_sent float 10 Total volume of XTZ sent by address in transactions related to hicetnunc

sells.json and sells.csv — of all purchases via "official" hicetnunc swaps

There is the following invariant:

price * count = total_royalties + total_comission + total_seller_income
field type example description
tr_iso_date
tr_stamp
tr_hash
tr_row_id
string
integer
string
integer
"2021-03-01T15:00:03Z"
1614610803
"oom5Ju6X9nYpBCi..."
42181049
When transaction occurred.
token_id string "205" Numeric string with token identifier.
count integer 10 Number of token instances transferred.
seller string "tz1UBZUk..." Seller address i.e. user who created the swap
buyer string "tz1MwvWa..." Buyer address
price float 2.5 Price per item in XTZ (how much buyer payed per item)
swap_id string "503" Numeric string with swap identifier.
by_author boolean 1 0 If 1 - swap was created by author, if 0 - by other user.
author string "tz1UBZUk..." Address of token issuer, which receives royalties.
Statistics fields
total_royalties float 12.75 Author income from royalties (already multiplied by count) for secondary sells and 0 otherwise
total_comission float 1.3 Comission per item in XTZ. Usually 2.5% of price.
total_seller_income float 1.3 Seller income (already multiplied by count).

transfers.json and transfers.csv — all token transfers

field type example description
tr_iso_date
tr_stamp
tr_hash
tr_row_id
string
integer
string
integer
"2021-03-01T15:00:03Z"
1614610803
"oom5Ju6X9nYpBCi..."
42181049
When transaction occurred.
category string mint->author
author->swap
swap->author
swap->user
user->user
author->burn
user->burn
user->ext
...
String, which characterizes transfer in following terms: author, swap, user, burn, ext.
This division is a convention used in this repository only.
token_id string "207" Numeric string with token identifier.
price float 1.5 Price per item in XTZ or 0 for direct transfers. For external swaps price is guessed heuristically as half of sum of abolute values of transaction money transers (since each money transaction is counted twice both as incoming and outcoming).
count integer 5 Number of token instances transferred.
swap_id string "523" Numeric string with swap identifier or empty string if it is a direct transfer.
sender string "tz1UBZUk..." Sender address or empty string in case of mint
receiver string "tz1MwvWa..." Receiver address
method string "mint_OBJKT" Called method of hicetnunc or external contract, otherwise empty string

swaps.json and swaps.csv — all "official" hicetnunc swaps ever created

field type example description
swap_id string "503" Numeric string with swap identifier.
token_id string "207" Numeric string with token identifier.
price float 1.5 Price per item in XTZ.
total_count integer 100 Total number of instances in swap.
created_iso_date
created_stamp
created_hash
created_row_id
string
integer
string
integer
"2021-03-01T15:00:03Z"
1614610803
"oom5Ju6X9nYpBCi..."
42181049
When swap was created.
closed_iso_date
closed_stamp
closed_hash
closed_row_id
string
integer
string
integer
"2021-03-01T15:00:03Z"
1614610803
"oom5Ju6X9nYpBCi..."
42181049
When swap was closed (cancelled). If closed, swap cannot be reopened again.
by_author boolean 1 0 If 1 - swap was created by author, if 0 - by other user.
created_by string "tz1UBZUk..." Address of the user, which created swap.
Statistics fields
sold_count integer 99 Number of sold (including with zero price) instances.
available_count integer 1 Number of available instances if the swap is not closed, 0 otherwise.
returned_count integer 0 Number of unsold instances returned to the swap creator if the swap was closed, 0 otherwise.
sold_price_sum float 148.5 Total price of sold instances in XTZ.

Essential information about contracts logic

There are 3 Tezos addresses, which are common to most of hicetnunc transactions:

  • "NFT" contract: KT1RJ6PbjHpwc3M5rw5s2Nbmefwbuwbdxton.
    • This is the registry of all owners of all NFT tokens.
      • This is the typical way how most of NFTs works on Ethereum or on Tezos.
      • Click on "Bigmap #511" tab in Tezos explorer to see registry of token owners.
      • This contract is the single "source of truth" about current owners of all NFT tokens issued by hicetnunc. If there is no information about token owner in the registry, than that person doesn't own any tokens.
    • This contract also the registry of tokens infos (metadata).
      • Token info is a small JSON structure stored on on IPFS.
        • Here is an example of such structure.
        • It contains link to IPFS with NFT binary contents (some image, for example).
        • It also contains title, description, creator and tags.
        • It does not contain price or related information.
      • Only link to IPFS is stored on the blockchain.
      • Note, that, however, there is no way to alter token metadata after minting.
      • Click on "Bigmap #514" tab in Tezos explorer to see mapping from tokens to IPFS urls.
    • Every token owner can call "transfer" method of the contract to send tokens to other address.
      • This contract can't do any money related operations. Money logic should be implemented in other contracts, which call "NFT" contract as a part of transaction operation.
    • There is also a "mint" method in this contract, but it can only be called by the "Art house" contract.
  • "Art house" contract: KT1Hkg5qeNhfwpKW4fXvq7HGZB9z2EnmCCA9.
    • This contract implements money related operations on hicetnunc.
    • It's main structure is a swap. It is some amount of tokens, which are available of sale for specific price.
      • Click on "Bigmap #523" tab in Tezos explorer to see all current swaps.
    • Note, that there may be other contracts implementing swap mechanism. These contracts may decide not to pay comission or royalties.
    • Objects can be minted only with this contract by calling "mint_OBJKT" method.
      • This contract keeps track of royalties and assigns tokens ids.
  • Comission wallet: tz1UBZUkXpKGhYsP5KtzDNqLLchwF4uHrGjw.
    • 2.5% of every purchase via "Art house" contract swaps is sent to this wallet.

There are several other contracts related to curation and hDAO mechanisms, which are independent from the contracts mentioned above.

Actually, hicetnunc was not created just as another NFT marketplace, it has much broader mission as hDAO (hicetnunc DAO). You can get the idea of the creators vision on hicetnunc blog. As a result, only a small subset of contract's logic is actually used during hicetnunc website operation.

Official and external swap mechanisms

  • Official swap mechanism by hicetnunc "Art house" contract.

    • Any token bought on hicetnunc website is a part of some swap.
    • Swap is just some amount of tokens, which are offered for sale by specific price.
    • When swap is created, the seller sends all offered tokens to the "Art house" smart contract.
    • Then anybody can send the required amount of money to the "collect" method of contract and get tokens in return.
      • 2.5% of comission is transferred to hicetnunc comission wallet.
      • 10% of royalties (this parameter is configurable in general) is transferred to token author.
      • rest of money is sent to swap creator.
      • proportional amount of hDAO tokens are also sent to buyer, seller, token author and comission wallet.
    • Seller can cancel swap any time and get unsold tokens back.
    • Swaps can be created by any token owner any number of times.
    • In this dataset official swaps are treated as author->swap, other->swap, swap->author, swap->other transfers.
  • External swap mechanisms.

    • Since "transfer" method of "NFT" contract can be called by any token owner directly, it is possible to make custom smart contracts, which implement any desired logic.
    • These custom contracts are not required not pay comission or royalties to hicetnunc.
    • In general, swap contracts can be used to exchange any types entities.
    • Example: https://quipuswap.com/swap
    • In this dataset external swaps are treated as other->other transfers. The related price is guessed heuristically (as half of money transferred in all operations) and may not be always correct.

Token lifecycle

In contrast to NFT definition, each NFT artwork in hicetnunc can have multiple copies, which are fungible. The NFT contract only tracks the amount of copies owned by each address. This means, that there is no way (even in theory) to track history of single copy like it can be done on OpenSea, for example.

It is possible, however, to track history of token groups to some extent. Here is a list of possible owner types in this dataset:

  • author — the person, who created the tokens during mint.
  • user — any other hicetnunc user.
  • ext — any external contract (external swap mechanism, for example).
  • burn — reserved address for burning tokens.
  • swap — when tokens are offered on sale in official swaps.

List of possible transitions:

  • mint->author, mint->user First, every token should be minted.
    • For each token type there may exist only single mint operation. It is impossible to mint additional tokens later.
    • The only way to mint a token is to call "mint_OBJKT" method in "Art house" contract.
    • Here is a typical mint transaction
    • Internally it calls "mint" method in "NFT" contract.
    • In dataset the sender is empty for mint operations.
    • As result of mint operation, all tokens are transferred to some address. In 99% of cases this is the transaction sender, but sometimes it is different.
    • Royalties are always sent to the mint transaction sender.
  • author->swap, user->swap Any token owner can create official swap.
    • Hicetnunc swap is created by calling "swap" method in "Art house" contract.
    • Here is a typical swap creation transaction by author.
    • Internally tokens are transferred to the "Art house" address.
  • swap->author, swap->other There are two situations, when tokens may be transferred from a swap.
    1. Purchase
      • When token is purchased on hicetnunc website, it is transferred to the buyer. This is the main operation on hicetnunc.
      • Buyer should call "collect" method of "Art house" contract and send required amount of money with it.
      • Here is an example of "collect" transaction.
        • First 3 internal operations send money to token creator (royalties), hicetnunc wallet (comission) and to the seller (which is the same as token creator in some cases) in that order.
        • Fourth operation creates hDAO tokens and sends them to the buyer, seller and hicetnunc wallet. These tokens have special meaning and are not tracked in this dataset.
        • Last internal operation does the actual token transfer.
      • Note, that case of zero price is handled differently.
        • Here is an example of purchase with zero price.
    2. Swap cancel
      • When swap creator decides to cancel swap, all remaining tokens are transferred back to him.
  • author->user, user->user Any token owner can transfer tokens directly to other users for free by calling "transfer" method of "NFT" contract.
    • Here is an example of direct transfer transaction from author to other user.
  • author->ext, user->ext, ext->user, ext->author — external swaps
    • Here is an example of external swap.
  • author->burn, other->burn Any token owner can transfer tokens to burn address tz1burnburnburnburnburnburnburjAYjjX.
    • Here is and example of burn transfer from author.
    • Tokens can never be transferred from burn address since it is impossible to retrieve its private key (similar to how it is impossible to reverse hash containing all zeros).

Details about edge cases

How to define the author of the token

  1. mint_sender The address of the sender of the "mint" transaction.
    • This is the person, who receives royalties in hicetnunc.
    • In this dataset token author this is equivalent to token author.
  2. issuer The address of the receiver of tokens after the mint transaction.
    • It is also the first parameter of the "mint" call in "Art house" contract.
    • Here is an example of mint, where transaction sender and token issuer are different.
  3. info_creator Field "creators" in JSON in token metadata.
    • Here is an example of mint, where metadata creator field is different from transaction sender and issuer.
    • As of 4th of April, it always has single entry.
    • Sometimes it is empty.
      • Here is an example of mint with empty metadata creator field.
      • Note, that corresponding token page has a bug, that it shows token owner controls on token page.

Hicetnunc core addresses can own NFTs as regular users

  • Any user can send any NFT tokens to "NFT" or "Art house contract"
    • Technically, it has the same effect as sending this tokens to burn address, since contracts were not programmed to send their own NFTs (except from swap mechanism) under any circumstances.
  • Comission wallet sometimes mint NFTs and buys them from other users.
    • Since it is not a contract and manipulated by a real person (hicetnunc creator).

Void transactions

  • It is possible to send 0 tokens. Example.
  • Sender and receiver can be the same. Example

Strange issue with negative swap count:

How to update dataset

Note, that the code is still experimental and may require substantial changes to make it run. But you can read introduction in HACKING.md




鲜花

握手

雷人

路过

鸡蛋
该文章已有0人参与评论

请发表评论

全部评论

专题导读
上一篇:
textileio/ios-ipfs-lite发布时间:2022-06-22
下一篇:
daijiale/ipfs-re-encryption: Some Re-encrytion Demo For IPFS Data发布时间:2022-06-22
热门推荐
阅读排行榜

扫描微信二维码

查看手机版网站

随时了解更新最新资讯

139-2527-9053

在线客服(服务时间 9:00~18:00)

在线QQ客服
地址:深圳市南山区西丽大学城创智工业园
电邮:jeky_zhao#qq.com
移动电话:139-2527-9053

Powered by 互联科技 X3.4© 2001-2213 极客世界.|Sitemap