Introducing Olive Oil Trust: subgraph

Published on 7 min read
View on GitHub

This post is part of a series of posts regarding Olive Oil Trust:

As we saw in the previous post, Olive Oil Trust contracts emit many events with important data that have to be collected, along with metadata that have to be aggregated.

Subscribing to all of these events in a front-end application would be inefficient, and any data emitted prior to the subscription would be missed.

Furthermore, processing all emitted events starting from the block when the contract was deployed onwards and filtering them out would also be inconvenient.

That is why I make use of a subgraph, which effectively acts as an indexing layer between contracts and user interface.

In this post I try to explain how to create a subgraph using TheGraph that can be used to query data from Olive Oil Trust in an efficient way.

In this article

The subgraph manifest

The subgraph manifest specifies what contracts the subgraph indexes, what events to react to and how event data are mapped to entities that Graph Node stores and allows to query.

The hardhat-env workspace in the Olive Oil Trust monorepository generates a deployments folder that contains a folder for each network where the contracts are deployed to.

Each folder of a network contains a JSON file for each contract that is deployed to that network.

These JSON files gather key information that is needed in order to generate a manifest that enables the subgraph to retrieve data from the blockchain efficiently.

The manifest is automatically generated from a template, using the script generateSubgraph in subgraph/scripts/index.ts, in order to support multiple deployments to different networks dynamically.

This is performed in the npm "pre" script npm run precodegen. The script npm run codegen runs the command graph codegen:

"precodegen": "ts-node scripts generateSubgraph --deployment localhost",
"codegen": "graph codegen subgraph.yaml --output-dir src/generated/types/",

generateSubgraph will replace key fields with values from a JSON file in subgraph/src/generated/config with the name that matches the network in the parameter --deployment.

That JSON file is generated when running npm run hardhat:share. If there are deployments to multiple networks, a JSON file will be generated for each network.

Fields

subgraph/templates/subgraph.template.yaml contains five fields with static data:

subgraph/templates/subgraph.template.yaml
specVersion: 0.0.4
description: Olive Oil Trust for Ethereum
repository: https://github.com/albertobas/olive-oil-trust/subgraph
schema:
  file: ./schema.graphql
features:
  - ipfsOnEthereumContracts
  - fullTextSearch

You can find the full specification for subgraph manifests here.

The subgraph template also contains the field dataSources, which features some fields with keys between curly brackets that will be replaced for values by generateSubgraph.

dataSources defines data that are ingested, as well as the transformation logic to derive the state of the subgraph's entities based on the source data.

There is one data source per contract. For instance, let's see the data source of the Bottling Company contract:

subgraph/templates/subgraph.template.yaml
- name: {{BottlingCompanyModule}}DataSource
  kind: ethereum/contract
  network: {{network}}
  source:
    address: '{{BottlingCompanyAddress}}'
    abi: {{BottlingCompanyModule}}
    startBlock: {{BottlingCompanyStartBlock}}
  mapping:
    kind: ethereum/events
    apiVersion: 0.0.6
    language: wasm/assemblyscript
    file: ./src/mappings/{{BottlingCompanyModule}}.ts
    entities:
      - Member
      - OwnershipTransferred
      - Contract
      - Transaction
      - Account
    abis:
      - name: {{BottlingCompanyModule}}
        file: ./src/generated/abis/{{BottlingCompanyModule}}.json
    eventHandlers:
      - event: NameSet(string)
        handler: handleNameSet
      - event: OwnershipTransferred(indexed address,indexed address)
        handler: handleOwnershipTransferred
      - event: TokenAncestrySet(indexed address,indexed bytes32,indexed bytes32,address[][],bytes32[][],bytes32[][],uint256[][])
        handler: handleTokenAncestrySet

The keys {{BottlingCompanyModule}}, {{network}}, {{BottlingCompanyAddress}} and {{BottlingCompanyStartBlock}} will be replaced by the module of that contract, the network name, contract address and start block respectively.

The Olive Oil Trust subgraph manifest is designed in a way that mappings can be reused among member contracts that inherit from the same Olive Oil Trust contract.

For example, Bottling Company 2 -being a bottling plant- would use the same mapping as Bottling Company since they would both inherit from BottlingPlantUpgradeable.sol.

ABIs are in src/generated/abis and also get generated when running npm run hardhat:share.

The first data source of the contracts of the same type (for instance, dependent tokens) contains a key of the name of the module it inherits from instead of the name of that contract:

subgraph/templates/subgraph.template.yaml
# ----------------------------------
#          DEPENDENT TOKENS
# ----------------------------------
- name: {{BottlingCompanyOliveOilBottleModule}}DataSource
  kind: ethereum/contract
  network: {{network}}
  source:
    address: '{{BottlingCompanyOliveOilBottleAddress}}'
    abi: {{BottlingCompanyOliveOilBottleModule}}
    startBlock: {{BottlingCompanyOliveOilBottleStartBlock}}
  mapping:
    kind: ethereum/events
    apiVersion: 0.0.6
    language: wasm/assemblyscript
    file: ./src/mappings/{{BottlingCompanyOliveOilBottleModule}}.ts
    entities:
      - Token
      - Account
      - Contract
      - TokenOperator
      - Transaction
      - TokenBalance
      - TokenTransfer
      - OwnershipTransferred
      - TokenTypeInfo
      - TokenTypeInstructionsSet
      - Member
    abis:
      - name: {{BottlingCompanyOliveOilBottleModule}}
        file: ./src/generated/abis/{{BottlingCompanyOliveOilBottleModule}}.json
    eventHandlers:
      - event: TokenTransferred(indexed address,indexed address,indexed address,bytes32,bytes32,uint256)
        handler: handleTokenTransferred
      - event: BatchTransferred(indexed address,indexed address,indexed address,bytes32[],bytes32[],uint256[])
        handler: handleBatchTransferred
      - event: TokenTypeInstructionsSet(indexed address,bytes32,address[],bytes32[],uint256[])
        handler: handleTokenTypeInstructionsSet
      - event: TokenTypesInstructionsSet(indexed address,bytes32[],address[][],bytes32[][],uint256[][])
        handler: handleTokenTypesInstructionsSet
      - event: ApprovalForAll(indexed address,indexed address,bool)
        handler: handleApprovalForAll
      - event: OwnershipTransferred(indexed address,indexed address)
        handler: handleOwnershipTransferred
- name: OliveOilMillCompanyOliveOilDataSource
  kind: ethereum/contract
  network: {{network}}
  source:
    address: '{{OliveOilMillCompanyOliveOilAddress}}'
    abi: {{OliveOilMillCompanyOliveOilModule}}
    startBlock: {{OliveOilMillCompanyOliveOilStartBlock}}
  mapping:
    kind: ethereum/events
    apiVersion: 0.0.6
    language: wasm/assemblyscript
    file: ./src/mappings/{{OliveOilMillCompanyOliveOilModule}}.ts
    entities:
      - Token
      - Account
      - Contract
      - TokenOperator
      - Transaction
      - TokenBalance
      - TokenTransfer
      - OwnershipTransferred
      - TokenTypeInfo
      - TokenTypeInstructionsSet
      - Member
    abis:
      - name: {{OliveOilMillCompanyOliveOilModule}}
        file: ./src/generated/abis/{{OliveOilMillCompanyOliveOilModule}}.json
    eventHandlers:
      - event: TokenTransferred(indexed address,indexed address,indexed address,bytes32,bytes32,uint256)
        handler: handleTokenTransferred
      - event: BatchTransferred(indexed address,indexed address,indexed address,bytes32[],bytes32[],uint256[])
        handler: handleBatchTransferred
      - event: TokenTypeInstructionsSet(indexed address,bytes32,address[],bytes32[],uint256[])
        handler: handleTokenTypeInstructionsSet
      - event: TokenTypesInstructionsSet(indexed address,bytes32[],address[][],bytes32[][],uint256[][])
        handler: handleTokenTypesInstructionsSet
      - event: ApprovalForAll(indexed address,indexed address,bool)
        handler: handleApprovalForAll
      - event: OwnershipTransferred(indexed address,indexed address)
        handler: handleOwnershipTransferred

This is so that there are generated types with the nomenclature of the Olive Oil Trust modules that can be imported in the AssemblyScript mapping files.

The GraphQL schema

In order to come up with a good concept on how to design the GraphQL schema, I have focused on how entities are defined in the OpenZeppelin subgraphs.

The idea is to build a dense subgraph, linking as many data types as it makes sense in order to make complex queries as easyly and efficiently as possible.

I have followed a few guidelines in this OpenZeppelin's blog post to design the schema.

It all starts with how contracts are designed so that everything can be indexed solely using events, but also there are some other aspects that are summarised in the following sections.

Creating entities for high-level concepts

In Olive Oil Trust there are high-level concepts like tokens, escrows, certificates, accounts, balances, etc.

For example, Account entities are objects that are used to gather data about any address that appears in an event (including the address of the contract that emitted the event).

We can see the type Account below:

subgraph/schema.graphql
type Account @entity {
  id: Bytes!
  asCertificateContract: CertificateContract
  asEscrowContract: EscrowContract
  asMemberContract: MemberContract
  asTokenContract: TokenContract
  escrowBalance: Balance
  events: [IEvent!] @derivedFrom(field: "emitter")
  ownerOfMember: [MemberContract!] @derivedFrom(field: "owner")
  ownerOfCertificateContract: [CertificateContract!]! @derivedFrom(field: "owner")
  ownerOfEscrowContract: [EscrowContract!]! @derivedFrom(field: "owner")
  ownerOfTokenContract: [TokenContract!]! @derivedFrom(field: "owner")
  ownershipTransferred: [OwnershipTransferred!] @derivedFrom(field: "owner")
  tokenBalances: [Balance!] @derivedFrom(field: "tokenAccount")
  tokenOperatorOwner: [TokenOperator!] @derivedFrom(field: "owner")
  tokenOperatorOperator: [TokenOperator!] @derivedFrom(field: "operator")
  tokenTransferFromEvent: [TokenTransfer!] @derivedFrom(field: "from")
  tokenTransferToEvent: [TokenTransfer!] @derivedFrom(field: "to")
  tokenTransferOperatorEvent: [TokenTransfer!] @derivedFrom(field: "operator")
}

Creating entities for low-level concepts

In Olive Oil Trust, low-level concepts refer to the events that are emitted by the Olive Oil Trust contracts.

For instance, the certificate contract emits an event of type TokenCertified when a certifier certifies a type of token and entity TokenCertification will give the structure of the object that will contain the data of the event emission:

subgraph/schema.graphql
type TokenCertification implements IEvent @entity(immutable: true) {
  id: String!
  certificate: Certificate!
  certificateContract: CertificateContract!
  emitter: Account!
  timestamp: BigInt!
  tokenContract: TokenContract!
  transaction: Transaction!
}

All the entities for low-level concepts are immutable and implement the interface IEvent since they all share some fields.

As we mentioned earlier, dense subgraphs help dealing with complex queries since there are many relationships between entities.

Let's say we log in as a certifier to a front-end app that consumes data from this subgraph and we would like the app to be served in a single query the name of the logged in member, its role, all the certificates it has issued along with the token types these certificates certify, and all the token types that exist in Olive Oil Trust.

An example of that query could be:

query CertificatesByMember($id: String!) {
  memberContract(id: $id) {
    id
    name
    role
    asAccount {
      ownerOfCertificateContract {
        id
        certificates {
          id
          tokenTypes {
            tokenType {
              id
            }
          }
        }
      }
    }
  }
  tokenTypes {
    id
  }
}

A certificate can certify multiple types of tokens, and a type of token can be certified by multiple certificates.

We can see in the query above that the highlighted field certificates, which represents an array of Certificate entities, contains the field tokenTypes.

This field represents an array of TokenTypeCertificateMapping entities which is a many-to-many relationship.

It is recommended to use mapping tables to store many-to-many relationship data in a performant way

The type TokenTypeCertificateMapping can be seen below:

subgraph/schema.graphql
type TokenTypeCertificateMapping @entity(immutable: true) {
  id: String!
  tokenType: TokenType!
  certificate: Certificate!
}

Writing mappings

Mappings are functions that map data from Ethereum to objects defined as entities.

As we saw earlier, entities can be seen as objects that contain data, these data are mapped from the blockchain through mapping functions.

All the mappings in the Olive Oil Trust subgraph can be found in subgraph/src/mappings.

npm run subgraph:codegen generates the subgraph manifest and typings which are needed in the mapping functions.

Let's say Bottling Company mints a new batch of olive oil bottle tokens of a certain type.

The function mint in hardhat-env/contracts/tokens/BottlingCompanyOliveOilBottle.sol will emit an event TokenTransferred that contains the following data:

hardhat-env/contracts/OliveOilTrust/interfaces/IBaseToken.sol
/// @dev Equivalent to IERC1155Upgradeable TransferSingle event but with a bytes32 id and tokenTypeId
event TokenTransferred(
    address indexed operator,
    address indexed from,
    address indexed to,
    bytes32 tokenTypeId,
    bytes32 tokenId,
    uint256 tokenAmount
);

In order to store these data in the Graph Node and given that BottlingCompanyOliveOilBottle is a dependent token, the subgraph will handle this event with the handleTokenTransferred mapping function in subgraph/src/mappings/DependentTokenUpgradeable.ts:

subgraph/templates/subgraph.template.yaml
- name: {{BottlingCompanyOliveOilBottleModule}}DataSource
  kind: ethereum/contract
  network: {{network}}
  source:
    address: '{{BottlingCompanyOliveOilBottleAddress}}'
    abi: {{BottlingCompanyOliveOilBottleModule}}
    startBlock: {{BottlingCompanyOliveOilBottleStartBlock}}
  mapping:
    kind: ethereum/events
    apiVersion: 0.0.6
    language: wasm/assemblyscript
    file: ./src/mappings/{{BottlingCompanyOliveOilBottleModule}}.ts
    entities:
      - Token
      - Account
      - Contract
      - TokenOperator
      - Transaction
      - TokenBalance
      - TokenTransfer
      - OwnershipTransferred
      - TokenTypeInfo
      - TokenTypeInstructionsSet
      - Member
    abis:
      - name: {{BottlingCompanyOliveOilBottleModule}}
        file: ./src/generated/abis/{{BottlingCompanyOliveOilBottleModule}}.json
    eventHandlers:
      - event: TokenTransferred(indexed address,indexed address,indexed address,bytes32,bytes32,uint256)
        handler: handleTokenTransferred
      - event: BatchTransferred(indexed address,indexed address,indexed address,bytes32[],bytes32[],uint256[])
        handler: handleBatchTransferred
      - event: TokenTypeInstructionsSet(indexed address,bytes32,address[],bytes32[],uint256[])
        handler: handleTokenTypeInstructionsSet
      - event: TokenTypesInstructionsSet(indexed address,bytes32[],address[][],bytes32[][],uint256[][])
        handler: handleTokenTypesInstructionsSet
      - event: ApprovalForAll(indexed address,indexed address,bool)
        handler: handleApprovalForAll
      - event: OwnershipTransferred(indexed address,indexed address)
        handler: handleOwnershipTransferred

The function handleTokenTransferred will then write several entities with the processed data to the Graph node store:

subgraph/src/mappings/DependentTokenUpgradeable.ts
function handleTransferred(
  event: ethereum.Event,
  tokenId: Bytes,
  tokenTypeId: Bytes,
  operatorAddress: Address,
  fromAddress: Address,
  toAddress: Address,
  value: BigInt
): void {
  let operator = ensureAccount(operatorAddress);
  let from = ensureAccount(fromAddress);
  let to = ensureAccount(toAddress);
  let token = ensureToken(tokenTypeId, tokenId, event.address);
  registerTokenTransfer(event, token, operator.id, from.id, to.id, value);
  if (from.id == Address.zero()) {
    let tokenType = ensureTokenType(token.contract, tokenTypeId, event.block.timestamp);
    token.tokenType = tokenType.id;
    token.mintingDate = event.block.timestamp;
    token.save();
  }
}
 
export function handleTokenTransferred(event: TokenTransferred): void {
  handleTransferred(
    event,
    event.params.tokenId,
    event.params.tokenTypeId,
    event.params.operator,
    event.params.from,
    event.params.to,
    event.params.tokenAmount
  );
}
 
export function handleBatchTransferred(event: BatchTransferred): void {
  for (let i = 0; i < event.params.tokenIds.length; i++) {
    handleTransferred(
      event,
      event.params.tokenIds[i],
      event.params.tokenTypeIds[i],
      event.params.operator,
      event.params.from,
      event.params.to,
      event.params.tokenAmounts[i]
    );
  }
}

As we can see, several new entities will be created in this process, including an entity TokenTransfer that has a relationship with the entity Balance through the balance field.

These entities will contain data emitted by the event and will reflect the changes in the balance of the Bottling Company's address in BottlingCompanyOliveOilBottle.sol.

Metadata

In order to simulate the workflow of the olive oil long value chain in the front-end app (that we'll see in the next post), we aggregate figurative metadata to the data written to Certificate and TokenType entities:

subgraph/src/utils/entities/TokenType.ts
export function ensureTokenType(contractId: Bytes, tokenTypeId: Bytes, timestamp: BigInt): TokenType {
  let id = getTokenTypeId(contractId, tokenTypeId);
  let tokenType = TokenType.load(id);
  if (tokenType === null) {
    let tokenTypeIdStr = tokenTypeId.toString();
    tokenType = new TokenType(id);
    tokenType.contract = contractId;
    tokenType.identifier = tokenTypeIdStr;
    tokenType.creationDate = timestamp;
    let metadata = getMetadata(tokenTypeIdStr);
    if (metadata) {
      tokenType.bottleQuality = metadata.bottleQuality;
      tokenType.bottleMaterial = metadata.bottleMaterial;
      if (metadata.bottleSize) {
        tokenType.bottleSize = BigInt.fromString(metadata.bottleSize!);
      }
      if (metadata.imageHeight) {
        tokenType.imageHeight = BigInt.fromString(metadata.imageHeight!);
      }
      tokenType.imagePath = metadata.imagePath;
      if (metadata.imageWidth) {
        tokenType.imageWidth = BigInt.fromString(metadata.imageWidth!);
      }
      tokenType.description = metadata.description;
      tokenType.oliveQuality = metadata.oliveQuality;
      if (metadata.oliveOilAcidity) {
        tokenType.oliveOilAcidity = BigDecimal.fromString(metadata.oliveOilAcidity!);
      }
      tokenType.oliveOilAroma = metadata.oliveOilAroma;
      tokenType.oliveOilBitterness = metadata.oliveOilBitterness;
      tokenType.oliveOilColour = metadata.oliveOilColour;
      tokenType.oliveOilFruitness = metadata.oliveOilFruitness;
      tokenType.oliveOilIntensity = metadata.oliveOilIntensity;
      tokenType.oliveOilItching = metadata.oliveOilItching;
      tokenType.oliveOrigin = metadata.oliveOrigin;
      tokenType.title = metadata.title;
    }
    tokenType.save();
  }
  return tokenType;
}

In this example, the metadata are local constants, but they could be stored in IPFS.

@graphprotocol/graph-ts provides us with helpers to deal with them. Once we had the IPFS hash we could just do ipfs.cat(hash) to get the JSON data.

Deploying the subgraph

Once the local Graph node is up and running, the process of creating and deploying the subgraph is fairly straightforward:

npm run subgraph:create && npm run subgraph:deploy

Now, the user interface will be able to consume data from the subgraph once the contracts are deployed and the subgraph indexes data from the events it reacts to.

Related posts

Microservices

Approach to a microservices-based architecture bank application

A microservices-based architecture bank application that includes back-end and front-end applications, as well as a...

Traceability

Introducing Olive Oil Trust

Introduction to a series of posts about Olive Oil Trust

Ready to #buidl?

Are you interested in Web3 or the synergies between blockchain technology, artificial intelligence and zero knowledge?. Then, do not hesitate to contact me by e-mail or on my LinkedIn profile. You can also find me on GitHub.