Emissions by louispt1 · Pull Request #1754 · quintel/etengine

louispt1 · 2026-05-21T18:50:11Z

Summary of changes

Emissions Calculation Framework

New DirectEmissions module implementing mass-balance equations for fossil and biogenic CO2
Tracks carbon flows: input content (A) + utilization (B) - output content (C) - capture (D) = emissions (E)
New MoleculeEmissions module for emissions reporting on molecule nodes
Support for CO2 capture (via ccs_capture_rate) and utilization (via co2_utilisation_per_mj)

Carbon Content Tracing

RecursiveFactor::DirectEmissions module for tracing CO2 content through supply chains
Handles mixed carriers (network gas, crude oil) by recursively calculating weighted composition
New emissions_skip_crude_oil_mix edge group for forcing weighted mix calculations

GQL & Dataset Integration

New EMISSIONS() GQL function for accessing emissions data from datasets
Emissions data loading infrastructure in Dataset::Import and Etsource::Loader
Graph-level emissions hash storage

Supporting Infrastructure

emissions node group for nodes participating in emissions tracking
ccus_captured node group for CO2 removal/capture technologies
- Uses lazy memoization for ccus_captured? check to avoid Marshal serialization issues
Updated biogenic emissions (primary) to use free_co2_factor for capture calculations with consistent results

Other

Reporting methods return nil for non-emissions nodes (checked via with_emissions_node)
Atlas gem updated to support emissions data structures

Note: Atlas reference should be updated once the Atlas emissions branch has been merged.
Goes with:

kndehaan · 2026-05-22T11:05:49Z

Note that direct_co2_output_content_carriers_biogenic doesn't give the correct results yet, also leading to incorrect results for methods that depend on this one.

This has still to with the potential_co2_conversion_per_mj attribute missing on carriers, as a consequence going to recursion where it should not always be so. This is something that we still need to look into.

louispt1 · 2026-05-27T08:39:10Z

@kndehaan I have pushed a fix for the biogenic output content carriers now

noracato

I'm worried that the UPDATE part does not have enough validation.

Can you please add tests that show updating nonexistent keys will not work? I have a feeling we will need the DatasetAttributes module after all (it is still on the emissions-gql branch)

noracato · 2026-05-28T17:27:38Z

    # See Qernel::Dataset#assign_dataset_attributes to understand what's going on:
    call_on_each_qernel_object(:assign_dataset_attributes)
+    # Manually assign emissions hash (not a DatasetAttributes object)
+    @emissions = dataset&.data&.[](:emissions) || {}


Can you explain once more why not to use DatasetAttributes? Like this there is no validation on what users can set as an emissions attribute.

Good point, I was not really looking at UPDATE yet, so I thought to keep it as light as possible for read.
However, you are right that it's cleaner to just use the DatasetAttributes approach for consistency and validation.

I re-instated the DatasetAttributes approach and added more tests to cover the case of nonexistent keys etc. I expect more spec will be needed based on the changes for 1990 but I think it's best to process that separately.

…ion methods

…ions

… and create boutique DE RF module to keep edge cases in one spot

…ular references for Marshall

…co2_output_production_emissions_fossil and spec

…ce of ccs_capture_rate. Update spec

…thod as direct_co2_input_utilisation_fossil is sufficient

…_utilised and emissions_lulucf_removals checks from spec

…f required

…now it covers more cases

…cus_captured?

…ut co2 content approach

…ents

…spec

* Expand ConfiguredCSVSerializer with node group expansion, molecule support, and emissions CSV endpoints * Add ghg_carrier method to DirectEmissions/MoleculeEmissions and update test fixture of the direct_emissions_csv * Add ghg_carrier to MoleculeEmissions specs and simplify DirectEmissions specs to use faster mock-based approach * Return symbols from ghg_carrier instead of strings

kndehaan · 2026-06-01T12:37:57Z

Adding myself as reviewer to check the descriptions for the methods (it should be understandable and correct from a modeller's perspective as well).

kndehaan

I put up some text suggestions and placed one comment about removing a method. Let me know if you have questions @louispt1

kndehaan · 2026-06-01T13:26:38Z

+      # Total CO2 utilised (consumed as feedstock) at this node.
+      #
+      # Currently returns only fossil utilisation, as biogenic utilisation is always 0.
+      #
+      # @return [Float, nil] Total CO2 utilised in kg, or nil if node is not in emissions group
+      def direct_co2_input_utilisation
+        with_emissions_node do
+          direct_co2_input_utilisation_fossil
+          # Potentially in the future: + direct_co2_input_utilisation_biogenic (currently 0)
+        end
+      end


I think we decided to remove this method for now, as we currently don't use it right now, right? @louispt1

Yes this method snuck back in with the csv serialiser merge - I will remove it

Co-authored-by: kndehaan <102598197+kndehaan@users.noreply.github.com>

noracato · 2026-06-02T06:24:57Z

+      VALID_GHG_TYPES = %w[co2 other_ghg n2o ch4 hfc pfc sf6 nf3].freeze
+      VALID_GHG_PATTERN = /^(#{VALID_GHG_TYPES.join('|')})(_\d{4})?$/.freeze


Sorry to comment again, but let's not use constants with carriers and patterns. These keys have nothing to do with engine validations, they are part of the dataset pipeline IMO - or at least ETSource. The engine should not carry this knowledge.

No need to apologise, you're spot on. I pushed a change removing the validation.
Edit:
However now I need to figure out how to handle validation a little smarter - still working on it :)

@noracato what do you think about this approach?

… the emissions.csv file

noracato

Sorry to keep nitpicking. There is still logic present in the module that I feel over complicates it.

noracato · 2026-06-02T10:24:51Z

+        # For setters, check if the emission key exists in the dataset
+        if method_name.to_s.end_with?('=')
+          data_key = scoped_method(method_name.to_s.sub(/=$/, ''))
+          @emissions.dataset_has_key?(data_key)
+        else
+          # Getters always respond (may return nil if key doesn't exist)
+          true
+        end


Suggested change

# For setters, check if the emission key exists in the dataset

if method_name.to_s.end_with?('=')

data_key = scoped_method(method_name.to_s.sub(/=$/, ''))

@emissions.dataset_has_key?(data_key)

else

# Getters always respond (may return nil if key doesn't exist)

true

end

# Getters always respond (may return nil if key doesn't exist)

return true unless method_name.to_s.end_with?('=')

@emissions.dataset_has_key?(

scoped_method(method_name.to_s.sub(/=$/, ''))

)

General approach is good. This is a bit pythonesque ;)

Actually, what was wrong with:

def respond_to_missing?(method_name, include_private = false) data_key = scoped_method(method_name).split('=').first @emissions.respond_to?(data_key) || super end

noracato · 2026-06-02T10:26:26Z

+      # Check both string and symbol keys since datasets may use either
+      dataset_attributes.key?(key.to_s) || dataset_attributes.key?(key.to_sym)


I don't feel this should be handled here?

…gql approach more closely

louispt1 requested a review from noracato May 21, 2026 18:50

louispt1 force-pushed the emissions branch from 06c23dd to d423987 Compare May 27, 2026 14:58

louispt1 marked this pull request as ready for review May 27, 2026 14:59

noracato requested changes May 28, 2026

View reviewed changes

This was referenced May 29, 2026

Add direct emissions method with data export quintel/etsource#3444

Open

Add direct emissions method with data export quintel/etmodel#4719

Open

Emissions quintel/atlas#189

Open

louispt1 and others added 21 commits June 1, 2026 09:38

Implement A, C and E: input and output without capture, & CO2 product…

362d87e

…ion methods

Add guard based on the emissions group

f2a177e

Apply recursive edge logic to output and input edges for direct emiss…

c67eb36

…ions

Dataset EMISSIONS method implementation

89faeb0

Add emissions methods for molecule nodes

9e360b8

Improve crude oil tests and recursion through multiple levels

21b7191

Optimise direct emissions checks using memoization of the group check…

4fe9433

… and create boutique DE RF module to keep edge cases in one spot

remove memoization from the edge group check as this was causing circ…

abe362c

…ular references for Marshall

Correction to emissions method

81dc529

Handle split LULUCF in molecule emissions

66702de

Biogenic input output and production methods for direct emissions

5d9b9a3

Direct emissions utilisation and tests

f1ef35e

Update attributes.rb with co2_utilisation_per_mj

829f77d

Refactor direct emissions, introduce capture methods, correct direct_…

de044fa

…co2_output_production_emissions_fossil and spec

Update existing engine methods to account for captured CO2

f8b9bc8

Update bio-emissions to use free_co2_factor, depending on the prescen…

5a3a89c

…ce of ccs_capture_rate. Update spec

Update methods based on excel formulas

87d6111

Adjust comments, add TODOs and remove direct_co2_input_utilisation me…

6eb705c

…thod as direct_co2_input_utilisation_fossil is sufficient

Implement emissions method for CO2 capture on molecule nodes

60cb93e

Handle co2_per_mj check when recursing for bio emissions, remove ccus…

b8db250

…_utilised and emissions_lulucf_removals checks from spec

Update atlas reference to explicit commit hash

004e85b

louispt1 and others added 10 commits June 1, 2026 09:38

Removed scoped accessor in favour of hash - can re-instate accessor i…

1cab4bb

…f required

return ccus_captured to eager memoization to match other methods and …

3bc39a3

…now it covers more cases

Resolve biogenic recursion issue and return to lazy memoization for c…

292b8fb

…cus_captured?

Node colour for Waste and LULUCF sectors

4d7ddf7

Fix for biogenic output

ea575bb

Add node level check for biogenic CO2, simplify direct emissions outp…

ed4f5d4

…ut co2 content approach

Clean up comments for PR

67a0ef4

Update atlas reference to latest commit hash

94c8174

Replace ABCDE references with worded fomulas in direct emissions comm…

2d89c74

…ents

Restore DatasetAttribute approach and add additional UPDATE tests

f136c00

louispt1 force-pushed the emissions branch from f4e3149 to f136c00 Compare June 1, 2026 07:38

louispt1 and others added 2 commits June 1, 2026 11:54

Fix bug for crude oil edge case biogenic direct emissions and expand …

3f36a4e

…spec

kndehaan self-requested a review June 1, 2026 12:37

kndehaan reviewed Jun 1, 2026

View reviewed changes

louispt1 and others added 2 commits June 1, 2026 15:39

Apply suggestions to modeller comments direct emissions methods

147632a

Co-authored-by: kndehaan <102598197+kndehaan@users.noreply.github.com>

Remove direct_co2_input_utilisation method

b15763f

noracato requested changes Jun 2, 2026

View reviewed changes

louispt1 added 2 commits June 2, 2026 09:14

Remove hardcoded validations from Qernel::Emissions

1bcc3eb

Validation relies on the set of emissions keys and types available in…

b7bce7c

… the emissions.csv file

noracato requested changes Jun 2, 2026

View reviewed changes

Revert to strict accessor and update specs to match, mimic emissions-…

8a52ddf

…gql approach more closely

		VALID_GHG_TYPES = %w[co2 other_ghg n2o ch4 hfc pfc sf6 nf3].freeze
		VALID_GHG_PATTERN = /^(#{VALID_GHG_TYPES.join('\|')})(_\d{4})?$/.freeze

		# Check both string and symbol keys since datasets may use either
		dataset_attributes.key?(key.to_s) \|\| dataset_attributes.key?(key.to_sym)

Conversation

louispt1 commented May 21, 2026 • edited by kndehaan Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Emissions Calculation Framework

Carbon Content Tracing

GQL & Dataset Integration

Supporting Infrastructure

Other

Uh oh!

kndehaan commented May 22, 2026

Uh oh!

louispt1 commented May 27, 2026

Uh oh!

noracato left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kndehaan commented Jun 1, 2026

Uh oh!

kndehaan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

louispt1 Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

noracato left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

louispt1 commented May 21, 2026 •

edited by kndehaan

Loading

louispt1 Jun 2, 2026 •

edited

Loading