[BUG] Compilation of larger contract doesn't generate verification keys in version `devnet.6-patch.1`

[ewynx]

What are you trying to do?

We worked in version devnet.5 with 2 example contracts that both worked fine (they compiled and we were able to test them in TS script with local network). Updating to devnet.6-patch.1 the smaller contract still works fine, while for the larger contract it seems no verification keys get generated. (I'm calling them smaller and larger contract because that is a particular difference I know exists between the 2, but of course the error for the larger contract could be because of something else)

In previous version we ran

aztec-nargo compile
aztec-postprocess-contract

Now just

aztec compile

This runs without errors, but then when you try to use the artifact generated with aztec codegen -o src/artifacts target, it'll run into errors.

Code Reference

The smaller contract that works fine is here.

The larger contract that gives issues is here.

In this script we run tests using both contracts and you can see it will give errors for the larger contract. In this version of the script we commented out the calls that give issues, so you can see that the smaller contract works fine.

Aztec Version

devnet.6-patch.1

OS

MacOS

Browser (if relevant)

No response

Node Version

No response

Additional Context

No response

[benesjan]

Hello,
I just tried to reproduce this on the latest nightly and it works:

I see that tyoure BusinessProgram.ts file is truly messed up as the methods there should not contain the __aztec_nr_internals__ prefix - this implies the build in that version was broken for whatever reason.

I recommend you to update. I needed to do a few changes to get this to compile.

Here is the updated contract:

use aztec::macros::aztec;

#[aztec]
pub contract BusinessProgram {
    use ::json_parser::{JSON2055, JSON2kb};
    use att_verifier_lib::{verify_attestation_comm, verify_attestation_hashing};
    use aztec::macros::{events::event, functions::{external, initializer}};
    use aztec::macros::storage::storage;
    use aztec::protocol::address::AztecAddress;
    use aztec::state_vars::{PublicImmutable, PublicMutable};
    use std::embedded_curve_ops::EmbeddedCurvePoint;

    global MAX_URL_LEN: u32 = 1024;
    global MAX_MSGS_LEN: u32 = 2054;
    global MAX_CONTENT_LEN: u32 = 1000;
    global MAX_COMMS: u32 = 65;

    #[event]
    struct SuccessEvent {
        sender: AztecAddress,
        contract_address: AztecAddress,
        id: Field,
    }

    #[storage]
    struct Storage<Context> {
        admin: PublicMutable<AztecAddress, Context>,
        allowed_url_hashes: PublicMutable<[Field; 3], Context>,
        H: PublicImmutable<EmbeddedCurvePoint, Context>,
    }

    #[external("public")]
    fn set_admin(new_admin: AztecAddress) {
        assert(self.storage.admin.read().eq(self.msg_sender()), "caller is not admin");
        self.storage.admin.write(new_admin);
    }

    #[initializer]
    #[external("public")]
    fn constructor(admin: AztecAddress, allowed_url_hashes: [Field; 3], H: EmbeddedCurvePoint) {
        assert(!admin.is_zero(), "invalid admin");
        self.storage.admin.write(admin);
        self.storage.allowed_url_hashes.write(allowed_url_hashes);
        self.storage.H.initialize(H);
    }

    // TODO - allow update single url with index?
    #[external("public")]
    fn update_allowed_url_hashes(allowed_url_hashes: [Field; 3]) {
        assert(self.storage.admin.read().eq(self.msg_sender()), "caller is not admin");
        self.storage.allowed_url_hashes.write(allowed_url_hashes);
    }

    #[external("public")]
    fn check_values_emit_event(
        sender: AztecAddress,
        contract_address: AztecAddress,
        id: Field,
        allowed_url_matches_hashes: [Field; 2],
        H: EmbeddedCurvePoint,
    ) {
        // Check allowed_url_matches_hashes are in storage
        let allowed_url_hashes: [Field; 3] = self.storage.allowed_url_hashes.read();
        for i in 0..2 {
            let check_hash = allowed_url_matches_hashes[i];
            let mut found = false;
            for j in 0..3 {
                if allowed_url_hashes[j] == check_hash {
                    found = true;
                }
            }
            assert(found);
        }

        // Check H equals the publicly known value
        let stored_H: EmbeddedCurvePoint = self.storage.H.read();
        assert(H == stored_H);

        self.emit(SuccessEvent { sender, contract_address, id });
    }

    #[external("public")]
    fn check_urls_emit_event(
        sender: AztecAddress,
        contract_address: AztecAddress,
        id: Field,
        allowed_url_matches_hashes: [Field; 2],
    ) {
        // Check allowed_url_matches_hashes are in storage
        let allowed_url_hashes: [Field; 3] = self.storage.allowed_url_hashes.read();
        for i in 0..2 {
            let check_hash = allowed_url_matches_hashes[i];
            let mut found = false;
            for j in 0..3 {
                if allowed_url_hashes[j] == check_hash {
                    found = true;
                }
            }
            assert(found);
        }

        self.emit(SuccessEvent { sender, contract_address, id });
    }

    // Check H equals the publicly known value
    // do asserts on msgs
    // Opcode count: 208423, Total gates by opcodes: 696795, Circuit size: 711763
    #[external("private")]
    fn verify_comm(
        public_key_x: [u8; 32],
        public_key_y: [u8; 32],
        hash: [u8; 32],
        signature: [u8; 64],
        request_urls: [BoundedVec<u8, MAX_URL_LEN>; 2],
        allowed_urls: [BoundedVec<u8, MAX_URL_LEN>; 3],
        coms: BoundedVec<EmbeddedCurvePoint, MAX_COMMS>,
        rnds: BoundedVec<Field, MAX_COMMS>,
        msgs_chunks: BoundedVec<Field, MAX_COMMS>,
        msgs: BoundedVec<u8, MAX_MSGS_LEN>,
        H: EmbeddedCurvePoint, // G is fixed, H is fixed per business case
        id: Field,
    ) -> bool {
        let allowed_url_matches_hashes: [Field; 2] = verify_attestation_comm(
            public_key_x,
            public_key_y,
            hash,
            signature,
            request_urls,
            allowed_urls,
            coms,
            rnds,
            msgs_chunks,
            H,
        );
        // Check values
        // TODO what assert to do here?
        let _numbers_as_strings = get_values(msgs);

        BusinessProgram::at(self.address)
            .check_values_emit_event(
                self.msg_sender(),
                self.address,
                id,
                allowed_url_matches_hashes,
                H,
            )
            .enqueue(self.context);

        true
    }

    // Opcode count: 253113, Total gates by opcodes: 778309, Circuit size: 794646
    #[external("private")]
    fn verify_hash(
        public_key_x: [u8; 32],
        public_key_y: [u8; 32],
        hash: [u8; 32],
        signature: [u8; 64],
        request_urls: [BoundedVec<u8, MAX_URL_LEN>; 2],
        allowed_urls: [BoundedVec<u8, MAX_URL_LEN>; 3],
        data_hashes: [[u8; 32]; 2],
        plain_json_response_contents: [BoundedVec<u8, MAX_CONTENT_LEN>; 2],
        id: Field, // id for this attestation
    ) -> bool {
        let allowed_url_matches_hashes: [Field; 2] = verify_attestation_hashing(
            public_key_x,
            public_key_y,
            hash,
            signature,
            request_urls,
            allowed_urls,
            data_hashes,
            plain_json_response_contents,
        );

        for i in 0..2 {
            // Parse plaintext to JSON and obtain the custom values we want to check
            let (_base_ccy, _inst_id_code) = get_values2(plain_json_response_contents[i]);

            // TODO what to assert about base_ccy and inst_id_code
        }

        BusinessProgram::at(self.address)
            .check_urls_emit_event(self.msg_sender(), self.address, id, allowed_url_matches_hashes)
            .enqueue(self.context);

        true
    }

    #[contract_library_method]
    fn get_values(plaintext: BoundedVec<u8, MAX_MSGS_LEN>) -> [BoundedVec<u8, 20>; 6] {
        let json = JSON2055::parse_bounded_vec_json(plaintext);

        // for attestation_data_grumpkin.json
        // obtain .data[0], this is an array of 6 decimal numbers, which we'll read out as string
        let key0: BoundedVec<u8, 4> = BoundedVec::from_array("data".as_bytes());
        let data0 = json.get_array(key0).unwrap().get_array_from_array(0).unwrap();

        [
            data0.get_string_from_array(0).unwrap(),
            data0.get_string_from_array(1).unwrap(),
            data0.get_string_from_array(2).unwrap(),
            data0.get_string_from_array(3).unwrap(),
            data0.get_string_from_array(4).unwrap(),
            data0.get_string_from_array(5).unwrap(),
        ]
    }

    #[contract_library_method]
    fn get_values2(plaintext: BoundedVec<u8, MAX_CONTENT_LEN>) -> (u64, BoundedVec<u8, 10>) {
        let json = JSON2kb::parse_bounded_vec_json(plaintext);
        // for eth_hash.json
        let key0: BoundedVec<u8, 15> = BoundedVec::from_array("makerCommission".as_bytes());
        let maker_commission = json.get_number(key0).unwrap();

        let key1: BoundedVec<u8, 15> = BoundedVec::from_array("commissionRates".as_bytes());
        let commission_rate_obj: JSON2kb = json.get_object(key1).unwrap();

        let key2: BoundedVec<u8, 5> = BoundedVec::from_array("maker".as_bytes());
        let maker_rate_str: BoundedVec<u8, 10> =
            commission_rate_obj.get_string_from_path([key2]).unwrap();
        (maker_commission, maker_rate_str)
    }

}

Sorry it took so long to address and that the response is so boring.

If you'll struggle with updating LMK by tagging me. Or if the issue persists feel free to reopen.

Thanks for the patience!

[ewynx]

Hi @benesjan, thanks for your answer!

I upgraded to 4.0.0-devnet.2-patch.1 and actually still see that strange generation of the TypeScript file with all the prefixes __aztec_nr_internals__. (I don't know if I'm still running into the same issue regarding the vks because the script fails sooner now, seemingly due to another issue on Mac Intel (related Discord question here)).
Would you expect this issue to be solved in this devnet version? Or should I wait it out for another upgrade?

[benesjan]

During the contract build pipeline the __aztec_nr_internals__ are stripped from the function names in the contract artifact after the vks are generated. So the fact that you see them means that they didn't get stripped from the artifact. Given that the stripping step is simple it's likely that this is indeed caused by a failure to generate vks.

It's likely there is some issue with version management.

Re-opened this and will re-investigate.

[benesjan]

@ewynx I see that you are contributor to the noir_json_parser project. I think it was implemented with an old Noir version in mind where accessing private functionality was only a warning while in newer Noir it's an error:

So please update that dep first.

Also I see that here you use an older Aztec.nr version. I did the migration in this PR so please merge that, update the noir_json_parser, try to compile it all again and LMK if it works 😊

[ewynx]

Hi @benesjan thanks for looking into it again! I made those changes to my local version of json parser and have been using that one. (Otherwise I indeed get an error) Same for the aztec.nr version, I updated that before testing - I just didn't push any changes.

I think this issue might be related or the same #20872 and it uses an Aztec repo.

[benesjan]

Can you push it all so that I can try to reproduce?

[ewynx]

@benesjan I pushed the code I'm working with to https://github.com/primus-labs/zktls-verification-noir/tree/upgrade_4.0.0-devnet.2-patch.1

Maybe good to repeat here, like in the potentially related issue I mentioned above (#20872) when I do

cd example/real_business_program
aztec compile; echo "Exit: $?"

It exits unsuccessfully. This is the log:

warning: unused variable base_ccy
    ┌─ src/main.nr:2:1
    │
  2 │ #[aztec]
    │ -------- While running this function attribute
    ·
177 │             let (base_ccy, inst_id_code) = get_values2(plain_json_response_contents[i]);
    │                  -------- unused variable
    │

warning: unused variable inst_id_code
    ┌─ src/main.nr:2:1
    │
  2 │ #[aztec]
    │ -------- While running this function attribute
    ·
177 │             let (base_ccy, inst_id_code) = get_values2(plain_json_response_contents[i]);
    │                            ------------ unused variable
    │

Postprocessing contract...
Scheme is: ultra_honk, num threads: 12 (mem: 7.50 MiB)
Found 1 contract artifact(s) to process (mem: 7.57 MiB)
Transpiling: ./target/real_business_program-BusinessProgram.json -> ./target/real_business_program-BusinessProgram.json (mem: 7.57 MiB)
Transpiled: ./target/real_business_program-BusinessProgram.json -> ./target/real_business_program-BusinessProgram.json (mem: 1144.38 MiB)
Generating verification keys for functions in real_business_program-BusinessProgram.json (mem: 1144.38 MiB)
Cache directory: xx/.bb/4.0.0-devnet.2-patch.1/vk_cache (mem: 1144.38 MiB)
Generating verification key: c9c41b9f574424c56803d9d9b038cf17c16715f6ff025f4539681d280c1e6e47 (mem: 1164.85 MiB)
ChonkComputeVk - deriving MegaVerificationKey for circuit '__aztec_nr_internals__verify_hash' (mem: 1164.87 MiB)
Generating verification key: a1a20bcf95cbbd598762ff3b7ac0b612a2f88a3042a4e1adf36e79a2b6a5ca9c (mem: 1504.08 MiB)
ChonkComputeVk - deriving MegaVerificationKey for circuit '__aztec_nr_internals__verify_comm' (mem: 1506.15 MiB)
Exit: 1

In the artifacts/BusinessProgram.ts like I mentioned it still contains the prefixes __aztec_nr_internals__, maybe this is because aztec compile doesn't complete? (But it errors or gets killed silently)

[benesjan]

Tried reproducing on my Macbook Air M1 with 16 GB of RAM and all happily compiles:

Postprocessing contract...
Scheme is: ultra_honk, num threads: 32 (mem: 6.06 MiB)
Found 1 contract artifact(s) to process (mem: 7.59 MiB)
Transpiling: ./target/real_business_program-BusinessProgram.json -> ./target/real_business_program-BusinessProgram.json (mem: 7.59 MiB)
Transpiled: ./target/real_business_program-BusinessProgram.json -> ./target/real_business_program-BusinessProgram.json (mem: 808.12 MiB)
Generating verification keys for functions in real_business_program-BusinessProgram.json (mem: 808.12 MiB)
Cache directory: /mnt/user-data/jan/.bb/4.0.0-devnet.2-patch.1/vk_cache (mem: 808.12 MiB)
Generating verification key: c9c41b9f574424c56803d9d9b038cf17c16715f6ff025f4539681d280c1e6e47 (mem: 985.63 MiB)
ChonkComputeVk - deriving MegaVerificationKey for circuit '__aztec_nr_internals__verify_hash' (mem: 985.63 MiB)
Generating verification key: a1a20bcf95cbbd598762ff3b7ac0b612a2f88a3042a4e1adf36e79a2b6a5ca9c (mem: 1261.63 MiB)
ChonkComputeVk - deriving MegaVerificationKey for circuit '__aztec_nr_internals__verify_comm' (mem: 1322.38 MiB)
ChonkComputeVk - VK derived, size: 4064 bytes (mem: 8833.50 MiB)
ChonkComputeVk - VK derived, size: 4064 bytes (mem: 8833.50 MiB)
Successfully processed: ./target/real_business_program-BusinessProgram.json -> ./target/real_business_program-BusinessProgram.json (mem: 8833.50 MiB)
Contract postprocessing complete! (mem: 8833.50 MiB)
Compilation complete!
% 

@ewynx how much RAM does your Mac have?

If it's only 8 GB it's quite likely we have not sufficiently tested it on that configuration and the vk generation errors out because of lack of memory.

[ewynx]

My laptop has 32GB of RAM! Do you know if anyone else with Intel chip has been able to compile?

[zac-williamson]

[EDIT] ignore below. I was using the wrong branch. leaving for posterity.

When I try to compile I do get errors, but at the transpiler stage stage. I get:

Postprocessing contract...
Scheme is: ultra_honk, num threads: 12 (mem: 14.44 MiB)
Found 1 contract artifact(s) to process (mem: 14.52 MiB)
Transpiling: ./target/real_business_program-BusinessProgram.json -> ./target/real_business_program-BusinessProgram.json (mem: 14.52 MiB)

thread '<unnamed>' panicked at src/transpile.rs:781:9:
Transpiler expects ForeignCall::CHECKNULLIFIEREXISTS to have 1 destinations and 1 input, got 1 and 2
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5, aborting