Xamarin.iOS build works locally but fails in DevOps pipelines

[mark-silver-puregym]

Before reporting a bug:

• I have read the project readme and followed all instructions.
• I have searched this project for potentially duplicate issues (even closed ones).

Describe the bug
As the title says, have been able to get this all building successfully locally on MacOS using Jetbrains Rider as the IDE for our Xamarin.iOS app (last version to be shipped before we switch to MAUI). But when running through our DevOps pipelines it seems to be failing. I've checked that main software versions in pipelines match my development environment and they do:

Hosted Agent - macOS-14
https://github.com/actions/runner-images/blob/main/images/macos/macos-14-Readme.md

macOS Sonoma - match
Xcode 15.4 - match
Microsoft (R) Build Engine version 16.10.1 for Mono (16.10.1.51301) - match
Xamarin.iOS.framework - 16.4.0.23 - match

It appears the build is unable to find GoogleAppMeasurement framework and other symbols APMAnalyticsConfiguration. It looks like a linking issues, have tried using PreserveAllMembers on the Firebase libraries. Any idea what the cause is here?

Built using the following yaml

  - task: MSBuild@1
      displayName: 'Build iOS'
    inputs:
      solution: 'src/App.Member.sln'
      configuration: '${{ parameters.buildConfiguration }}'
      msbuildArguments: /p:Platform="iPhone" /p:CodesignKey="$(APPLE_CERTIFICATE_SIGNING_IDENTITY)" /p:CodesignProvision="$(APPLE_PROV_PROFILE_UUID)"'

Expected behavior
Successful build in DevOps pipelines

Error messages
Output from DevOps build log

Undefined symbols for architecture arm64:
"_APMAnalyticsConfiguration", referenced from:
+[FIRAnalytics startWithConfiguration:options:] in FirebaseAnalytics10
"_APMAppMeasurementOriginFirebase", referenced from:
+[FIRAnalytics startWithConfiguration:options:] in FirebaseAnalytics10
"_APMConsentSettings3P", referenced from:
+[FIRAnalytics setConsent:] in FirebaseAnalytics10
"_APMFormattedEventName", referenced from:
+[FIRAnalytics logEventWithOrigin:name:parameters:] in FirebaseAnalytics10
"_APMFormattedUserPropertyName", referenced from:
+[FIRAnalytics setUserPropertyString:forName:] in FirebaseAnalytics10
"_APMIsAnalyticsCollectionDeactivated", referenced from:
+[FIRAnalytics startWithConfiguration:options:] in FirebaseAnalytics10
"_APMIsAnalyticsCollectionEnabled", referenced from:
+[FIRAnalytics startWithConfiguration:options:] in FirebaseAnalytics10
"_APMIsValidTransactionDeviceVerification", referenced from:
+[FIRAnalytics verifyTransactionWithDeviceVerification:UUID:nonce:] in FirebaseAnalytics10
"_APMMonitorLogTagOptionKey", referenced from:
+[FIRAnalytics startWithConfiguration:options:] in FirebaseAnalytics10
"_APMUserDataFieldEmailAddress", referenced from:
+[FIRAnalytics initiateOnDeviceConversionMeasurementWithEmailAddress:] in FirebaseAnalytics10
"_APMUserDataFieldHashedEmailAddress", referenced from:

(_CompileToNative target) ->
clang : error : linker command failed with exit code 1 (use -v to see invocation) [/Users/runner/work/1/s/src/App.Member.iOS/App.Member.iOS.csproj]
error MT5209 : Native linking error : warning: ignoring duplicate libraries: '-lc++', '-lz' [/Users/runner/work/1/s/src/App.Member.iOS/App.Member.iOS.csproj]
error MT5209 : Native linking error : warning: mixed ObjC ABI, /Users/runner/work/1/s/src/App.Member.iOS/obj/iPhone/Release/mtouch-cache/arm64/Firebase.Analytics.dll.o compiled without category class properties [/Users/runner/work/1/s/src/App.Member.iOS/App.Member.iOS.csproj]
error MT5209 : Native linking error : warning: mixed ObjC ABI, /Library/Frameworks/Xamarin.iOS.framework/Versions/16.4.0.23/SDKs/MonoTouch.iphoneos.sdk/lib/libxamarin.a[arm64]3 compiled with category class properties [/Users/runner/work/1/s/src/App.Member.iOS/App.Member.iOS.csproj]
error MT5209 : Native linking error : warning: mixed ObjC ABI, /Library/Frameworks/Xamarin.iOS.framework/Versions/16.4.0.23/SDKs/MonoTouch.iphoneos.sdk/lib/libxamarin.a[arm64]4 compiled with category class properties [/Users/runner/work/1/s/src/App.Member.iOS/App.Member.iOS.csproj]
error MT5209 : Native linking error : warning: mixed ObjC ABI, /Library/Frameworks/Xamarin.iOS.framework/Versions/16.4.0.23/SDKs/MonoTouch.iphoneos.sdk/lib/libxamarin.a[arm64]5 compiled with category class properties [/Users/runner/work/1/s/src/App.Member.iOS/App.Member.iOS.csproj]
error MT5209 : Native linking error : warning: mixed ObjC ABI, /Library/Frameworks/Xamarin.iOS.framework/Versions/16.4.0.23/SDKs/MonoTouch.iphoneos.sdk/lib/libxamarin.a[arm64]6 compiled with category class properties [/Users/runner/work/1/s/src/App.Member.iOS/App.Member.iOS.csproj]
error MT5209 : Native linking error : warning: mixed ObjC ABI, /Library/Frameworks/Xamarin.iOS.framework/Versions/16.4.0.23/SDKs/MonoTouch.iphoneos.sdk/lib/libxamarin.a[arm64]7 compiled with category class properties [/Users/runner/work/1/s/src/App.Member.iOS/App.Member.iOS.csproj]
error MT5209 : Native linking error : warning: mixed ObjC ABI, /Library/Frameworks/Xamarin.iOS.framework/Versions/16.4.0.23/SDKs/MonoTouch.iphoneos.sdk/lib/libxamarin.a[arm64]8 compiled with category class properties [/Users/runner/work/1/s/src/App.Member.iOS/App.Member.iOS.csproj]
error MT5209 : Native linking error : warning: mixed ObjC ABI, /Library/Frameworks/Xamarin.iOS.framework/Versions/16.4.0.23/SDKs/MonoTouch.iphoneos.sdk/lib/libxamarin.a[arm64]9 compiled with category class properties [/Users/runner/work/1/s/src/App.Member.iOS/App.Member.iOS.csproj]
error MT5209 : Native linking error : warning: mixed ObjC ABI, /Library/Frameworks/Xamarin.iOS.framework/Versions/16.4.0.23/SDKs/MonoTouch.iphoneos.sdk/lib/libxamarin.a[arm64]12 compiled with category class properties [/Users/runner/work/1/s/src/App.Member.iOS/App.Member.iOS.csproj]
error MT5209 : Native linking error : warning: mixed ObjC ABI, /Library/Frameworks/Xamarin.iOS.framework/Versions/16.4.0.23/SDKs/MonoTouch.iphoneos.sdk/lib/libxamarin.a[arm64]15 compiled with category class properties [/Users/runner/work/1/s/src/App.Member.iOS/App.Member.iOS.csproj]
error MT5209 : Native linking error : warning: mixed ObjC ABI, /Library/Frameworks/Xamarin.iOS.framework/Versions/16.4.0.23/SDKs/MonoTouch.iphoneos.sdk/lib/libxamarin.a[arm64]16 compiled with category class properties [/Users/runner/work/1/s/src/App.Member.iOS/App.Member.iOS.csproj]
error MT5209 : Native linking error : warning: Could not find or use auto-linked framework 'GoogleAppMeasurement': framework 'GoogleAppMeasurement' not found [/Users/runner/work/1/s/src/App.Member.iOS/App.Member.iOS.csproj]
MTOUCH : error MT5210: Native linking failed, undefined symbol: _APMAnalyticsConfiguration. Please verify that all the necessary frameworks have been referenced and native libraries are properly linked in. [/Users/runner/work/1/s/src/App.Member.iOS/App.Member.iOS.csproj]
MTOUCH : error MT5210: Native linking failed, undefined symbol: _APMAppMeasurementOriginFirebase. Please verify that all the necessary frameworks have been referenced and native libraries are properly linked in. [/Users/runner/work/1/s/src/App.Member.iOS/App.Member.iOS.csproj]
MTOUCH : error MT5210: Native linking failed, undefined symbol: _APMConsentSettings3P. Please verify that all the necessary frameworks have been referenced and native libraries are properly linked in. [/Users/runner/work/1/s/src/App.Member.iOS/App.Member.iOS.csproj]
MTOUCH : error MT5210: Native linking failed, undefined symbol: _APMFormattedEventName. Please verify that all the necessary frameworks have been referenced and native libraries are properly linked in. [/Users/runner/work/1/s/src/App.Member.iOS/App.Member.iOS.csproj]

Environment (please complete the following information):

• OS: MacOS Sonoma 14.6.1
• IDE (if applicable): Jetbrains Rider (builds successfully)
• .NET version

.csproj

  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|iPhone' ">
    <DebugType>none</DebugType>
    <Optimize>true</Optimize>
    <OutputPath>bin\iPhone\Release</OutputPath>
    <ErrorReport>prompt</ErrorReport>
    <WarningLevel>4</WarningLevel>
    <MtouchArch>ARM64</MtouchArch>
    <ConsolePause>false</ConsolePause>
    <CodesignKey>iPhone Developer</CodesignKey>
    <CodesignEntitlements>Entitlements.Release.plist</CodesignEntitlements>
    <BuildIpa>true</BuildIpa>
    <CodesignProvision>App Member Profile</CodesignProvision>
    <DefineConstants>__IOS__;__MOBILE__;__UNIFIED__;</DefineConstants>
    <MtouchI18n>cjk,mideast,other,west</MtouchI18n>
    <MtouchHttpClientHandler>NSUrlSessionHandler</MtouchHttpClientHandler>
    <MtouchLink>SdkOnly</MtouchLink>
    <MtouchExtraArgs>--gcc_flags -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/iphoneos</MtouchExtraArgs>
    <MSBuildWarningsAsMessages>MSB3277</MSBuildWarningsAsMessages>
    <AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="AdamE.Firebase.iOS.Analytics">
      <Version>11.8.0</Version>
    </PackageReference>
    <PackageReference Include="AdamE.Firebase.iOS.Core">
      <Version>11.8.0</Version>
    </PackageReference>
    <PackageReference Include="AdamE.Firebase.iOS.Crashlytics">
      <Version>11.8.0</Version>
    </PackageReference>
...

[AdamEssenmacher]

I've seen the error message you're encountering reported probably dozens of times. You're right that it's a "linker" issue, but it's the native linker, not ILLink (so PreserveAllMembers won't help).

Native linking is failing because the compiler can't find the code it's looking for. The missing symbols are included in an xcframework that is downloaded and extracted during the build by XamarinBuildDownload (XBD). If this process fails, or of the XBD cache gets in a bad state, then you get this error message.

[mark-silver-puregym]

Thanks @AdamEssenmacher, So you'd recommend trying to delete the XBD cache on the agent?

[AdamEssenmacher]

It's been a while since I've used a Microsoft-hosted agent, but I think it's a fresh image each time? So I don't know if you would have a persistent cache that actually needs to be cleared.

[mark-silver-puregym]

That was my thinking too, not sure if that's going to be the issue. I also noticed that the hosted agents use named Xcode versions. Which could have been related also but doesn't seem to be the issue:

On Legacy Xamarin.iOS, FirebaseAnalytics 10.17+ requires that additional mtouch arguments be added to the iOS project:

For iPhone:
--gcc_flags -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/iphoneos

For simulator:
--gcc_flags -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/iphonesimulator/

You may need to adjust the XCode location if it's not default -

did need to change that to Xcode_15.4.app

[mark-silver-puregym]

Clearing the cache didn't help