Skip to content

respect catalog selections#17

Open
pnadolny13 wants to merge 2 commits intoPathlight:masterfrom
pnadolny13:respect_catalog
Open

respect catalog selections#17
pnadolny13 wants to merge 2 commits intoPathlight:masterfrom
pnadolny13:respect_catalog

Conversation

@pnadolny13
Copy link

@pnadolny13 pnadolny13 commented May 18, 2023

Someone in the Meltano community posted in slack about how this tap didnt respect their catalog selection criteria, this PR should add support for enabling/disabling streams and properties using meltano.

I dont have access to an account to test but I did validate that deselecting all streams passed with logs saying that they were skipped, then enabling one stream caused it to raise a 401 error due to invalid credentials.

@vorpal56 @kyuhanpathlight I know this is a fork of the original tap but it looks like this repo has been the most active and has seen the most recent commits so thought it was appropriate to create the PR here 😄 , let me know what you think!

@walkabout21
Copy link

@pnadolny13 @kyuhanpathlight @vivqu
I believe this merge will fix an issue I am having with the purecloud tap. I have experimented with different loaders including target-csv and it appears that because I am unable to select specific streams and the purecloud tap schema definitions don't match the stream output, any loader that expects values matching the schema will fail.
For example, the users schema definition is

{
    "type": "object",
    "properties": {
        "email": {
            "type": "string"
        },
        "id": {
            "type": "string"
        },
        "name": {
            "type": "string"
        },
        "username": {
            "type": "string"
        }
    }
}

But the csv loader fails with

ValueError: dict contains fields not in fieldnames: 'selfUri', 'division__id', 'division__selfUri', 'businessUnit__id', 'businessUnit__selfUri' 

Ideally the schema would match the output and selecting specific streams would work, but just respecting catalog selections would likely solve the immediate issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants