NexsonParseException when indexing study

[jar398]

Most studies seem to work ok, but indexing some of them yields this error:

Indexing study pg_2739 from http://devapi.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2739.json
Calling "http://127.0.0.1:7478/db/data/ext/studies/graphdb/index_study" with data="{'url': 'http://devapi.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2739.json'}"

Indexing failed for http://devapi.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2739.json

500 Server Error: Internal Server Error

{
"exception" : "NexsonParseException",
"fullname" : "org.opentree.nexson.io.NexsonParseException",
"stacktrace" : [ "org.opentree.nexson.io.NexsonTree.parseNexson(NexsonTree.java:249)", "org.opentree.nexson.io.NexsonTree.(NexsonTree.java:85)", "org.opentree.nexson.io.NexsonSource.parseNexson(NexsonSource.java:191)", "org.opentree.nexson.io.NexsonSource.(NexsonSource.java:52)", "org.opentree.oti.plugins.studies.readRemoteNexson(studies.java:285)", "org.opentree.oti.plugins.studies.index_study(studies.java:217)", "java.lang.reflect.Method.invoke(Method.java:497)", "org.neo4j.server.plugins.PluginMethod.invoke(PluginMethod.java:57)", "org.neo4j.server.plugins.PluginManager.invoke(PluginManager.java:168)", "org.neo4j.server.rest.web.ExtensionService.invokeGraphDatabaseExtension(ExtensionService.java:300)", "org.neo4j.server.rest.web.ExtensionService.invokeGraphDatabaseExtension(ExtensionService.java:122)", "java.lang.reflect.Method.invoke(Method.java:497)", "org.neo4j.server.rest.security.SecurityFilter.doFilter(SecurityFilter.java:112)" ]
}

[jar398]

In case this helps in any way:
Development: 2900 studies, of which 8 fail.
Production (different phylesystem): 3055 studies, of which 562 fail.

I'd say this is a pretty high priority bug.

[jimallman]

Agreed, esp. since the error message is so vague. For what it's worth, the example study (pg_2739.json) appears to be valid JSON. If we have a list of failed study ids, I can look for common elements.

[jar398]

Here are a few of them (there are 556 in all)

Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2405.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2404.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_14.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/ot_208.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/ot_209.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/ot_200.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/ot_201.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/ot_202.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/ot_203.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/ot_204.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/ot_205.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/ot_206.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/ot_207.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_1988.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_1981.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_386.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_385.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2618.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_598.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_599.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_596.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_597.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_594.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_595.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2797.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2796.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2792.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2791.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2790.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2422.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2883.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2882.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2881.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2880.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2886.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2885.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2884.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2889.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2888.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_901.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_1144.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_754.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2843.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2842.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2849.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_1338.json
Indexing failed for http://api.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_1336.json

[jimallman]

Thanks. I'll try testing some of these in isolation, since we're not getting the NexsonParseException.errorMessage that actually explains the problem.

[josephwb]

I tried playing with this (on a now deleted branch of ot-base) but the results returned seem to reflect old code (i.e. not any of the changes made). Apparently oti does not register a change in ot-base, and so is not recompiled.

[jimallman]

OK, now that we're getting clearer error messages (thanks @chinchliff), it's clear that all the studies that failed to index suffer from the same problem

The specified root node {NODE_ID} is different from the observed 
root of the tree in the NexSON object hierarchy. This is nonsensical.

Here's the test that's failing. It seems we're either submitting bad trees, or the surrounding test needs a closer look.

[jimallman]

Here are the failing studies in my latest indexoti attempt, all with the root-node error above:

http://devapi.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2926.json
http://devapi.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2739.json
http://devapi.opentreeoflife.org/phylesystem/v1/../default/v1/study/ot_29.json
http://devapi.opentreeoflife.org/phylesystem/v1/../default/v1/study/ot_19.json
http://devapi.opentreeoflife.org/phylesystem/v1/../default/v1/study/ot_38.json
http://devapi.opentreeoflife.org/phylesystem/v1/../default/v1/study/ot_39.json
http://devapi.opentreeoflife.org/phylesystem/v1/../default/v1/study/ot_32.json
http://devapi.opentreeoflife.org/phylesystem/v1/../default/v1/study/ot_33.json

[chinchliff]

Joseph and I did make a small adjustment to that test that should avoid a
potentially sneaky java bug. If the test-fix branch is what is being used
on devapi, then the actual code for the test (which is very basic) is here:

https://github.com/OpenTreeOfLife/ot-base/blob/test-fix/src/main/java/org/opentree/nexson/io/NexsonTree.java#L248

On Wed, May 20, 2015 at 12:30 PM Jim Allman notifications@github.com
wrote:

Here are the failing studies in my latest indexoti attempt, all with the
root-node error above:

http://devapi.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2926.json

http://devapi.opentreeoflife.org/phylesystem/v1/../default/v1/study/pg_2739.json

http://devapi.opentreeoflife.org/phylesystem/v1/../default/v1/study/ot_29.json

http://devapi.opentreeoflife.org/phylesystem/v1/../default/v1/study/ot_19.json

http://devapi.opentreeoflife.org/phylesystem/v1/../default/v1/study/ot_38.json

http://devapi.opentreeoflife.org/phylesystem/v1/../default/v1/study/ot_39.json

http://devapi.opentreeoflife.org/phylesystem/v1/../default/v1/study/ot_32.json

http://devapi.opentreeoflife.org/phylesystem/v1/../default/v1/study/ot_33.json

—
Reply to this email directly or view it on GitHub
#40 (comment).

[jar398]

Subproblem of #34