Ingest bones sources #577

kaseyLee123 · 2025-02-08T16:15:19Z

Short description: ingest sources in photometry from the BONES archive

Link to relevant issue: Closes #544 (sources only)

For data ingests:

includes script used for ingest
includes modified JSON files
Add new tests
Update the Versions table

script is now running but just change logic so if source does not exist then ingest

Kelle's changes

…IMPLE-db into ingestBonesSources

kelle · 2025-02-14T17:08:46Z

If you run the script with DB_Save = True, how many JSON files get modified?

kelle · 2025-02-14T17:16:06Z

Tests aren't passing because of changes made to astrodb_utils. I'll work on that.

kelle · 2025-02-14T18:00:08Z

I've run the script and saved the JSON files and we have two problems:

its not recognizing the unicode minus sign as a minus sign and is adding it as an other name.
{ "other_name": "WISEA J000430.66-260402.3" }, { "other_name": "WISEA J000430.66\u2212260402.3" }
Also, I noticed it confused LHS 292 with LHS 2924.

Both of these are tricky and I will think on them.

kelle · 2025-02-14T21:24:02Z

scripts/ingests/bones_archive/ingest_bones_sources.py

+            ingest_publication(
+                db,
+                bibcode = ads,
+                reference=adsRef


Suggested change

reference=adsRef

This is the line which is giving the inconsistent formatting for the references.

scripts/ingests/bones_archive/ingest_bones_sources.py

kelle · 2025-02-14T22:10:36Z

Ok, I fixed the minus sign problem and got around the LHS 292/2924 confusion but I found a new and BIG bug. The CSV file we're reading from got messed up somehow. The discovery references are linked to the wrong sources.

Here are three rows from the CSV file y'all made:

LHS 2090,135.0981088,21.83469481,sdM7,Lodieu et al. 2017,13.192,13.022,0.014,0.016,17.051,18.826,15.793,0.003,0.016,0.005,,,,,,,,,,,,,,,,,14.065,13.575,13.396,0.030,0.033,0.034,,,16.307,15.616,15.319,,,0.003,0.006,0.000,,,,,,,,,,,,
LP 426-61,137.2654234,19.69564333,sdT4.5,Burningham et al. 2010,18.059,17.096,0.130,0.182,,,,,,,,,,,,,19.950,18.464,,18.793,,0.145,0.064,,0.122,,,,,,,,,,21.483,21.060,20.250,,,0.047,0.109,0.174,,,,,,,,,,,,
ULAS J092605.47+083516.9,141.522819,8.588039,sdM8.5,Lodieu et al. 2017,16.859,16.866,0.047,0.144,20.892,21.238,19.597,0.015,0.178,0.073,,,,,,,18.156,17.645,,17.487,17.208,0.030,0.029,,0.075,0.077,,,,,,,,21.531,20.090,19.246,18.870,,0.040,0.014,0.026,0.029,,,,,,,,,,,,

and those three rows from the Bones Archive

LHS 2090	133.9529947	36.6004475	sdM7	Rojas-Ayala et al. 2012													
LP 426-61	135.0981088	21.8346948	sdM7	Lodieu et al. 2017	3088.000		-0.100	Marfil et al. 2021									
ULAS J092605.47+083516.9	137.2654234	19.6956433	sdT4.5	Burningham et al. 2010

Please work on making a correct CSV file and re-running the updated script.

…IMPLE-db into ingestBonesSources

kelle

Seems like some investigation needs to go into the find_source_in_db function.

kelle · 2025-03-05T16:40:23Z

data/source/2mass_j06453153-6646120.json

+        {
+            "other_name": "WISEA J071121.36-573634.2"
        }
    ],


Uh oh. This doesn't look like the code found the right source here.

kelle · 2025-03-05T16:42:56Z

data/source/gj_1111.json

+        {
+            "other_name": "LP  258-28"
+        },
        {


Wrong match. LP 258-28 is LHS 2023. https://simbad.u-strasbg.fr/simbad/sim-id?Ident=LP++258-28&NbIdent=1&Radius=2&Radius.unit=arcmin&submit=submit+id

kelle · 2025-03-05T16:43:09Z

data/source/lehpm_2-59.json

+        {
+            "other_name": "2MASS J04524567-3608412"
+        },


wrong match. :(

kelle · 2025-03-05T16:43:33Z

data/source/lp_368-128.json

+        {
+            "other_name": "LHS 2090"
+        },
        {
            "other_name": "LP 368-128"
+        },
+        {
+            "other_name": "LP 426-61"
        }


should only have one LP name so something is off.

kelle · 2025-03-05T16:43:54Z

data/source/sdss_j075054.74+445418.7.json

            "other_name": "SDSS J075054.74+445418.7"
+        },
+        {
+            "other_name": "ULAS J075335.23+200622.4"
        }


wrong match.

kelle · 2025-03-05T16:44:38Z

data/source/ulas_j033350.84+001406.1.json

        {
            "other_name": "2MASS J03335076+0014062"
        },
+        {
+            "other_name": "2MASS J03361338+0010129"
+        },


maybe wrong match? The numbers usually agree really well and these are just a bit more off than usual.

kelle · 2025-03-05T16:44:51Z

data/source/ulas_j092605.47+083517.0.json

+        {
+            "other_name": "SDSS J093244.46+011259.8"
+        },
+        {
+            "other_name": "ULAS J092605.47+083516.9"
+        },


wrong match.

kelle · 2025-03-05T16:45:04Z

data/source/ulas_j095047.28+011734.3.json

+        {
+            "other_name": "SDSS J101200.28+204611.6"
+        },
        {
            "other_name": "ULAS J095047.28+011734.3"


wrong match.

kelle · 2025-03-05T16:46:08Z

data/source/wisea_j135501.90−825838.9.json

There shouldn't be \u2212. Those should be replaced with hyphens: "-".

This reverts commit 540cc77.

kelle · 2025-04-02T21:20:20Z

Hunter has fixed a bug in the sheet - please re-download the CSV file: https://docs.google.com/spreadsheets/d/1b9SUUshady_efTJaWEXiy4g-1n-JOzG_Mcy6j1CFE8k/edit?gid=65788592#gid=65788592

kelle

This looks great! Amazing progress!

I only see two problems

Burg24 publication has no data
I don't see the UNCOVER-bd-1 source.

When I run the script, a new uncover-bd-1.json file is made and the Burg24 publication has data in it so I'm not sure what's going on.

kelle · 2025-04-09T22:20:05Z

data/reference/Publications.json

+        "reference": "Burg24",
+        "bibcode": "",
+        "doi": "",
+        "description": ""
+    },


Hmmm, this should have some things filled in.

kelle · 2025-04-09T22:23:29Z

scripts/ingests/bones_archive/ingest_bones_sources.py

+no_sources = 0
+inside_if = 0
+


Suggested change

no_sources = 0

inside_if = 0

These variables are never used.

kelle · 2025-04-09T22:24:14Z

scripts/ingests/bones_archive/ingest_bones_sources.py

+    if match is None:
+        match = find_source_in_db(
+            db,
+            bones_name,
+            ra=source["RA"],
+            dec=source["DEC"],
+        )


Suggested change

if match is None:

match = find_source_in_db(

db,

bones_name,

ra=source["RA"],

dec=source["DEC"],

)

This code is never called

kaseyLee123 and others added 9 commits January 29, 2025 13:35

ingest source script

4b01493

ingest bones script

790a056

script is now running but just change logic so if source does not exist then ingest

tried to fix logic but error w utils

c358b2c

ingest source

bb51fa6

finished script

e78039f

got it to work for me

efe9be0

Merge pull request #1 from kelle/ingestBonesSources

fdd4fa7

Kelle's changes

implemented dr cruz's changes

29a3d4d

Merge branch 'ingestBonesSources' of https://github.com/kaseyLee123/S…

2a12ce6

…IMPLE-db into ingestBonesSources

kelle added 3 commits February 14, 2025 12:20

drop 3.10

0f5e9ec

recommend python 3.13

b5cb174

fix astrodb_utils imports

881479c

untested unicode fix

50a4dad

kelle reviewed Feb 14, 2025

View reviewed changes

working script

05cefef

kelle reviewed Feb 14, 2025

View reviewed changes

scripts/ingests/bones_archive/ingest_bones_sources.py Show resolved Hide resolved

kelle reviewed Feb 14, 2025

View reviewed changes

scripts/ingests/bones_archive/ingest_bones_sources.py Show resolved Hide resolved

kaseyLee123 and others added 9 commits February 17, 2025 19:57

use dr cruz's changes

ff24a90

updaes

8b71472

upadtes

7ee3b69

csv fix

a7a36f8

updates

e2961c6

set column names

2675e84

delete unused files

4a19cac

Update ingest_bones_sources.py

366db50

Merge branch 'ingestBonesSources' of https://github.com/kaseyLee123/S…

bdc0b33

…IMPLE-db into ingestBonesSources

updated post pytest

489215d

kelle requested changes Mar 5, 2025

View reviewed changes

updated fuzzy search, names not properly updating..

540cc77

kaseyLee123 force-pushed the ingestBonesSources branch from b3997c3 to 540cc77 Compare March 26, 2025 16:27

kaseyLee123 added 2 commits March 26, 2025 12:35

Revert "updated fuzzy search, names not properly updating.."

7811b57

This reverts commit 540cc77.

added one more replace

404dd51

updated json files

6b8870e

kelle requested changes Apr 9, 2025

View reviewed changes

kelle reviewed Apr 9, 2025

View reviewed changes

kelle closed this Apr 14, 2025

Ingest bones sources #577

Ingest bones sources #577

Uh oh!

Conversation

kaseyLee123 commented Feb 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kelle commented Feb 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kelle commented Feb 14, 2025

Uh oh!

kelle commented Feb 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kelle commented Feb 14, 2025

Uh oh!

kelle left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kelle commented Apr 2, 2025

Uh oh!

kelle left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kaseyLee123 commented Feb 8, 2025 •

edited

Loading

kelle commented Feb 14, 2025 •

edited

Loading