Implemented import API endpoint by ErikBjare · Pull Request #41 · ActivityWatch/aw-server

ErikBjare · 2018-04-01T23:33:17Z

Stuff left to do:

REST API endpoint
Proper testing
- Try importing an old valid export
- Import an export and compare
~~Implement an import dialog in the web UI~~ (not in this PR)

ErikBjare · 2018-09-10T11:01:30Z

Just wrote https://github.com/ErikBjare/smartertime2activitywatch that should convert a SmarterTime export into an importable ActivityWatch bucket.

ErikBjare · 2018-09-10T18:39:52Z

With the latest commit I was able to import a SmarterTime export through the API!

johan-bjareholt · 2019-02-26T17:54:55Z

This does not follow the format used in /api/0/export (which is already on master) where the format is {"bucketname": {bucketdata}} while this expects {"buckets": [{bucket_A}, [bucket_B]]}

johan-bjareholt · 2019-02-26T19:27:27Z

I fixed it locally and imported my current database on my desktop. For my export of 94mb it took 19minutes to import in aw-server with peewee. In aw-server-rust it took less than 2 minutes lol.

johan-bjareholt · 2019-02-26T20:40:20Z

As we have a export schema in aw-core I decided to break backwards compatibility and use that everywhere.
The export button in the web-ui is still worthless anyway as it doesn't care about metadata and only gets all the events which is stupid.
It is also now no longer possible to import from aw-server-rust as the datastructure has changed.

ErikBjare · 2019-02-26T20:41:04Z

I think {"buckets": [{bucket_A}, [bucket_B]]} may be a better format though. What do you think?

I thought I fixed the export button?

johan-bjareholt · 2019-02-26T20:42:49Z

@ErikBjare Pretty much everything is inconsistent. Export button doesn't have metadata, we use different formats in /0/api/export and /0/api/import. I now try to use {"buckets":[{bucket1}, {bucket2}]} everywhere.

ErikBjare · 2019-02-26T21:01:14Z

What is differing between the formats? Perhaps we should start versioning the exports? (using a version of 0 until stabilized)

Perhaps the best is to have the export-all format be {"buckets": {bucket1.id: bucket1, ...}}, the current export-all format just seems to be [bucket1, bucket2, ...].

Also, apparently I had almost fixed the export button: https://github.com/ActivityWatch/aw-webui/blob/eb56c6194e65ce73ab29b868ea8121d936ba5144/src/views/Buckets.vue#L33

Anyway, the most urgent issue should be to implement the import view (both for imports and export of the whole-instance and specific buckets).

johan-bjareholt · 2019-02-26T21:10:42Z

Perhaps the best is to have the export-all format be {"buckets": {bucket1.id: bucket1, ...}}, the current export-all format just seems to be [bucket1, bucket2, ...].

The current format is {"buckets": [{bucket1}, {bucket2}]}

The export format was {"buckets": {bucket1.id: bucket1, ...}} which seemed like the best option to me as well, but in aw-core you created a schema which is the same as the import format.

Also, apparently I had almost fixed the export button: https://github.com/ActivityWatch/aw-webui/blob/eb56c6194e65ce73ab29b868ea8121d936ba5144/src/views/Buckets.vue#L33

That's a comment, not a fix. IMO it doesn't make sense to even have a button for it if we know that we will change the format, just makes it confusing for the end-user since the exported data won't be able to be imported anyway.

Anyway, the most urgent issue should be to implement the import view

Why do we need a view for this? What's wrong with just having a "Export all" and "Import bucket" buttons in the buckets view?

IMO the most important thing is that the data is actually consistent, because otherwise everything else is simply unusable.

ErikBjare · 2019-02-26T21:20:16Z

The export format was {"buckets": {bucket1.id: bucket1, ...}} which seemed like the best option to me as well, but in aw-core you created a schema which is the same as the import format.

Edit the schema, it's not like we use it for anything. The schema should be the specification (edit: On second thought, maybe not, it wasn't quite as expressive that I'd like it to be iirc).

IMO it doesn't make sense to even have a button for it if we know that we will change the format, just makes it confusing for the end-user since the exported data won't be able to be imported anyway.

We should settle on a format, and a button would help us validate through usage that they are working properly.

Why do we need a view for this? What's wrong with just having a "Export all" and "Import bucket" buttons in the buckets view?

That'd be preferable, yes.

IMO the most important thing is that the data is actually consistent, because otherwise everything else is simply unusable.

Even inconsistent exports have some value (for those seeking to use the data outside ActivityWatch, for example), but I get what you saying. We should finalize the export schema and test it asap, sorry that I did such a sloppy job last time.

In addition, getting export/import to work on Android requires some extra work (both because of aw-server-rust, but also due to how the WebView needs special code to handle downloads/uploads) but once it works it would enable easily performed (but inconvenient) manual sync through the export/import cycle. This should help us prepare the web UI for cross-device functionality.

johan-bjareholt · 2019-02-27T07:10:53Z

Alright, when I get back home I'll revert it so it's back to the {"buckets": {bucket1.id: bucket1, ...}} format again. I will use the same format to export a single bucket. Any objections?

Even inconsistent exports have some value (for those seeking to use the data outside ActivityWatch, for example), but I get what you saying. We should finalize the export schema and test it asap, sorry that I did such a sloppy job last time.

Yes it has some some value, but I don't think it's worth the price of people later starting asking "why doesn't my buckets get imported?". If someone wants to export without the export button they could do that through the API and if they did it from the current web-ui button they'd have to know programming to parse the export anyway so at that point they'd probably either ask how to export or be able to find out how to do it themselves.

In addition, getting export/import to work on Android requires some extra work (both because of aw-server-rust, but also due to how the WebView needs special code to handle downloads/uploads) but once it works it would enable easily performed (but inconvenient) manual sync through the export/import cycle. This should help us prepare the web UI for cross-device functionality.

I'd assume that importing/exporting from files would be a rarely used feature on a smartphone, IMO we should prioritize getting sync working instead of spending time on this.

johan-bjareholt · 2019-02-27T19:09:27Z

I have now tested:

Exporting all buckets with /api/0/export
Importing all buckets with /api/0/import
Exporting one bucket /api/0/buckets/bid/export
Importing one bucket /api/0/import

The format is now {"buckets": {bucket1.id: bucket1, ...}} for everything.

johan-bjareholt · 2019-02-28T06:44:36Z

aw_server/rest.py

-        buckets = data["buckets"]
-        return app.api.import_all(buckets), 200
+        # If import comes from a form in th web-ui
+        if len(request.files) > 0:


Not really sure how I'm supposed to add support for this in aw-server-rust, I will have to investigate that.

ErikBjare · 2019-02-28T11:44:59Z

Also, apparently I did fix the export button thing, but the fix is in this PR: https://github.com/ActivityWatch/aw-webui/pull/119/files#diff-32c6cbf8ae7a966f7e41ec78a6d775cbR16

johan-bjareholt · 2019-02-28T12:48:10Z

@ErikBjare Looks identical to the code I wrote so I assume it has the same issue with browsers showing the preview of the JSON instead of prompting a file download?

johan-bjareholt · 2019-02-28T17:46:54Z

@ErikBjare Was able to fix the issue with it not being downloaded as a file by adding a header in this commit daa9e0b

ErikBjare

LGTM! (Can't approve my own PR)

started working on an import API endpoint

3126644

ghost assigned ErikBjare Apr 1, 2018

ghost added the review label Apr 1, 2018

ErikBjare requested a review from johan-bjareholt August 28, 2018 19:26

Merge branch 'master' into dev/import

4f7aac3

ErikBjare mentioned this pull request Aug 29, 2018

Started drafting really basic sync #50

Draft

4 tasks

implemented (untested) import REST endpoint, fixed API spec

ba4f833

ErikBjare added 2 commits September 10, 2018 13:05

fixed unsupported use of f-string

d1198a9

fixed bug in import

f348488

ErikBjare mentioned this pull request Jan 18, 2019

How to import data from ActivityWatch into new ActivityWatch database? ActivityWatch/activitywatch#263

Closed

Otto-AA mentioned this pull request Feb 7, 2019

Add way to merge buckets ActivityWatch/activitywatch#267

Closed

Made import/export data consistent as defined in aw-core schema

f98ed55

ghost assigned johan-bjareholt Feb 26, 2019

Fix mypy typecheck

d7bd781

johan-bjareholt added 2 commits February 27, 2019 20:05

Made exports/import consistents, no longer following aw-core schema

09519ee

Fixed mypy issues

a42dc6a

Allow bucket import from browser forms

e35fda2

johan-bjareholt reviewed Feb 28, 2019

View reviewed changes

Add HTTP header to export so it's downloaded as file from browsers

daa9e0b

ErikBjare commented Mar 2, 2019

View reviewed changes

johan-bjareholt merged commit f7b5103 into master Mar 2, 2019

ghost removed the review label Mar 2, 2019

johan-bjareholt deleted the dev/import branch March 2, 2019 14:58

Uh oh!

Conversation

ErikBjare commented Apr 1, 2018 • edited by johan-bjareholt Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ErikBjare commented Sep 10, 2018

Uh oh!

ErikBjare commented Sep 10, 2018

Uh oh!

johan-bjareholt commented Feb 26, 2019

Uh oh!

johan-bjareholt commented Feb 26, 2019

Uh oh!

johan-bjareholt commented Feb 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ErikBjare commented Feb 26, 2019

Uh oh!

johan-bjareholt commented Feb 26, 2019

Uh oh!

ErikBjare commented Feb 26, 2019

Uh oh!

johan-bjareholt commented Feb 26, 2019

Uh oh!

ErikBjare commented Feb 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

johan-bjareholt commented Feb 27, 2019 • edited by ErikBjare Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

johan-bjareholt commented Feb 27, 2019

Uh oh!

johan-bjareholt Feb 28, 2019

Choose a reason for hiding this comment

Uh oh!

ErikBjare commented Feb 28, 2019

Uh oh!

johan-bjareholt commented Feb 28, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

johan-bjareholt commented Feb 28, 2019

Uh oh!

ErikBjare left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ErikBjare commented Apr 1, 2018 •

edited by johan-bjareholt

Loading

johan-bjareholt commented Feb 26, 2019 •

edited

Loading

ErikBjare commented Feb 26, 2019 •

edited

Loading

johan-bjareholt commented Feb 27, 2019 •

edited by ErikBjare

Loading

johan-bjareholt commented Feb 28, 2019 •

edited

Loading