GPII-2630: Implement the new GPII data model #591

cindyli · 2018-02-23T21:00:30Z

The new GPII data model can be found here.

This pull request includes:

Removed the raw preferences server module;
The preferences server and the flow manager now share the same database called "gpii";
The preferences server in the dev config no longer reads/writes individual preferences files from the file system for preferences. These files are converted then loaded into PouchDB or CouchDB. Both the preferences server and flow manager reads/writes from pouchDB when running in the dev config, and CouchDB when running in the production config.
Removed "vagrant-configs" directory. This directory has the VM that starts GPII in the production mode. This VM has been replaced by the VM running in the universal root directory in which GPII in the production config can be started using the universal docker image. The details can be found in the Production setup instruction documentation.

These ops repository pull requests need to be merged once this pull request gets in the master:

These ops repos for setting up the preferences server are no longer needed:

The repo docker-preferences-dataloader should be renamed to gpii-dataloader.

Note:

ansible-flow-manager repo can be removed too if it's only used by the VM in the "vagrant-configs" directory, which has been removed in this branch.
The docker image cindyqili/gpii-dataloader used at vagrantCloudBasedContainers.sh should point to gpii/gpii-dataloader when the latter is ready.

… fixing db data store tests.

…ture.

…tion.

…ests. In progress of fixing node tests.

…nd prefsSafes.json in the new data structure.

…e base.

…abase port will be shared by not only the auth server, but also the preferences server.

…fs safes.

…ce http endpoints with internal invokers for getting, creating and updating preferences.

…ferences set (POST to /preferences) with an invoker createPrefsSet().

…in progress of adding updatePreferences().

…of fixing node tests that use the prefs server.

…ests.

… level.

…sted development environment locally.

…sion.

…l-tests.js.

… to include generated gpiiKeys.json and prefsSafes.json.

amb26 · 2018-05-11T14:36:40Z

@cindyli - Further renaming point through from Gregg - throughout the code, "preferences set" should be globally replaced by "preference set"

…d "preferences set" to "preference set".

…stions.

… npm postinstall script as well as adding code comments as per code review suggestions.

…he cloud URL to untrusted local flow managers.

… in with vladimir on windows VM.

gpii-bot · 2018-05-15T01:25:28Z

CI job passed: https://ci.gpii.net/job/universal-tests/846/

cindyli · 2018-05-15T01:39:37Z

@amb26, @simonbates, all review comments have been addressed. Also tested this branch with gpii/windows repo using its windows VM in these aspects:

"npm install" the published gpii-universal based off this branch
Running gpii/windows tests
login and logout with keys "carla" and "vladimir" in these configs:
** "all-in-local" config that uses the default gpii.config.development.all.local
** Running the cloud and the untrusted LFM as separate node instances on different ports: the cloud uses gpii.config.cloudBased.development.all.local and the untrusted LFM uses gpii.config.untrusted.development.

All above works well. Let me know if I missed anything.

This pull request is ready for more reviews.

gpii-bot · 2018-05-15T17:35:39Z

CI job failed: https://ci.gpii.net/job/universal-tests/847/

cindyli · 2018-05-15T18:18:16Z

ok to test

gpii-bot · 2018-05-15T18:45:51Z

CI job passed: https://ci.gpii.net/job/universal-tests/848/

amb26 · 2018-05-15T21:05:20Z

gpii/node_modules/dbOperation/index.js

+
+var fluid = require("infusion");
+
+fluid.module.register("gpii-db-operation", __dirname, require);


I meant by the rename for the entire module to be renamed - the main purpose of the rename is for its name in package.json to be a valid npm module name. This directory, and the name in package.json should be renamed as well. We can retain the camel-casing in the individual file names.

amb26 · 2018-05-15T21:13:52Z

gpii/node_modules/dbOperation/src/DbConst.js

+        isError: true
+    },
+    missingDoc: {
+        message: "The record of %docName is not found",


Why is this named "docName" when at all call sites it is actually sourced from a "docType"?
Also, the wording of the error is not terribly clear. Perhaps it should be something like "A record of type %docType was not found"?

Bump this comment

This has been addressed: https://github.com/cindyli/universal/blob/GPII-2630/gpii/node_modules/gpii-db-operation/src/DbConst.js#L39

Refresh browser?

amb26 · 2018-05-15T21:14:36Z

gpii/node_modules/dbOperation/src/DbConst.js

+        isError: true
+    },
+    mismatchedDocType: {
+        message: "The document type must be \"%docType\"",


It would probably be helpful to specify what type was actually chosen by the user

amb26 · 2018-05-15T21:18:54Z

gpii/node_modules/preferencesServer/src/preferencesServer.js

+ * @param {Component} ontologyHandler - An instance of gpii.ontologyHandler component.
+ * @param {Object} preferences - A preferences object.
+ * @param {String} view - An view that the provided preferences is based upon.
+ * @param {String} gpiiKey - (Optional) The GPII key to be created.


Optional params are signalled in JSDocs by putting the parameter name in square brackets - http://usejsdoc.org/tags-param.html#optional-parameters-and-default-values - although we tend to also retain the text [optional] for clarity in the description

amb26 · 2018-05-15T21:23:12Z

gpii/node_modules/dbOperation/src/DbDataStoreUtils.js

+
+    if (data && data.doc && data.value) {
+        result = {};
+        fluid.set(result, "oauth2ClientId", data.key);


Another clutch of unnecessary fluid.sets

amb26 · 2018-05-15T21:23:44Z

gpii/node_modules/dbOperation/src/DbDataStoreUtils.js

+
+    if (data && data.doc && data.value) {
+        result = {};
+        fluid.set(result, "accessToken", data.key);


gpii-bot · 2018-05-16T15:30:51Z

CI job passed: https://ci.gpii.net/job/universal-tests/849/

cindyli · 2018-05-16T15:55:05Z

@amb26, all comments have been addressed. Ready for more reviews. Thanks.

amb26 · 2018-05-16T16:21:34Z

gpii/node_modules/dbOperation/src/DataSource-pouchDB.js

@@ -84,7 +84,7 @@ var gpii = fluid.registerNamespace("gpii");

    /**
     * Decode the URL to find out the view/map function name and parameters for this function.
-     * @param url {String} The URL to be decoded
+     * @param {String} url - The URL to be decoded
     * @return {Object} The returned object is in the structure of:
     * {
     *     viewName: {String},


Note that the "else" branch in this function is untested - is it necessary?

I added a test to cover the "else" block.

gpii-bot · 2018-05-16T17:30:32Z

CI job passed: https://ci.gpii.net/job/universal-tests/850/

cindyli · 2018-05-16T17:31:24Z

@amb26, comments so far have been addressed again.

cindyli added 30 commits November 22, 2017 16:37

GPII-2630: Added new data files, synced database name, in progress of…

d58caf3

… fixing db data store tests.

GPII-2630: Continue to fix the auth server code with the new db struc…

bc09abb

…ture.

GPII-2630: In progress of adding a new GPII node module named dbOpera…

3997554

…tion.

Merge branch 'GPII-2558' into GPII-2630

0d8a650

GPII-2630: Completed the new "dbOperation" module and fixed all web t…

b1f0be6

…ests. In progress of fixing node tests.

GPII-2630: Fixed node tests.

81d6d01

Merge branch 'GPII-2558' into GPII-2630

b08da6a

GPII-2630: Converted testData/preferences/*.json into gpiiKeys.json a…

7d60234

…nd prefsSafes.json in the new data structure.

GPII-2630: Sync up the use of the new term "GPII keys" across the cod…

a645774

…e base.

GPII-2630: Renamed "authDBServerPort" to "dbServerPort" since the dat…

d5bf07a

…abase port will be shared by not only the auth server, but also the preferences server.

GPII-2630: Added views and db operation for retrieving and saving pre…

1487b35

…fs safes.

GPII-2630: In progress of refactoring the preferences server to repla…

48bb7f5

…ce http endpoints with internal invokers for getting, creating and updating preferences.

GPII-2630: Added a DbDataStore invoker for adding an GPII key.

0d60d1d

GPII-2630: Fixed a typo.

fbbb0c4

Merge branch 'GPII-2558' into GPII-2630

a5b3363

GPII-2630: Replaced the preferences server API for creating a new pre…

23c13d7

…ferences set (POST to /preferences) with an invoker createPrefsSet().

GPII-2630: Refactored the preferences server createPreferences() and …

39c21cd

…in progress of adding updatePreferences().

GPII-2630: Completed refactoring the preferences server. In progress …

87b94f4

…of fixing node tests that use the prefs server.

Merge branch 'GPII-2558' into GPII-2630

7526b2d

GPII-2630: In progress modifying config files and fixing acceptance t…

3297ffd

…ests.

GPII-2630: A couple of minor improvements on config files.

29c942c

Merge branch 'GPII-2558' into GPII-2630

e93a45f

GPII-2630: In progress of fixing acceptance tests.

2bc8dd5

GPII-2630: Moved the data files used for acceptance tests to a higher…

cc809fd

… level.

GPII-2630: Updated the config file and the doc for running GPII untru…

c2a274a

…sted development environment locally.

GPII-2630: Fixed more acceptance tests.

e9c5aea

GPII-2630: Fixed all node tests except SaveTests.js that is in discus…

47fa3b0

…sion.

GPII-2630: Added node tests that were not previously included into al…

738bdf8

…l-tests.js.

GPII-2630: Cleaned up unneeded preference data files and config files.

7383eaf

GPII-2630: More code clean up.

be3c901

cindyli added 2 commits May 10, 2018 21:05

GPII-2630: Added a prepare step for published gpii-universal packages…

ccd0afd

… to include generated gpiiKeys.json and prefsSafes.json.

GPII-2630: Renamed PCP to PSP.

1618b8f

cindyli added 6 commits May 11, 2018 11:18

GPII-2630: Converted all config files from json to json5. Also rename…

8e649ec

…d "preferences set" to "preference set".

GPII-2630: Fixed JSDoc code comments and some other code review sugge…

e1a608a

…stions.

GPII-2630: Working on publishing gpii-universal.

99a8c82

GPII-2630: Use the cross platform && to join multiple commands in the…

11c731e

… npm postinstall script as well as adding code comments as per code review suggestions.

GPII-2630: Added an environment variable GPII_CLOUD_URL for passing t…

b7031e3

…he cloud URL to untrusted local flow managers.

GPII-2630: Fixed the issue of "request entity too large" when logging…

ccb3fbc

… in with vladimir on windows VM.

GPII-2630: Revert the mistaken renaming: APSP -> APCP.

b1e1a53

amb26 reviewed May 15, 2018

View reviewed changes

GPII-2630: Addressed more code review comments.

5a5f118

amb26 reviewed May 16, 2018

View reviewed changes

GPII-2630: Improved the test coverage and an error message.

cdf652f

amb26 merged commit cdf652f into GPII:master May 16, 2018

cindyli deleted the GPII-2630 branch November 28, 2018 14:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GPII-2630: Implement the new GPII data model #591

GPII-2630: Implement the new GPII data model #591

cindyli commented Feb 23, 2018 •

edited

amb26 commented May 11, 2018

gpii-bot commented May 15, 2018

cindyli commented May 15, 2018

gpii-bot commented May 15, 2018

cindyli commented May 15, 2018

gpii-bot commented May 15, 2018

amb26 May 15, 2018

amb26 May 15, 2018

amb26 May 16, 2018

cindyli May 16, 2018

amb26 May 15, 2018

amb26 May 15, 2018

amb26 May 15, 2018

amb26 May 15, 2018

gpii-bot commented May 16, 2018

cindyli commented May 16, 2018

amb26 May 16, 2018

cindyli May 16, 2018

gpii-bot commented May 16, 2018

cindyli commented May 16, 2018


		var fluid = require("infusion");

		fluid.module.register("gpii-db-operation", __dirname, require);

GPII-2630: Implement the new GPII data model #591

GPII-2630: Implement the new GPII data model #591

Conversation

cindyli commented Feb 23, 2018 • edited

amb26 commented May 11, 2018

gpii-bot commented May 15, 2018

cindyli commented May 15, 2018

gpii-bot commented May 15, 2018

cindyli commented May 15, 2018

gpii-bot commented May 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gpii-bot commented May 16, 2018

cindyli commented May 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gpii-bot commented May 16, 2018

cindyli commented May 16, 2018

cindyli commented Feb 23, 2018 •

edited