Manually perform a Data Migration

You can download the code for this example here:

OnyxDevTools/onyx-database-samples/model-updates/ ... ManualMigrationDemo.java.

There are 2 parts to this example. The first part is to create an existing database containing the original structure of your model. The second is to change the data model and write a migration script to make it conform to the bulk schema updates.

When you have model updates that cannot be handled automatically by the lightweight migration there are use cases where you will have to perform a manual migration. In this even you will want to use the stream API within the PersistenceManager.

The stream API has several uses. It can be used to run perform analytics, run complex queries that require calculation, or in this case perform a manual migration.

This example will showcase one of the worst case scenarios which is moving a field from the Account entity and applying it to the related entity's Invoice. Since the balanceDue attribute has been removed from the Account entity, we must use the QueryMapStream lambda as opposed to the QueryStream. The primary difference is the QueryStream iterates through the serialized entities rather than a generic map. The generic map can be useful for accessing attributes that no longer exist on the serialized entity but still remain in the store.

1. Define a model including the Account and Invoice entities.

   The first iteration of our data model has the balanceDue on the Account.

   Notes:

   There is an inherent flaw in this data model. The balanceDue should not be represented in the Account entity and should be part of the latest Invoice.
2. Create a script to seed some test data
   Notes:

   Prior to running this script, ensure the database is deleted so that we can start with a clean slate.

   If the script runs successfully you have created an account with several invoices.
3. Run the Main class

   Connect to a new database and seed the data.

   Notes:

   Take note of the database location as you will need it later to use in another script.
4. Modify the Account and Invoice entities

   The balanceDue attribute on the Account has been removed.
5. Re-Connect to the Database

   Create a script to reconnect to the database.

   Notes:

   The database location should be the same as what was declared in the first script.
6. Define a Query to Stream

   The Query specifies what records to iterate through. The Stream API requires a Query. This Query is designed to pull all Accounts.

   Notes:

   The Query requires a QueryCriteria so in this case, we are using the QueryCriteria.NOT_NULL in order to specify a predicate that will give us all the Accounts as a result.
7. Use the Stream API with Lambdas

   The Stream API requires a Lambda or an extension of the QueryMapStream or QueryStream as a callback for each record in the query results. This example shows how to use a Lambda as the callback.

   Within the QueryMapStream we fetch the latest Invoice in order to update the balanceDue. We then use the PersistenceManager to persist the change.

   Notes:

   Onyx has a limitation using the QueryCriteria. You cannot specify a relationship as the first predicate. Instead the invoiceId not null is the first predicate.

   Use the QueryMapStream so that the entity is in format of a Map and we can see the hidden field(s) that no longer exist. Alternatively you can use the QueryStream to iterate through de-serialized entities.

   If the account was saved after model updates, the attribute balanceDue will no longer be available for us to read.

   The accountId was originally persisted as an int so you must typecast it as such.
8. Android NoSQL Database