Views
Data Migration
Current work being done on this project can be found at Data Migration Tools Project.
Overview
In Phase 1, we will focus on providing tools to migrate open clients and open accounts (see below for details). Support to migrate historical data will not be part of Phase I. The migration of historical data raises significant complexities-- ie, how to record client transfer between groups; how to handle Loan Officer movement between centers; how to handle changes in product definitions, how to handle changes that were made to the client record over time, etc. For this reason, we will focus on current data in Phase 1 of data migration.
Desired Process for Migrating Data
- MFI Installs, Configures and Deploys Mifos
- MFI Manually enters the following data into Mifos(in order):
- Offices
- Roles
- System Users
- Fees
- Funds (if applicable)
- Product categories
- Products (ie Product Definitions)
3. MFI Migrates Legacy Data into interim DB structure defined by Mifos Note, in this first phase, we will only support import for current data -- current clients, opening balances for savings accounts, and current loans. See below for details. 4. MFI Exports DB into XML format 5. MFI Imports Data into Mifos 6. MFI QAs? Data Migration and Repeats as necessary
Data that can be Migrated:
- Active Centers (if applicable):
- Active Groups
- Active Clients
- Active accounts:
- Client/Group/Center Accounts ("Client/Group/Center Charges" in the UI): Fee schedule, opening balance, amount overdue.
- Savings Accounts: Mandatory/Voluntary savings deposit schedule, opening balance, despoits overdue (if mandatory savings)
- Loan Accounts: There are 2 approaches we can take to migratinng active loan accounts. My instinct tells me that it is better to migrate the full transaction history of open loans. This is preferred from a client perspective. I suspect this approach will also minimize technical issues and gotchas that will come up if we only migrate the opening balance.
- Migrate Opening Loan Balances: This would mean migrating the outstanding amount of the loan, # of remaining installments, amount of remaining installments, etc. The downside to this approach is that performance statistics would not be accurate and it would be difficult to capture outstanding/overdue amounts.
- Migrate Full transaction history of Open Loans: For any open loan, we would capture the complete (original) repayment schedule and the full transaction history of the loan.
Selected Historical Data for the above entities:
There are some key historical data that are critical to capture even if full transactional histories are not migrated. Most of this data are typically reflected in the "Performance History" sections on the center, group, client, and account pages. I don't know how this data is captured in the DB, so I don't know if it's easy to migrate, for example, the amount of the last loan without migrating the entire transactional history of the last loan. This data will also need to be available via reports-- so if the data is denormalized, then reports need to know to look in the right place.
- Centers: Center Start Date
- Groups: Amount of Last Loan; Loan Cycle per Product; Group Approved Date (Mifos currently auto-generates this date; we should be able to back-date it to a date in the past)
- Clients: Amount of Last loan; Loan Cycle Number; Loan Cycle per Product; Meetings Attended; Meetings Missed; Client Start Date Mifos currently auto-generates this date; we should be able to back-date it to a date in the past)
- Savings Accounts: Date Account Opened; Total Deposits; Total Withdrawals; Total interest earned; Missed Deposits
- Loan Accounts: If we migrate the full transaction history for open loans, then I assume that all performance statistics will automatically automated correctly. If we just migrate "opening balances" of loans, then the following data needs to be collected: # of payments; # of missed payments; # of days in arrears.
Out of scope for migration in Phase 1
- Closed centers, groups, clients
- Closed accounts: savings, loans, center/group/client accounts
- Transactional history for savings accounts
- Tracking the change of Loan Officer during the migrated loan history; LO at time of migration will be assumed to be the LO for the entire history of the loan.
Other Requirements
- When migrating data, "Performance Statistics" Boxes on center, group, client, and account pages should be updated.
- Avoid data inconsistency in Performance Statistics: If MFI doesn't have the complete set of historical data captured in Performance Stats, system should still allow import and data inconsistencies should be avoided. For example, if an MFI doesn't have total historical deposits for savings accounts, this figure shouldn't be zero in mifos-- b/c it will contradict the current balance of the account. Also, if performance stats are calculated via counters, we need to make sure that migrated data doesnt break these counters in perpetuity. Data needs to be migrated in a way so that going forward, these performance statistics are correct. Note this could be a big issue and require a lot of work-- so need to validate the importance.
- Handling Configurations: MFIs can hide certain fields; they can configure certain fields to be mandatory; they can add "custom fields"; they can add look-up values. Our data migration solution needs to recognize this and be flexible enough to handle the variance between MFIs. Because there's no way to know in advance what custom fields an MFI will add, there needs to be a way for the MFI to easily add and change the XML template to handle their configurations. Ditto for the other configuration settings. This may mean that that less validation can be enforced through the XML and that data validation has to be deferred until after the data has been loaded into Mifos.
Open Issues and Questions
- Currently, Mifos doesn't allow loan disbursals on a non-meeting day (although we want to change this). Would migrated data be subjected to this restriction? If so, we would need to remove this restriction from Mifos.
- Meeting Schedules: As part of GK's migration process, it is my understanding that historical meeting schedules had to be created before loan repayment schedules could be migrated. I don't understand the issues entirely, but in order to migrate historical transactions of active loans, does Mifos need to generate a historical meeting schedule for these loans? Is there any way to avoid this requirement and simply capture expected due dates and amount and actual payment dates and payment amounts. Would this require additional work to remove some of the dependencies Mifos has on meetings (something we want to do anyways)?
- What if there is a mandatory field in Mifos that the MFI doesn't currently collect? Of all the fields currrently mandatory for centers, clients, groups-- the only potential problem I saw was with the mandatory field "father/spouse name"-- some MFIs may not collect his data.
- If we provide a tool to migrate current data, what additional technical work (if any) is required to support the migration of historical data? What issues are involved with migrating historical data?
- Amount of data validation: We need to determine amount of data validation we want to do for migrated data. This will have impact on scope.
- Localization: How will a localized version of Mifos (ie, Mifos in French) impact the XML? Or will it?
| subtopics: |
