FICO
FICO Xpress Optimization Examples Repository
FICO Optimization Community FICO Xpress Optimization Home
Back to examples browserPrevious exampleNext example

Reading sparse data from text files, spreadsheets and databases, and from memory

Description

Suppose we want to read in data of the form

i, j, value(i,j)

from a file, setting up a dynamic array A(range, range) with just the A(i,j)= value(i,j) for the pairs (i,j) which exist in the file. Here is an example which shows different ways of doing this. We read data from

  1. differently formatted text files (example trio.mos described in the 'Mosel User Guide', Section 3.4 Reading sparse data): using 'initializations from', 'readln', or 'diskdata',
  2. spreadsheets or databases (examples duo.mos, duosheet.mos, duooci.mos, duoexc.mos, and duodd.mos described in the Xpress Whitepaper 'Using ODBC and other database interfaces with Mosel') ,
  3. the application (C/Java) executing the Mosel model (examples datainputc.mos and datainputj.mos
into different arrays, and using writeln show that the arrays hold identical data.



Accessing spreadsheets and databases

With initializations from data from spreadsheets or databases is read in a similar way as text files are accessed; switching between data sources can be done by replacing the I/O driver name and the data source in the extended filename, such as "mmsheet.excel:data.xls" to access the spreadsheet data.xls or "mmodbc.odbc:data.mdb" to read from the MS Access database data.mdb and the string "mmodbc.odbc:DSN=mysql;DB=data" could be used to access a mysql database named data.

Instead of using an initializations block, we may read data from a database into a Mosel model by writing out the corresponding SQL statements (this format allows fine-tuning of the queries and gives access to a larger range of commands, in particular when working with databases). When developing applications with ODBC/SQL statements, the user is advised to use the following two parameter settings to obtain error messages and debugging information from ODBC:

 setparam("SQLverbose", true)
 setparam("SQLdebug", true)

The module mmoci (requires a separate license) defines a software-specific interface to Oracle databases that is used in a similar way as the ODBC connection.

Using initializations from the extended filename will take a form like "mmoci.oci:myname/mypassword@dbname" and it is also possible to work with PL/SQL statements like the following:

 OCIlogon('myname/mypassword@dbname')
 OCIexecute("select Index_i,Index_j,Value from MyDataTable", A5)
 OCIlogoff


Data input in memory using I/O drivers

Using initializations from it is also possible to read data into a Mosel model from the calling application (C/C++, C#, Java) using a combination of the I/O drivers mem (data held in memory) and raw driver (data in binary format) in C, drivers dotnet and dotnetraw in C# programs, and drivers java and jraw in Java programs.



Other options for data input

Other possibilities of inputting data into a Mosel model include

  • Static modules: data exchange between the calling application and a model via custom functions, using the Mosel Native Interface - see the example of embedding the cutting stock problem in the Whitepaper 'Embedding Optimization Algorithms'.
  • Custom format/custom data sources: the Mosel Native Interface allows the user to implement his own I/O drivers (to be used with initializations blocks and all other Mosel functionality involving file access, such as matrix export or model compilation). Possible uses of such user-written drivers include encoding/decoding of data on the fly or compression/decompression of data. See the Xpress Whitepaper 'Generalized file handling in Mosel' for the complete code and documentation of an example implementing a compression driver with the zip library.

    Furthermore, the Mosel IO driver pipe may be used to recover the output of an external program. See the section on I/O drivers in the 'Mosel Libraries Reference Manual' for more information.



Further explanation of this example: Whitepapers 'Using ODBC and other database interfaces with Mosel' and 'Generalized file handling in Mosel'


Source Files

Data Files





duo.mos

(!******************************************************
   Mosel Example Problems
   ====================== 

   file duo.mos
   ````````````
   Two ways of reading sparse data tables from
   spreadsheets or databases via ODBC.
   
   (c) 2008 Fair Isaac Corporation
       author: S. Heipcke, 2006, rev. Aug. 2023
*******************************************************!)

model "Duo input (ODBC)"
 uses "mmodbc"

 parameters
                                 ! Use Excel spreadsheet `data.xls'
!  CNCT = 'DSN=Excel Files;DBQ=C:/xpress/examples/mosel/WhitePapers/MoselData/data.xls'
!  CNCT = 'data.xls'
!  CNCTIO = "data.xls"
                                 ! Use Access database `data.mdb'
!  CNCT = 'DSN=MS Access Database;DBQ=C:/xpress/examples/mosel/WhitePapers/MoselData/data.mdb'
  CNCT = 'data.mdb'
  CNCTIO = "debug;data.mdb"
                                 ! Use mysql database `data' (not provided)
!  CNCT = 'DSN=mysql;DB=data'
!  CNCTIO = "debug;DSN=mysql;DB=data"
                                 ! Use SQLite database `data.sqlite' via ODBC
!  CNCT = 'DSN=sqlite;DATABASE=data.sqlite'
!  CNCTIO = "debug;DSN=sqlite;DATABASE=data.sqlite"
                                 ! Use SQLite database `data.sqlite' directly
!  CNCT = 'data.sqlite'
!  CNCTIO = "debug;data.sqlite"
 end-parameters

 declarations
  A4, A5: dynamic array(range,range) of real
 end-declarations

! First method: use an initializations block with the odbc driver
 initializations from "mmodbc.odbc:"+CNCTIO
  A4 as 'MyDataTable'
 end-initializations

! Second method: use SQL statements
 SQLconnect(CNCT)
 if not getparam("SQLsuccess"): setioerr("Database connection failed")
 SQLexecute("select Index_i,Index_j,Value from MyDataTable", A5)
 SQLdisconnect

! Now let us see what we have
 writeln('A4 is: ', A4)
 writeln('A5 is: ', A5)

end-model

Back to examples browserPrevious exampleNext example